diff --git a/doc/bin/cli.md b/doc/bin/cli.md index b1d716afe07..d69e77e5eb0 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -85,7 +85,16 @@ moq --connect https://relay.example.com/rooms/1 --connect-token "$TOKEN" --broad The `ts`, `fmp4`, and `flv` imports publish the input's own timestamps, mapped to the wall time the first frame arrived. A timeline that rewinds, such as a restarted encoder or a looping file wrapping to the top, ends the import with -an error; run it again to publish anew. A flagged MPEG-TS discontinuity that +an error; run it again to publish anew. `import ts` and `import srt` republish +instead when an MPEG-TS clock steps onto another timeline: the PCR and every +stream stepping back, flagged or not, or more than 10 s forward without a +`discontinuity_indicator`. The broadcast finishes there, and the rest of the +input publishes as a new instance under a new +[epoch](/concept/moq-lite#publisher-epochs), on the same SRT connection, which +`export ts --stitch --follow` carries on into. An unflagged step usually means +redundant encoders behind a changeover switch: time-lock their clocks, or have +the switch set `discontinuity_indicator`. `--program all` and `--passthrough` +still end at a rewind. A flagged MPEG-TS discontinuity that jumps forward continues the broadcast and is declared on the exported clock. MPEG-TS timestamps follow their source history across the 33-bit wrap (every 26.5 hours), including files imported faster than real time and flagged diff --git a/doc/lib/rs/moq-mux.md b/doc/lib/rs/moq-mux.md index e6e682ac030..b4595f52ea7 100644 --- a/doc/lib/rs/moq-mux.md +++ b/doc/lib/rs/moq-mux.md @@ -29,7 +29,9 @@ stream whose decoder identity changes, such as an SPS with a new profile or leve or an audio header with a new sample rate, continues under a new track name (`/` for the codec importers) while the old track ends; everything else, such as the coded size, updates the rendition in place. A -timeline that rewinds ends the import. fMP4 export fixes its track +timeline that rewinds ends the import. The MPEG-TS import ends at a clock step +onto another timeline with `ts::Restart`, which carries the input from the step +for the caller to publish as a new broadcast. fMP4 export fixes its track set at the init segment, so a new rendition or a changed configuration ends the export. The [CLI page](/bin/cli#import) covers these and the MPEG-TS specifics: one program per broadcast, private PES, damaged packets, and feed checks. diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index e8bb1e074b2..8d87d4aa19b 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -99,7 +99,6 @@ catalog linger (#5305). ## Required -- [A timebase step restarts the TS import](/quest/m1/ts-reliability/step-restart.md) - an unflagged step or a rewind republishes under a new epoch instead of ending the import and every subscriber's export - [A restart keeps its catalog](/quest/m1/ts-reliability/restart-catalog.md) - a restarted import publishes its catalog with its first media - [PAT changes](/quest/m1/ts-reliability/pat-lifecycle.md) - a running export follows a moved PMT PID - [A small step back keeps the TS export's video](/quest/m1/ts-reliability/step-back.md) - the export re-acquires on a sub-group step back from any source instead of dropping its video diff --git a/quest/m1/ts-reliability/restart-catalog.md b/quest/m1/ts-reliability/restart-catalog.md index 3253f7b1d21..33d0ea8aa88 100644 --- a/quest/m1/ts-reliability/restart-catalog.md +++ b/quest/m1/ts-reliability/restart-catalog.md @@ -22,7 +22,3 @@ Tests: replaying a step through the supervisor, the second epoch's catalog is published with its first group. Public API: none. Wire: none. - -## Required - -- [A timebase step restarts the TS import](/quest/m1/ts-reliability/step-restart.md) - the restart this speeds up diff --git a/quest/m1/ts-reliability/step-back.md b/quest/m1/ts-reliability/step-back.md index 567778d5a95..d8c65eb6236 100644 --- a/quest/m1/ts-reliability/step-back.md +++ b/quest/m1/ts-reliability/step-back.md @@ -23,8 +23,8 @@ The fix belongs where video decode time is reconstructed, not in the jitter buffer: the step has to reach the jitter buffer as a regression it can act on. -[A timebase step restarts the TS import](/quest/m1/ts-reliability/step-restart.md) -turns any rewind from a TS or SRT import into a new epoch, so this remains +Since [moq-dev/moq#7144](https://github.com/moq-dev/moq/pull/7144), a TS or SRT +import turns any rewind into a new epoch, so this remains for every other source (fMP4, MKV, FLV, RTMP, hang publishers). Kept a separate quest for that reason. @@ -32,7 +32,3 @@ Tests: an end-to-end TS export of an H.264 + AAC source with the −560 ms step, which fails on main and keeps video rendering with the fix. Public API: none. Wire: none. - -## Related - -- [A timebase step restarts the TS import](/quest/m1/ts-reliability/step-restart.md) - handles a TS or SRT source's rewind at import diff --git a/quest/m1/ts-reliability/step-restart.md b/quest/m1/ts-reliability/step-restart.md deleted file mode 100644 index b77d17ee47f..00000000000 --- a/quest/m1/ts-reliability/step-restart.md +++ /dev/null @@ -1,61 +0,0 @@ -# [M] A timebase step restarts the TS import under a new epoch - -## Goal - -`moq import ts` and `moq import srt` keep a broadcast available across a -source timebase step. A rewind, flagged or not, and an unflagged step in -either direction end the importer's instance. The CLI then republishes in the -same process as a new broadcast under a fresh epoch, instead of exiting. -Subscribers follow the new instance like any republish, and no `moq export ts` -exits on a step it reads as clock drift. - -## Plan - -Root cause of [#7102](https://github.com/moq-dev/moq/issues/7102), established -2026-10-28: the contribution comes from two redundant encoders behind a -changeover switch. Their clocks share a frequency reference (the offset holds -at ~16,844.5 s, a few ppm over days) but start ~4 h 40 m apart, and the switch -sets no `discontinuity_indicator` (ISO/IEC 13818-1 §2.4.3.5, TR 101 290 2.3b). -That is an upstream misconfiguration: redundant encoders are normally -time-locked, or the switch flags the break. MoQ tolerates it anyway, as ffmpeg, -GStreamer and VLC do by inferring a break from the jump. - -Decided 2026-10-28 (maintainer): - -- **Two encoders are two broadcasts.** An unflagged step means a different - source, so it gets a new epoch rather than a splice. This keeps #4543's rule - (group starts never go backwards within an instance) and the m1 decision - that a redundant ingest pair runs on distinct epochs. A flagged forward step - still splices with break markers, as today. -- **Detection:** PCR and every timed PID step together, forward beyond ~10 s - or backward by any amount that is not a 33-bit wrap. These are ffmpeg's - `dts_delta_threshold` bounds, not TR 101 290's 100 ms, which SRT loss can - exceed. `ts/health.rs` already computes the modular step for 2.3b; reuse it. -- **The supervisor lives in the CLI, not moq-mux.** The importer ends its - instance with a typed restart cause. `moq import ts/srt` republishes under a - fresh epoch on the same path, keeping the SRT connection. This is the - supervisor the 2026-10-08 decision allows ("a supervisor that wants to - continue publishes a new broadcast under a new epoch"). An in-process - `import.restart(broadcast)` in moq-mux stays rejected. -- Rejected: inferring the flag forward (a splice across two sources), and - reopening #4543 with a catalog timeline generation (a second epoch under - another name, conflicting with time addressing). - -Tests: a synthetic clip with the whole-multiplex unflagged step both ways (the -reporter offered a public reproducer) restarts under a new epoch. A -subscriber's `moq export ts --stitch` follows it. Lossy PCR gaps under the -bound never restart. - -Docs: `doc/bin/cli.md` ("run it again to publish anew") describes the restart -and names the upstream fix (time-locked encoders, or a flagged switch). - -Public API: the importer's error gains a restart cause. Wire: none. - -## Closes - -- [#7102](https://github.com/moq-dev/moq/issues/7102) - an unflagged timebase step ends `moq import` and `moq export ts` - -## Related - -- [A restart keeps its catalog](/quest/m1/ts-reliability/restart-catalog.md) - removes the catalog hold a restart pays -- [Decide whether a stitched export demotes its old leg](/quest/m1/ts-reliability/stitch-catch-up.md) - may speed the replacement's first keyframe at each restart's switch diff --git a/rs/moq-cli/src/main.rs b/rs/moq-cli/src/main.rs index 5327c308134..43808bc97a1 100644 --- a/rs/moq-cli/src/main.rs +++ b/rs/moq-cli/src/main.rs @@ -751,8 +751,7 @@ fn spawn_import( let name = require_broadcast(name, "import ts --program all")?; Publish::ts_programs(origin.clone(), moq_net::Path::new(&name)?, config) } else { - let broadcast = origin.create_broadcast(&name).context("failed to create broadcast")?; - Publish::from_publication(broadcast, &format, config)? + Publish::stdin(origin, &name, &format, config)? } .with_linger(linger); publish.announce(epoch)?; diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index eb24a02091f..cc895443222 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -320,6 +320,8 @@ pub struct Publish { report: Option, // How long a stdin import keeps its broadcast up after EOF; see [`Self::with_linger`]. linger: std::time::Duration, + // How a stdin TS import republishes when its source's clock steps; see [`Self::stdin`]. + republish: Option, } /// A broadcast's announcement, held until its catalog first goes out. @@ -383,6 +385,7 @@ impl Publish { route: None, report: None, linger: moq_mux::Linger::DEFAULT, + republish: None, }); } @@ -432,17 +435,37 @@ impl Publish { route: None, report: None, linger: moq_mux::Linger::DEFAULT, + republish: None, }) } - /// Retain origin announcement control alongside the stdin decoder's fill handle. - pub(super) fn from_publication( - publication: moq_net::origin::Publication, + /// Build a publisher decoding `format` from stdin into a new broadcast at `name` on + /// `origin`, retaining its announcement control alongside the decoder's fill handle. + /// + /// A TS import whose source's clock steps onto another timeline ([`ts::Restart`]), such as + /// a changeover between two encoders whose clocks were never locked, finishes the broadcast + /// and republishes the rest of the input at `name` under a new epoch. + pub(super) fn stdin( + origin: &moq_net::origin::Producer, + name: &str, format: &PublishFormat, config: moq_mux::catalog::Config, ) -> anyhow::Result { + let publication = origin.create_broadcast(name).context("failed to create broadcast")?; + let republish = match *format { + PublishFormat::Ts { program } => Some(Republish { + origin: origin.clone(), + name: name.to_string(), + config: config.clone(), + program, + report: None, + echo: None, + }), + _ => None, + }; let mut result = Self::new(publication.broadcast().clone(), format, config)?; result.publication = Some(publication); + result.republish = republish; Ok(result) } @@ -467,6 +490,7 @@ impl Publish { route: None, report: None, linger: moq_mux::Linger::DEFAULT, + republish: None, } } @@ -510,6 +534,7 @@ impl Publish { route: None, report: None, linger: moq_mux::Linger::DEFAULT, + republish: None, }) } @@ -522,6 +547,9 @@ impl Publish { let (Some(broadcast), Source::Stream { catalog, .. }) = (self.broadcast.as_mut(), &self.source) else { anyhow::bail!("--report reports a stdin import's one catalog"); }; + if let Some(republish) = &mut self.republish { + republish.report = Some(connection.clone()); + } let report = match catalog { PublishCatalog::Media(catalog) => crate::report::Stats::new(broadcast, catalog, connection)?, PublishCatalog::Ts(catalog) => crate::report::Stats::new(broadcast, catalog, connection)?, @@ -540,6 +568,9 @@ impl Publish { #[cfg(feature = "capture")] Source::Capture { .. } => anyhow::bail!("--echo-path solicits feedback in a stdin import's one catalog"), }; + if let Some(republish) = &mut self.republish { + republish.echo = Some(echo.clone()); + } match catalog { PublishCatalog::Media(catalog) => catalog.mutate(|catalog| catalog.echo = Some(echo))?, PublishCatalog::Ts(catalog) => catalog.mutate(|catalog| catalog.echo = Some(echo))?, @@ -579,10 +610,12 @@ impl Publish { _ => None, }; let publication = Publication { + control: self.publication, broadcast: self.broadcast, pending, report: self.report, linger: self.linger, + republish: self.republish, }; decode(decoder, catalog, tokio::io::stdin(), publication).await } @@ -631,6 +664,8 @@ impl Publish { /// What a stdin import does around its media: announce, report, and retain the end. #[derive(Default)] struct Publication { + /// The announcement control, held while the broadcast is up so its path stays announced. + control: Option, /// The broadcast to keep up after EOF, or `None` when the decoder owns its broadcasts. broadcast: Option, /// The announcement held until the catalog first goes out. @@ -640,6 +675,74 @@ struct Publication { report: Option, /// How long the finished broadcasts stay up after EOF. linger: std::time::Duration, + /// How a step of the source's clock republishes, or `None` to end the import there. + republish: Option, +} + +/// How a stdin TS import republishes its path when the source's clock steps onto another +/// timeline ([`ts::Restart`]): as a new broadcast under a new epoch, with the same catalog +/// sections. +struct Republish { + origin: moq_net::origin::Producer, + name: String, + config: moq_mux::catalog::Config, + program: Option, + /// The connection `--report` measures, when the import reports. + report: Option>, + echo: Option, +} + +impl Republish { + /// A new instance of the broadcast under a new epoch, announced once its catalog goes out. + fn instance(&self) -> anyhow::Result { + let control = self + .origin + .create_broadcast(&self.name) + .context("failed to create broadcast")?; + let mut broadcast = control.broadcast().clone(); + let (import, mut catalog) = ts_import(&mut broadcast, self.config.clone(), self.program)?; + // Before the import declares a track, so the sections ride the first catalog. + let report = match &self.report { + Some(connection) => Some(crate::report::Stats::new(&mut broadcast, &catalog, connection.clone())?), + None => None, + }; + if let Some(echo) = &self.echo { + catalog.mutate(|catalog| catalog.echo = Some(echo.clone()))?; + } + let epoch = moq_net::Epoch::mint(); + tracing::info!(%epoch, "publisher instance"); + let pending = Pending::new(control.clone(), moq_net::origin::Route::default().with_epoch(epoch))?; + Ok(Instance { + decoder: PublishDecoder::Ts(Box::new(import)), + catalog: PublishCatalog::Ts(catalog), + control: Some(control), + broadcast: Some(broadcast), + pending: Some(pending), + report, + }) + } +} + +/// One publisher instance of a stdin import: the decoder filling its broadcast, and what +/// announces and reports it. +struct Instance { + decoder: PublishDecoder, + catalog: PublishCatalog, + control: Option, + broadcast: Option, + pending: Option, + report: Option, +} + +/// The input read so far, and the import's counters logged as they move. +struct Reader { + input: R, + buffer: bytes::BytesMut, + // Counters reported so far, so only the change is logged. A live feed is + // diagnosed by the rate at which these climb, and stdin may never end, so + // they have to surface as they accumulate rather than at exit. + log: ts::stats::Log, + sampled: moq_clock::Instant, } /// Decode `input` into the broadcast until EOF. @@ -649,116 +752,219 @@ struct Publication { /// as removed tracks, and a catalog dropped unfinished reads as a publisher that vanished. /// The finished broadcasts then stay servable for the linger, so a subscriber that saw the /// final catalog only at EOF can still request them. +/// +/// A step of the source's clock ends the instance there. With `republish`, the rest of the +/// input publishes a new instance on the same path; otherwise the import fails with it. async fn decode( - mut decoder: PublishDecoder, - mut catalog: PublishCatalog, - mut input: impl tokio::io::AsyncRead + Unpin, + decoder: PublishDecoder, + catalog: PublishCatalog, + input: impl tokio::io::AsyncRead + Unpin, publication: Publication, ) -> anyhow::Result<()> { let Publication { + control, broadcast, - mut pending, - mut report, + pending, + report, linger, + republish, } = publication; - // An empty BytesMut grows to only 64 bytes and clearing it keeps that capacity. - // Stdin dispatches every read to a blocking thread, so those tiny reads can - // backpressure a live feed. Reserve a full IO chunk; read_buf still returns - // whatever is available without waiting to fill it. - let mut buffer = bytes::BytesMut::with_capacity(64 * 1024); + let mut instance = Instance { + decoder, + catalog, + control, + broadcast, + pending, + report, + }; + let mut reader = Reader { + input, + // An empty BytesMut grows to only 64 bytes and clearing it keeps that capacity. + // Stdin dispatches every read to a blocking thread, so those tiny reads can + // backpressure a live feed. Reserve a full IO chunk; read_buf still returns + // whatever is available without waiting to fill it. + buffer: bytes::BytesMut::with_capacity(64 * 1024), + log: ts::stats::Log::default(), + sampled: moq_clock::now(), + }; - // Counters reported so far, so only the change is logged. A live feed is - // diagnosed by the rate at which these climb, and stdin may never end, so - // they have to surface as they accumulate rather than at exit. - let mut log = ts::stats::Log::default(); - let mut sampled = moq_clock::now(); - let sections = report.as_ref().map(crate::report::Stats::sections); - let report_mpegts = |snapshot: &ts::stats::Snapshot| { - if let Some(sections) = §ions { - sections.set(|report| report.mpegts = Some(snapshot.clone())); + let result = loop { + let restart = match instance.read(&mut reader).await { + Ok(Some(restart)) => restart, + Ok(None) => break Ok(()), + Err(err) => break Err(err), + }; + let Some(republish) = &republish else { + break Err(restart.into()); + }; + if let Err(err) = instance.restart(republish, restart, &mut reader.log) { + break Err(err); } }; + instance.end(result, &mut reader.log, linger).await +} + +impl Instance { + /// Read and decode until EOF, returning `None`, or until a step of the source's clock ends + /// the instance. + async fn read( + &mut self, + reader: &mut Reader, + ) -> anyhow::Result> { + let Self { + decoder, + pending, + report, + .. + } = self; + let sections = report.as_ref().map(crate::report::Stats::sections); + // Run the read/decode loop so an error surfaces to the caller rather than + // dropping the decoder (and its tracks) with a bare Error::Dropped. + let read = async { + loop { + reader.buffer.clear(); + let n = tokio::io::AsyncReadExt::read_buf(&mut reader.input, &mut reader.buffer).await?; + if n == 0 { + return Ok(None); // EOF + } + if let Some(restart) = feed(decoder, pending, &reader.buffer)? { + return Ok(Some(restart)); + } + + if reader.sampled.elapsed() >= ts::stats::Log::INTERVAL { + reader.sampled = moq_clock::now(); + if let Some(stats) = decoder.stats() { + if let Some(sections) = §ions { + sections.set(|report| report.mpegts = Some(stats.clone())); + } + reader.log.sample(stats); + } + } + } + }; + match report.as_mut() { + Some(report) => tokio::select! { + result = read => result, + never = report.run() => match never {}, + }, + None => read.await, + } + } - // Run the read/decode loop so an error surfaces here rather than - // dropping the decoder (and its tracks) with a bare Error::Dropped. - let read = async { + /// The source's clock stepped: finish this instance before the step, and replace it with a + /// new one on the same path that takes the rest of the input. + fn restart(&mut self, republish: &Republish, restart: ts::Restart, log: &mut ts::stats::Log) -> anyhow::Result<()> { + let mut restart = restart; loop { - buffer.clear(); - let n = tokio::io::AsyncReadExt::read_buf(&mut input, &mut buffer).await?; - if n == 0 { - return Ok(()); // EOF + tracing::warn!(step = %restart.step, "the source's clock stepped; republishing"); + self.decoder.finish()?; + if let Some(stats) = self.decoder.stats() { + self.report_mpegts(&stats); + log.finish(&stats); } - decoder.decode_chunk(&buffer)?; - if let Some(announce) = &mut pending - && announce.announce()? - { - pending = None; + if let Some(report) = &mut self.report { + report.finish(); } - - if sampled.elapsed() >= ts::stats::Log::INTERVAL { - sampled = moq_clock::now(); - if let Some(stats) = decoder.stats() { - report_mpegts(&stats); - log.sample(stats); - } + self.catalog.finish()?; + // The finished broadcast ends as its handles drop, withdrawing its announcement. + *self = republish.instance()?; + *log = ts::stats::Log::default(); + match feed(&mut self.decoder, &mut self.pending, &restart.remainder)? { + Some(next) => restart = next, + None => return Ok(()), } } - }; - let result: anyhow::Result<()> = match report.as_mut() { - Some(report) => tokio::select! { - result = read => result, - never = report.run() => match never {}, - }, - None => read.await, - }; + } - // Flush on a clean EOF; on any error (read, decode, or the flush - // itself) abort with the real cause so subscribers see it instead of - // a bare Error::Dropped. - let outcome = result.and_then(|()| decoder.finish()); - // The drain can release the first catalog. Without one the input declared no track, and - // announcing would only advertise an empty broadcast. - let outcome = outcome.and_then(|finished| { - if let Some(announce) = &mut pending - && !announce.announce()? - { - anyhow::bail!("input ended before it declared a track"); + /// Set the report's `mpegts` section, when the import reports. + fn report_mpegts(&self, snapshot: &ts::stats::Snapshot) { + if let Some(report) = &self.report { + report.sections().set(|report| report.mpegts = Some(snapshot.clone())); } - Ok(finished) - }); - // The drain at end of input can publish a frame nothing vouched for, so the - // final snapshot is only complete after `finish`. - if let Some(stats) = decoder.stats() { - report_mpegts(&stats); - log.finish(&stats); - } - match outcome { - Ok(mut finished) => { - // Finished before the broadcast is, so it lingers with the media. - if let Some(report) = &mut report { - report.finish(); + } + + /// Finish on a clean EOF and serve the finished broadcasts for the linger, or abort with + /// the real cause of `result`, so subscribers see it instead of a bare Error::Dropped. + async fn end( + mut self, + result: anyhow::Result<()>, + log: &mut ts::stats::Log, + linger: std::time::Duration, + ) -> anyhow::Result<()> { + // Flush on a clean EOF; on any error (read, decode, or the flush + // itself) abort with the real cause. + let outcome = result.and_then(|()| self.decoder.finish()); + // The drain can release the first catalog. Without one the input declared no track, and + // announcing would only advertise an empty broadcast. + let outcome = outcome.and_then(|finished| { + if let Some(announce) = &mut self.pending + && !announce.announce()? + { + anyhow::bail!("input ended before it declared a track"); } - catalog.finish()?; - finished.extend(broadcast); - tracing::info!(?linger, "input ended; serving the finished broadcast"); - // The decoder and catalog hold the finished tracks, so they go only once it closes. - moq_mux::Linger::new(finished, (decoder, catalog, report), linger) - .wait() - .await; - Ok(()) + Ok(finished) + }); + // The drain at end of input can publish a frame nothing vouched for, so the + // final snapshot is only complete after `finish`. + if let Some(stats) = self.decoder.stats() { + self.report_mpegts(&stats); + log.finish(&stats); } - Err(err) => { - let reason = moq_net::Error::Transport(err.to_string()); - if let Some(report) = report { - report.abort(reason.clone()); + let Self { + decoder, + mut catalog, + control, + broadcast, + report, + .. + } = self; + match outcome { + Ok(mut finished) => { + // Finished before the broadcast is, so it lingers with the media. + let mut report = report; + if let Some(report) = &mut report { + report.finish(); + } + catalog.finish()?; + finished.extend(broadcast); + tracing::info!(?linger, "input ended; serving the finished broadcast"); + // The decoder and catalog hold the finished tracks, so they go only once it closes. + moq_mux::Linger::new(finished, (decoder, catalog, report, control), linger) + .wait() + .await; + Ok(()) + } + Err(err) => { + let reason = moq_net::Error::Transport(err.to_string()); + if let Some(report) = report { + report.abort(reason.clone()); + } + catalog.abort(reason.clone()); + decoder.abort(reason); + Err(err) } - catalog.abort(reason.clone()); - decoder.abort(reason); - Err(err) } } } +/// Decode `chunk` into `decoder`, announcing `pending` once its catalog goes out, or return the +/// step of the source's clock that ended the instance. +fn feed( + decoder: &mut PublishDecoder, + pending: &mut Option, + chunk: &[u8], +) -> anyhow::Result> { + if let Err(err) = decoder.decode_chunk(chunk) { + return err.downcast::().map(Some); + } + if let Some(announce) = pending + && announce.announce()? + { + *pending = None; + } + Ok(None) +} + #[cfg(feature = "capture")] impl CaptureArgs { /// The video source named by the flags, defaulting to the default camera. @@ -1136,11 +1342,15 @@ pub(crate) mod tests { // streams land in the broadcast instead of being dropped by the media-only path. // The broadcast is created on a throwaway origin so the exporter can resolve it by path. let origin = moq_tokio::origin::spawn(); - let broadcast = origin.create_broadcast("cli").unwrap(); - broadcast.announce(Default::default()).unwrap(); - settle().await; let mut publish = - Publish::from_publication(broadcast, &PublishFormat::Ts { program: None }, Default::default()).unwrap(); + Publish::stdin(&origin, "cli", &PublishFormat::Ts { program: None }, Default::default()).unwrap(); + publish + .publication + .as_ref() + .unwrap() + .announce(Default::default()) + .unwrap(); + settle().await; #[cfg_attr( not(feature = "capture"), expect( @@ -1332,14 +1542,19 @@ pub(crate) mod tests { }; let origin = moq_tokio::origin::spawn(); - let broadcast = origin.create_broadcast("pt.hang").unwrap(); - broadcast.announce(Default::default()).unwrap(); - let mut publish = Publish::from_publication( - broadcast, + let mut publish = Publish::stdin( + &origin, + "pt.hang", &PublishFormat::TsPassthrough { pcr_pid: None }, Default::default(), ) .unwrap(); + publish + .publication + .as_ref() + .unwrap() + .announce(Default::default()) + .unwrap(); #[cfg_attr( not(feature = "capture"), expect( @@ -1778,8 +1993,8 @@ pub(crate) mod tests { let origin = moq_tokio::origin::spawn(); let consumer = origin.consume(); let mut announced = consumer.announced(); - let broadcast = origin.create_broadcast("demo").unwrap(); - let publish = Publish::from_publication(broadcast.clone(), &format, Default::default()).unwrap(); + let publish = Publish::stdin(&origin, "demo", &format, Default::default()).unwrap(); + let broadcast = publish.publication.clone().unwrap(); #[cfg_attr( not(feature = "capture"), expect( @@ -1876,6 +2091,81 @@ pub(crate) mod tests { ); } + /// A source whose clock steps back republishes the path under a new epoch, announced once + /// its catalog goes out, with the sections the first instance's catalog carried. + #[tokio::test(start_paused = true)] + async fn a_clock_step_republishes_under_a_new_epoch() { + // 5 s of H.264 and AAC, its clock on PID 0x100. + const CLIP: &[u8] = include_bytes!("../../moq-mux/src/container/ts/test_data/scte35/bbb5s.ts"); + let mut flag = vec![0x47, 0x01, 0x00, 0x20, 183, 0x80]; + flag.resize(188, 0xff); + + let origin = moq_tokio::origin::spawn(); + let mut announced = origin.consume().announced(); + let mut publish = Publish::stdin( + &origin, + "demo", + &PublishFormat::Ts { program: None }, + Default::default(), + ) + .unwrap() + .with_echo(hang::catalog::Echo::new( + moq_net::path::Relative::new("viewer").unwrap(), + )) + .unwrap(); + let first = moq_net::Epoch::mint(); + publish.announce(first.clone()).unwrap(); + #[cfg_attr( + not(feature = "capture"), + expect( + irrefutable_let_patterns, + reason = "`Source::Capture` exists only with the capture feature" + ) + )] + let Source::Stream { decoder, catalog } = publish.source else { + panic!("expected a stream source"); + }; + let control = publish.publication.unwrap(); + let publication = Publication { + pending: Some(Pending::new(control.clone(), publish.route.unwrap()).unwrap()), + control: Some(control), + broadcast: publish.broadcast, + republish: publish.republish, + ..Default::default() + }; + let (mut stdin, input_end) = tokio::io::duplex(2 * CLIP.len() + flag.len()); + let mut import = std::pin::pin!(decode(decoder, catalog, input_end, publication)); + + // The epoch of the next instance announced on the path. + let mut next = async |import: &mut std::pin::Pin<&mut _>| { + tokio::select! { + res = import => panic!("the import returned before its input ended: {res:?}"), + _ = tokio::time::sleep(Duration::from_secs(30)) => panic!("never announced"), + epoch = async { + loop { + use moq_net::announce::Event; + if let Event::Start(announce) | Event::Restart(announce) = announced.next().await.unwrap() { + break announce.route.epoch.unwrap(); + } + } + } => epoch, + } + }; + tokio::io::AsyncWriteExt::write_all(&mut stdin, CLIP).await.unwrap(); + assert_eq!(next(&mut import).await, first); + // The encoder restarts its clock and plays the clip again: a rewind, flagged. + tokio::io::AsyncWriteExt::write_all(&mut stdin, &[&flag[..], CLIP].concat()) + .await + .unwrap(); + assert_ne!(next(&mut import).await, first, "the new instance has its own epoch"); + let broadcast = origin.consume().request("demo", None).await.unwrap(); + let mut catalogs = hang::catalog::Catalog::<()>::subscribe(&broadcast).await.unwrap(); + let catalog = catalogs.next().await.unwrap().expect("a catalog"); + assert!(catalog.echo.is_some(), "the new instance solicits feedback too"); + drop(stdin); + import.await.unwrap(); + } + /// Input that ends before declaring any track fails, rather than announcing a broadcast /// with nothing in it. These inputs hold only metadata (a PMT, an FLV header, an AUD) /// that configures no track. @@ -1892,8 +2182,8 @@ pub(crate) mod tests { ] { let origin = moq_tokio::origin::spawn(); let mut announced = origin.consume().announced(); - let broadcast = origin.create_broadcast("demo").unwrap(); - let publish = Publish::from_publication(broadcast.clone(), &format, Default::default()).unwrap(); + let publish = Publish::stdin(&origin, "demo", &format, Default::default()).unwrap(); + let broadcast = publish.publication.clone().unwrap(); #[cfg_attr( not(feature = "capture"), expect( diff --git a/rs/moq-mux/src/container/ts/demux.rs b/rs/moq-mux/src/container/ts/demux.rs index d1833c17da9..15f977f1f61 100644 --- a/rs/moq-mux/src/container/ts/demux.rs +++ b/rs/moq-mux/src/container/ts/demux.rs @@ -123,6 +123,29 @@ impl Demux { }) } + /// Where [`next`](Self::next) frames from: the input offset of the first byte not yet framed, + /// and the framer's state there, for [`peek`](Self::peek) to read ahead from. + pub fn cursor(&self) -> (u64, Framer) { + (self.position + self.off as u64, self.framer) + } + + /// Frame the next whole packet at or after input offset `*at` without routing it, advancing + /// `*at` and `framer` as [`next`](Self::next) would. `None` once only a partial packet + /// remains, with both left where more input resumes. + pub fn peek(&self, at: &mut u64, framer: &mut Framer) -> Option<(u64, &[u8; TsPacket::SIZE])> { + let mut off = usize::try_from(*at - self.position).expect("an offset in the pushed input"); + let found = framer.next(&self.scratch, &mut off); + *at = self.position + off as u64; + let found = found?; + let pkt = self.scratch[found..found + TsPacket::SIZE].try_into().unwrap(); + Some((self.position + found as u64, pkt)) + } + + /// The input pushed but not yet framed: what [`next`](Self::next) would read on. + pub fn remainder(&self) -> Bytes { + Bytes::copy_from_slice(&self.scratch[self.off..]) + } + /// Drop the framed input, keeping a trailing partial packet for the next push. pub fn drain(&mut self) { self.health.drained(&self.scratch, self.off); diff --git a/rs/moq-mux/src/container/ts/health.rs b/rs/moq-mux/src/container/ts/health.rs index 3a7ae9a3976..4c1bda7352b 100644 --- a/rs/moq-mux/src/container/ts/health.rs +++ b/rs/moq-mux/src/container/ts/health.rs @@ -29,7 +29,7 @@ const TABLE_LIMIT: u64 = PCR_HZ / 2; /// left TS 101 154 in 2005. const PCR_LIMIT: u64 = PCR_HZ / 10; /// `PTS_error`: 700 ms. -const PTS_LIMIT: u64 = PCR_HZ * 7 / 10; +pub(super) const PTS_LIMIT: u64 = PCR_HZ * 7 / 10; /// ISO/IEC 13818-1 G.1: sync is lost after two consecutive corrupt sync bytes, and acquired /// after five consecutive correct ones. const SYNC_LOST: u8 = 2; @@ -241,14 +241,14 @@ impl Health { let Some(last) = self.pcr.replace(value) else { return; }; - let step = (value + PCR_WRAP - last) % PCR_WRAP; - let forward = step <= PCR_WRAP / 2; let errors = self.errors.pid(pid); - if forward && step > PCR_LIMIT { - errors.pcr_repetition_error += 1; - } - if !forward || step > PCR_LIMIT { - errors.pcr_discontinuity_indicator_error += 1; + match Delta::between(last, value, PCR_WRAP) { + Delta::Forward(step) if step > PCR_LIMIT => { + errors.pcr_repetition_error += 1; + errors.pcr_discontinuity_indicator_error += 1; + } + Delta::Forward(_) => {} + Delta::Back(_) => errors.pcr_discontinuity_indicator_error += 1, } } @@ -303,6 +303,28 @@ impl Health { } } +/// The step between two readings of a clock that wraps: forward by up to half the range, +/// otherwise back, so a wrap reads as the small forward step it is. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum Delta { + /// Ahead by this many ticks. + Forward(u64), + /// Behind by this many ticks. + Back(u64), +} + +impl Delta { + /// From `last` to `value`, both below `wrap`. + pub(super) fn between(last: u64, value: u64, wrap: u64) -> Self { + let step = (value % wrap + wrap - last % wrap) % wrap; + if step <= wrap / 2 { + Self::Forward(step) + } else { + Self::Back(wrap - step) + } + } +} + /// The table_id of the first section a PSI packet starts, if it starts one. fn section_start(pkt: &[u8; TsPacket::SIZE]) -> Option { if pkt[1] & 0x40 == 0 { @@ -679,7 +701,7 @@ mod test { /// The feed from `from` to `to` ms on a clock reading `clock(t)` PCR ticks at `t` ms: the /// PAT and PMT every 100 ms, a PCR every 10 ms, a PES on each video PID every 40 ms /// stamped on the same clock, and a null packet closing each 10 ms. `keep` drops parts. - fn run(&mut self, from: u64, to: u64, clock: impl Fn(u64) -> u64, keep: impl Fn(u64, Part) -> bool) { + fn run(&mut self, from: u64, to: u64, clock: impl Fn(u64, Part) -> u64, keep: impl Fn(u64, Part) -> bool) { for t in (from..to).step_by(10) { if t % 100 == 0 { if keep(t, Part::Pat) { @@ -690,12 +712,12 @@ mod test { } } if keep(t, Part::Pcr) { - self.pcr(clock(t) % PCR_WRAP, false); + self.pcr(clock(t, Part::Pcr) % PCR_WRAP, false); } if t % 40 == 0 { for pid in [VIDEO, PEER] { if keep(t, Part::Pes(pid)) { - self.pes(pid, clock(t) / 300, false); + self.pes(pid, clock(t, Part::Pes(pid)) / 300, false); } } } @@ -704,7 +726,7 @@ mod test { } fn clean(&mut self, from: u64, to: u64) { - self.run(from, to, |t| t * MS, |_, _| true); + self.run(from, to, |t, _| t * MS, |_, _| true); } /// Where each null packet starts. @@ -762,15 +784,16 @@ mod test { } /// The PCR is graded on its values: a gap over 100 ms counts under both checks, a shorter - /// one under neither, the wrap is no gap, and a step back is a discontinuity only. + /// one under neither, the wrap is no gap, and a step back is a discontinuity only. The clock + /// steps back alone: the media stepping with it would restart the import instead. #[test] fn pcr_gaps_are_graded_on_their_values() { - let stats = |clock: &dyn Fn(u64) -> u64, keep: &dyn Fn(u64, Part) -> bool| { + let stats = |clock: &dyn Fn(u64, Part) -> u64, keep: &dyn Fn(u64, Part) -> bool| { let mut feed = Feed::default(); feed.run(0, 2_000, clock, keep); feed.stats() }; - let pcr = |clock: &dyn Fn(u64) -> u64, keep: &dyn Fn(u64, Part) -> bool| { + let pcr = |clock: &dyn Fn(u64, Part) -> u64, keep: &dyn Fn(u64, Part) -> bool| { let stats = stats(clock, keep); let health = stats.health(); assert_eq!( @@ -780,20 +803,27 @@ mod test { ); (stats.pcr_repetition_error, stats.pcr_discontinuity_indicator_error) }; - let steady = |t| t * MS; + let steady = |t: u64, _: Part| t * MS; assert_eq!(pcr(&steady, &gap(Part::Pcr, 150)), (1, 1), "a 150 ms gap"); assert_eq!(pcr(&steady, &gap(Part::Pcr, 80)), (0, 0), "an 80 ms gap"); assert_eq!(pcr(&steady, &gap(Part::Pcr, 100)), (0, 0), "a gap of exactly 100 ms"); - assert_eq!(pcr(&|t| PCR_WRAP - 1_000 * MS + t * MS, &all), (0, 0), "the wrap"); + assert_eq!(pcr(&|t, _| PCR_WRAP - 1_000 * MS + t * MS, &all), (0, 0), "the wrap"); assert_eq!( - pcr(&|t| if t < 1_000 { t * MS } else { (t - 500) * MS }, &all), + pcr( + &|t, part| if t < 1_000 || part != Part::Pcr { + t * MS + } else { + (t - 500) * MS + }, + &all + ), (0, 1), "an unsignalled step back" ); // An unsignalled 500 ms jump also stretches the program clock by 510 ms, which makes // the PAT and the PMT each just late. The 40 ms PES cadence absorbs it. - let jump = stats(&|t| if t < 1_000 { t * MS } else { (t + 500) * MS }, &all); + let jump = stats(&|t, _| if t < 1_000 { t * MS } else { (t + 500) * MS }, &all); assert_eq!( (jump.pcr_repetition_error, jump.pcr_discontinuity_indicator_error), (1, 1), @@ -809,7 +839,7 @@ mod test { let mut feed = Feed::default(); feed.clean(0, 1_000); feed.pcr(1_500 * MS, true); - feed.run(1_000, 2_000, |t| (t + 500) * MS, all); + feed.run(1_000, 2_000, |t, _| (t + 500) * MS, all); let stats = feed.stats(); assert_eq!(stats.health(), [0; 9], "{stats:?}"); } @@ -820,7 +850,7 @@ mod test { fn a_late_pat_counts_once_per_half_second() { for (len, count) in [(400, 0), (500, 0), (800, 1), (1_200, 2)] { let mut feed = Feed::default(); - feed.run(0, 3_000, |t| t * MS, gap(Part::Pat, len)); + feed.run(0, 3_000, |t, _| t * MS, gap(Part::Pat, len)); let stats = feed.stats(); assert_eq!(stats.pat_error, count, "PAT missing for {len} ms"); assert_eq!(stats.pmt_error, 0, "PAT missing for {len} ms"); @@ -843,7 +873,7 @@ mod test { #[test] fn a_late_pmt_counts_on_its_own_check() { let mut feed = Feed::default(); - feed.run(0, 3_000, |t| t * MS, gap(Part::Pmt, 800)); + feed.run(0, 3_000, |t, _| t * MS, gap(Part::Pmt, 800)); let stats = feed.stats(); assert_eq!((stats.pmt_error, stats.pat_error), (1, 0), "{stats:?}"); } @@ -852,7 +882,7 @@ mod test { #[test] fn a_pts_gap_counts_on_its_own_pid() { let mut feed = Feed::default(); - feed.run(0, 3_000, |t| t * MS, gap(Part::Pes(PEER), 1_000)); + feed.run(0, 3_000, |t, _| t * MS, gap(Part::Pes(PEER), 1_000)); let stats = feed.stats(); assert_eq!(stats.pts_error, 1, "{stats:?}"); assert_eq!(stats.streams[&PEER].pts_error, 1); @@ -869,7 +899,7 @@ mod test { feed.run( t, t + 100, - |t| t * MS, + |t, _| t * MS, |_, part| part != Part::Pmt && part != Part::Pes(PEER), ); } diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 1beab2c952f..78bbf8f0af3 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -134,6 +134,8 @@ pub struct Import { /// Frames the input [`decode`](Self::decode) reads. [`Programs`](super::Programs) runs one /// parser for every program instead and hands each its packets. demux: Demux, + /// Watches the input [`decode`](Self::decode) reads for a clock step that ends the instance. + steps: super::step::Steps, /// PIDs whose route changed while handling the current packet, for the parser to restart. restarts: Vec, /// Whether a PAT moved the PMT, or a PMT was handled, since @@ -270,6 +272,7 @@ impl Import { held: std::time::Duration::ZERO, held_edge: None, demux: Demux::new(supports_mpegts), + steps: Default::default(), restarts: Vec::new(), claims_changed: false, pmt: None, @@ -372,6 +375,14 @@ impl Import { /// Append `buf` to the internal scratch and demux every whole TS packet it /// now completes. The buffer is fully consumed; a trailing partial packet /// (< 188 bytes) is retained for the next call. + /// + /// A step of the source's clock onto another timeline ends the instance with + /// [`Restart`](super::Restart): the program clock and every timed stream stepping together + /// without `discontinuity_indicator`, forward by more than 10 s or back, or the streams + /// stepping back after a flagged break. Until the streams have spoken, which a cadenced one + /// does within 0.7 s, the input after a suspected step is held rather than published, up to + /// half a group's bytes. Every later call repeats the restart; [`finish`](Self::finish) ends + /// the instance before the step. pub fn decode(&mut self, data: &[u8]) -> anyhow::Result<()> { let mut demux = std::mem::take(&mut self.demux); let result = self.decode_with(&mut demux, data); @@ -386,6 +397,29 @@ impl Import { loop { for pid in self.restarts.drain(..) { demux.restart(pid); + self.steps.forget(pid); + } + let streams = &self.streams; + let timed = |pid: u16| { + Pid::new(pid) + .ok() + .and_then(|pid| streams.get(&pid)) + .is_some_and(|stream| !matches!(stream, Stream::Ignored)) + }; + let routes = super::step::Routes { + pcr_pid: self.pcr_pid.map(|pid| pid.as_u16()), + timed: &timed, + now: self.liveness.now(), + }; + match self.steps.judge(demux, &routes) { + super::step::Verdict::Route => {} + super::step::Verdict::Hold => break, + super::step::Verdict::Restart(step) => { + let remainder = demux.remainder(); + demux.drain(); + self.flush_batch()?; + return Err(super::Restart { step, remainder }.into()); + } } let Some(packet) = demux.next() else { break }; self.packet(&packet)?; @@ -1801,6 +1835,10 @@ impl Import { /// Flush any buffered PES and finish every track. pub fn finish(&mut self) -> anyhow::Result<()> { + // A step still being judged never completed: what it held routes as read. + if !self.finished && self.steps.settle() { + self.decode(&[])?; + } self.finished = true; self.demux.finish(); for pid in self.headers.keys().copied().collect::>() { @@ -1985,7 +2023,7 @@ fn section_framed(stream_type: u8, cuei: bool) -> bool { } // Match the raw TS accumulator: half a MoQ group bounds one program's staged input. -const STAGING_BYTES_MAX: usize = (moq_net::group::MAX_CACHE_BYTES / 2) as usize; +pub(super) const STAGING_BYTES_MAX: usize = (moq_net::group::MAX_CACHE_BYTES / 2) as usize; const STAGING_UNITS_MAX: usize = moq_net::group::MAX_GROUP_FRAMES / 2; #[derive(Default)] @@ -2829,7 +2867,7 @@ impl Liveness { /// /// 0x47 is the packet sync byte but also occurs freely in payload (TS has no byte /// stuffing), so a lone 0x47 isn't a boundary. -#[derive(Default)] +#[derive(Clone, Copy, Default)] pub(super) struct Framer { /// False until a candidate is confirmed by the next packet's sync byte; once true we /// stride 188 at a time and trust the per-packet check. Persists across calls so a @@ -6019,10 +6057,11 @@ pub(super) mod test { } // A looping publisher plays part of a file and wraps to the top (#2729). The wrap rewinds - // the audio PTS, which is an encoder restart, so the import ends and the caller publishes - // a new broadcast. Every frame published before it is still a whole AC-3 frame. + // the PCR and the audio PTS together, which is an encoder restart, so the instance ends at + // the wrap and the caller publishes the rest as a new broadcast. Every frame published + // before it is still a whole AC-3 frame. #[tokio::test(start_paused = true)] - async fn legacy_refuses_a_looping_file_wrap() { + async fn legacy_restarts_at_a_looping_file_wrap() { let data = include_bytes!("test_data/ac3.ts"); // Wrap mid-file on a packet boundary, so the TS layer stays aligned. @@ -6037,7 +6076,16 @@ pub(super) mod test { let err = import .decode(&bytes::BytesMut::from(&looped[..])) .expect_err("a loop wrap is a restart"); - assert!(is_rewind(&err), "{err:?}"); + let restart = err.downcast_ref::().expect("a restart"); + assert!( + matches!(restart.step, crate::container::ts::Step::Rewind { flagged: false, .. }), + "{restart:?}" + ); + assert!( + looped.ends_with(&restart.remainder) && restart.remainder.len() <= data.len(), + "the new instance starts at the wrap" + ); + import.finish().unwrap(); let frames = read_audio_frames(&consumer, &catalog).await; assert!(!frames.is_empty(), "the pass before the wrap published"); @@ -8037,7 +8085,11 @@ pub(super) mod test { .unwrap(); assert!(!import.published); assert!(!import.pending.is_empty()); + // The break routes once the stream shows it is no rewind. import.decode(clock_break_packet(PCR_PID).as_slice()).unwrap(); + import + .decode(mp2_pes(PEER_PID, 1, 20 * 90_000, [0xCC, 0xDD]).as_slice()) + .unwrap(); assert!(import.pending.is_empty(), "old PES survived the timebase reset"); } @@ -8280,11 +8332,12 @@ pub(super) mod test { } } - /// A backwards restart is the same signal, but a rewind: a new broadcast, so the import - /// ends. The unwrapper must not read the step back as the 33-bit field wrapping, which - /// would put the new timeline 26 hours out and publish it. + /// A backwards restart is the same signal, but a rewind: a new broadcast, so the instance + /// ends at the break, once every stream stepped back, and hands the input from there on. + /// The unwrapper must not read the step back as the 33-bit field wrapping, which would put + /// the new timeline 26 hours out and publish it. #[tokio::test(start_paused = true)] - async fn a_flagged_backwards_restart_is_refused() { + async fn a_flagged_backwards_restart_ends_the_instance() { let (_consumer, _catalog, mut import) = two_stream_import(); for pid in [PCR_PID, PEER_PID] { @@ -8292,13 +8345,24 @@ pub(super) mod test { .decode(mp2_pes(pid, 0, 45 * 90_000, [0xAA, 0xBB]).as_slice()) .unwrap(); } - import.decode(clock_break_packet(PCR_PID).as_slice()).unwrap(); - let err = [PCR_PID, PEER_PID] - .into_iter() - .try_for_each(|pid| import.decode(mp2_pes(pid, 1, 90_000, [0xCC, 0xDD]).as_slice())) - .and_then(|()| import.finish()) + let mut after = clock_break_packet(PCR_PID); + import.decode(&after).unwrap(); + after.extend(mp2_pes(PCR_PID, 1, 90_000, [0xCC, 0xDD])); + import.decode(&after[188..]).unwrap(); + after.extend(mp2_pes(PEER_PID, 1, 90_000, [0xCC, 0xDD])); + let err = import + .decode(&after[2 * 188..]) .expect_err("a restart below the live edge is a new broadcast"); - assert!(is_rewind(&err), "{err:?}"); + let restart = err.downcast_ref::().expect("a restart"); + assert_eq!( + restart.step, + crate::container::ts::Step::Rewind { + by: Duration::from_secs(44), + flagged: true + } + ); + assert_eq!(restart.remainder, after, "the new instance starts at the break"); + import.finish().unwrap(); } /// Adaptation stuffing leaves exactly these bytes in a TS payload. @@ -9790,17 +9854,26 @@ pub(super) mod test { } /// An encoder restart that declares the new clock on the PCR PID still restarts lower - /// than the live edge: a new broadcast, not a continuation. + /// than the live edge: a new broadcast, not a continuation, from the break on. #[tokio::test(start_paused = true)] - async fn h264_flagged_backward_restart_is_refused() { + async fn h264_flagged_backward_restart_ends_the_instance() { let mut mux = Mux { out: synth_pmt(&[(StreamType::H264, VIDEO)], false), ..Default::default() }; mux.gops(VIDEO, 45 * 90_000, 2); + let at = mux.out.len(); mux.out.extend_from_slice(&clock_break_packet(VIDEO)); mux.gops(VIDEO, 90_000, 2); - import_refused(&mux.out); + let Err(err) = import_all(&mux.out) else { + panic!("the import carried on past a rewind"); + }; + let restart = err.downcast_ref::().expect("a restart"); + assert!( + matches!(restart.step, crate::container::ts::Step::Rewind { flagged: true, .. }), + "{restart:?}" + ); + assert_eq!(restart.remainder, mux.out[at..]); } /// The same restart unflagged, as a looping playout wraps to the top of its file. @@ -10040,10 +10113,13 @@ pub(super) mod test { .decode(&synth_pmt(&[(StreamType::H264, PCR_PID)], false)) .unwrap(); let waiter = kio::Waiter::noop(); - // Each epoch carries 0.96 s of media, then a flagged PCR restarts the clock. + // Each epoch carries 0.96 s of media, then a flagged PCR restarts the clock, which the + // media leaps forward from: a rewind would end the instance. for epoch in 0..6 { for frame in 0..25 { - import.decode(&pes_packet(PCR_PID, 90_000 + frame * 3_600)).unwrap(); + import + .decode(&pes_packet(PCR_PID, 90_000 + epoch * 60 * 90_000 + frame * 3_600)) + .unwrap(); } import.decode(&flag_discontinuity(pcr_packet(PCR_PID, 0))).unwrap(); assert!( diff --git a/rs/moq-mux/src/container/ts/mod.rs b/rs/moq-mux/src/container/ts/mod.rs index ac1956e9509..c6b278b52cc 100644 --- a/rs/moq-mux/src/container/ts/mod.rs +++ b/rs/moq-mux/src/container/ts/mod.rs @@ -37,6 +37,7 @@ mod psi; mod release; mod schedule; mod si; +mod step; mod unclaimed; // The `mpegts` catalog section (per-track PID + descriptors plus verbatim carriage @@ -51,6 +52,7 @@ pub use import::*; pub use passthrough::Passthrough; pub use programs::Programs; pub use release::Release; +pub use step::{Restart, Step}; pub mod stats; @@ -76,3 +78,5 @@ mod release_test; mod revision_test; #[cfg(test)] mod routes_test; +#[cfg(test)] +mod step_test; diff --git a/rs/moq-mux/src/container/ts/step.rs b/rs/moq-mux/src/container/ts/step.rs new file mode 100644 index 00000000000..8e76b2cb79c --- /dev/null +++ b/rs/moq-mux/src/container/ts/step.rs @@ -0,0 +1,648 @@ +//! Whether the source's clock stepped onto another timeline, which ends the import's instance. +//! +//! A contribution switched between two encoders whose clocks were never locked steps every +//! timestamp at once without `discontinuity_indicator`. Splicing the two would put two sources +//! on one timeline, and a step back would break the rule that group starts never go +//! backwards within an instance, so the import ends its instance at the step and hands the +//! input from there to a caller that republishes it under a new epoch. +//! +//! The step is judged the way ffmpeg infers an unflagged one: the program clock and every +//! elementary stream that keeps a timestamp cadence step together, forward by more than +//! [`LEAP`] or back by any amount short of the 33-bit wrap. TR 101 290's 100 ms bound would +//! read SRT loss as a step. One clock stepping alone, such as a corrupt PCR or a stream whose +//! PTS jumps, is not a new source and is left to the import. After a flagged break, the +//! streams stepping back is a rewind; one stepping forward splices as before. + +use std::collections::{HashMap, HashSet}; +use std::time::Duration; + +use bytes::Bytes; +use mpeg2ts::ts::TsPacket; + +use super::demux::Demux; +use super::health::{Continuation, Continuity, Delta, PTS_LIMIT}; +use super::import::{Framer, Payload, STAGING_BYTES_MAX, adaptation_valid, discontinuity_indicator, payload, pcr}; +use super::mux_rate::PCR_WRAP; +use super::passthrough::PCR_HZ; +use super::psi::PesStart; + +/// A forward step beyond this is another timeline: ffmpeg's `dts_delta_threshold`. +const LEAP: Duration = Duration::from_secs(10); +const PTS_HZ: u64 = 90_000; +const PTS_WRAP: u64 = 1 << 33; + +/// The source's clock stepped onto another timeline, so the [`Import`](super::Import) ended its +/// instance there. +/// +/// What follows is another broadcast. A caller that carries on finishes this import and +/// publishes [`remainder`](Self::remainder) into a new one, as a new instance under a new epoch. +#[derive(Clone, Debug, thiserror::Error)] +#[non_exhaustive] +#[error("the source's clock {step}; what follows is a new broadcast")] +pub struct Restart { + /// How the clock stepped. + pub step: Step, + /// The input from the step on, which this import did not read. + pub remainder: Bytes, +} + +/// How the source's clock stepped, ending an import's instance. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub enum Step { + /// Back by `by`, after a `discontinuity_indicator` when `flagged`. + Rewind { + /// How far back. + by: Duration, + /// Whether the clock PID flagged the break. + flagged: bool, + }, + /// Forward by `by`, more than 10 s, without a `discontinuity_indicator`. + Leap { + /// How far forward. + by: Duration, + }, +} + +impl std::fmt::Display for Step { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Rewind { by, flagged: true } => write!(f, "went back {by:?} after a flagged break"), + Self::Rewind { by, flagged: false } => write!(f, "went back {by:?} without a discontinuity_indicator"), + Self::Leap { by } => write!(f, "jumped forward {by:?} without a discontinuity_indicator"), + } + } +} + +/// What the import routes: which packets carry the clocks being watched. +pub(super) struct Routes<'a> { + /// The PID the PMT names for the program clock. + pub pcr_pid: Option, + /// Whether a PID routes to an elementary stream whose PES are timestamped. + pub timed: &'a dyn Fn(u16) -> bool, + /// The program clock: PCR ticks summed since the first, as [`Import::now`](super::Import::now). + pub now: Option, +} + +/// What to do with the next packet. +pub(super) enum Verdict { + /// Route it. + Route, + /// Route nothing yet: a step is being judged on input still to come. + Hold, + /// The clock stepped at it: end the instance there. + Restart(Step), +} + +/// Watches the program clock and each timed PID, holding the input while a step is judged. +#[derive(Default)] +pub(super) struct Steps { + /// The clock PID, and the last PCR routed on it. + clock: Option<(u16, Option)>, + /// Each timed PID's last DTS, or PTS without one, as routed. + pids: HashMap, + /// Packets before this input offset were judged already, so they route without a lookahead. + cleared: u64, + /// The step being judged, from the next packet to route on. + ahead: Option, + /// The PES headers begun on the timed PIDs, as routed. + headers: Headers, +} + +/// A PID's last timestamp. +#[derive(Clone, Copy)] +struct Seen { + /// Raw 90 kHz. + ticks: u64, + /// The program clock when it was routed. + at: Option, +} + +/// Which way a clock stepped. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Way { + Back, + Leap, +} + +/// One clock reading against the last. +#[derive(Clone, Copy)] +enum Reading { + Inline, + Stepped(Way, Duration), +} + +impl Reading { + fn new(last: u64, value: u64, wrap: u64, hz: u64) -> Self { + let duration = |ticks: u64| { + Duration::from_nanos(u64::try_from(u128::from(ticks) * 1_000_000_000 / u128::from(hz)).unwrap_or(u64::MAX)) + }; + match Delta::between(last, value, wrap) { + Delta::Back(ticks) => Self::Stepped(Way::Back, duration(ticks)), + Delta::Forward(ticks) if duration(ticks) > LEAP => Self::Stepped(Way::Leap, duration(ticks)), + Delta::Forward(_) => Self::Inline, + } + } + + fn way(self) -> Option { + match self { + Self::Inline => None, + Self::Stepped(way, _) => Some(way), + } + } +} + +/// The clock readings one packet carries. +#[derive(Default)] +struct Marks { + /// A `discontinuity_indicator` on the clock PID. + flagged: bool, + /// A PCR on the clock PID. + pcr: Option, + /// The DTS, or PTS without one, of a PES header this packet completes on a timed PID. + pes: Option, + /// Its timed PID's chain after it, or `None` when it leaves the chain as it was. + chain: Option, +} + +impl Marks { + fn read(pkt: &[u8; TsPacket::SIZE], routes: &Routes, headers: &Headers) -> Self { + let mut marks = Self::default(); + let pid = (u16::from(pkt[1] & 0x1f) << 8) | u16::from(pkt[2]); + // A timed PID's chain follows every packet the import's does, the damaged ones too. + if (routes.timed)(pid) { + (marks.chain, marks.pes) = headers.read(pid, pkt); + } + // A packet the demodulator flagged corrupt, or whose adaptation field overruns itself, + // says nothing the import trusts. + if pkt[1] & 0x80 != 0 || !adaptation_valid(pkt) { + return marks; + } + if routes.pcr_pid == Some(pid) { + marks.flagged = discontinuity_indicator(pkt); + marks.pcr = pcr(pkt); + } + marks + } +} + +/// One timed PID's packets so far: whether they still chain, and the PES header begun on it. +#[derive(Clone, Default)] +struct Chain { + continuity: Continuity, + begun: Option>, +} + +/// The PES headers begun on each timed PID, so one split across packets still reads. +/// +/// Packets join under the import's continuity rules, so a retransmission or a loss cannot +/// splice a timestamp out of bytes the import never joins. +#[derive(Clone, Default)] +struct Headers(HashMap); + +impl Headers { + /// What `pkt` does on `pid` without retaining it: the chain after it, `None` when unchanged, + /// and the DTS, or PTS without one, of a header it completes. + fn read(&self, pid: u16, pkt: &[u8; TsPacket::SIZE]) -> (Option, Option) { + let mut chain = self.0.get(&pid).cloned().unwrap_or_default(); + match chain.continuity.observe(pkt) { + // The import ignores a retransmission entirely. + Continuation::Duplicate => return (None, None), + Continuation::Contiguous => {} + // The bytes joined so far are lost, as the import's partial PES is. + Continuation::Broken => chain.begun = None, + Continuation::Corrupt => { + chain.begun = None; + return (Some(chain), None); + } + } + // The import refuses a malformed adaptation field and cannot read a scrambled payload. + if !adaptation_valid(pkt) || pkt[3] & 0xc0 != 0 { + chain.begun = None; + return (Some(chain), None); + } + let Payload::Bytes(payload) = payload(pkt) else { + return (Some(chain), None); + }; + // A PES start abandons whatever header was begun before it, as the import's does. + let begun = chain.begun.take(); + let header = if pkt[1] & 0x40 != 0 { + payload.to_vec() + } else if let Some(mut begun) = begun { + begun.extend_from_slice(payload); + begun + } else { + return (Some(chain), None); + }; + // The optional fields are at most 255 bytes, so a held header stays small. + let ticks = match PesStart::header_len(&header) { + Ok(len) if header.len() >= len => match PesStart::parse(&header) { + Ok(PesStart { + pts: Some(pts), dts, .. + }) => Some(dts.unwrap_or(pts)), + _ => None, + }, + Ok(_) => { + chain.begun = Some(header); + None + } + Err(_) => None, + }; + (Some(chain), ticks) + } + + /// Retain the chain a packet left on `pid`. + fn apply(&mut self, pid: u16, chain: Option) { + if let Some(chain) = chain { + self.0.insert(pid, chain); + } + } +} + +/// A step being judged, from the packet that showed it on. +struct Ahead { + /// The input offset of that packet, where a new instance starts. + at: u64, + /// The way it stepped and by how much, or `None` for the clock PID's flagged break, which + /// ends the instance only if the streams step back. + stepped: Option<(Way, Duration)>, + /// Whether the PCR still has to step with it. + pcr: bool, + /// The PIDs that still have to step with it. + pids: HashSet, + /// The PIDs whose first PES since was judged. + heard: HashSet, + /// The PES headers begun on the timed PIDs, as read ahead. + headers: Headers, + /// How far back the first PID to step back went, which a flagged rewind reports. + back: Option, + /// The first PCR read since, which times how long the silent streams are waited for. + since: Option, + /// The next input offset to read, and the framer's state there. + scan: u64, + framer: Framer, + /// The step, once every clock confirmed it. + confirmed: Option, +} + +impl Steps { + /// Judge the next packet the import would route on `demux`. + pub(super) fn judge(&mut self, demux: &Demux, routes: &Routes) -> Verdict { + if self.clock.map(|(pid, _)| pid) != routes.pcr_pid { + self.clock = routes.pcr_pid.map(|pid| (pid, None)); + } + let (mut cursor, mut framer) = demux.cursor(); + let start = (cursor, framer); + let Some((at, pkt)) = demux.peek(&mut cursor, &mut framer) else { + return Verdict::Route; + }; + let pid = (u16::from(pkt[1] & 0x1f) << 8) | u16::from(pkt[2]); + let marks = Marks::read(pkt, routes, &self.headers); + + if self.ahead.is_none() && at >= self.cleared { + self.ahead = self.trigger(at, pid, &marks, routes, start); + } + let Some(ahead) = &mut self.ahead else { + self.observe(pid, marks, routes); + return Verdict::Route; + }; + if let Some(step) = ahead.confirmed { + return Verdict::Restart(step); + } + match ahead.scan(demux, routes, &self.clock, &self.pids) { + Some(Ok(step)) => { + ahead.confirmed = Some(step); + Verdict::Restart(step) + } + Some(Err(cleared)) => { + self.cleared = cleared; + self.ahead = None; + self.observe(pid, marks, routes); + Verdict::Route + } + None => Verdict::Hold, + } + } + + /// No input follows. A step still being judged never completed, so its input routes as + /// read; returns whether there is any. + pub(super) fn settle(&mut self) -> bool { + if self.ahead.as_ref().is_some_and(|ahead| ahead.confirmed.is_none()) { + self.ahead = None; + self.cleared = u64::MAX; + return true; + } + false + } + + /// `pid` changed route: its timestamps no longer continue the last. + pub(super) fn forget(&mut self, pid: u16) { + self.pids.remove(&pid); + self.headers.0.remove(&pid); + } + + /// Start judging a step at the packet at `at` if it shows one, with the clocks expected to + /// step with it. + fn trigger( + &self, + at: u64, + pid: u16, + marks: &Marks, + routes: &Routes, + (scan, framer): (u64, Framer), + ) -> Option { + let last_pcr = self.clock.and_then(|(_, last)| last); + let stepped = if marks.flagged { + None + } else { + // Without a clock to step with them, the streams alone say nothing of the source. + last_pcr?; + let pcr = marks + .pcr + .zip(last_pcr) + .map(|(value, last)| Reading::new(last, value, PCR_WRAP, PCR_HZ)); + let pes = marks + .pes + .zip(self.pids.get(&pid)) + .map(|(value, last)| Reading::new(last.ticks, value, PTS_WRAP, PTS_HZ)); + let stepped = [pcr, pes].into_iter().flatten().find_map(|reading| match reading { + Reading::Stepped(way, by) => Some((way, by)), + Reading::Inline => None, + }); + Some(stepped?) + }; + // A PID that kept TR 101 290's PTS cadence up to here is expected to step too. + let mut pids: HashSet = self + .pids + .iter() + .filter(|(pid, seen)| { + (routes.timed)(**pid) + && match (seen.at, routes.now) { + (_, None) => true, + (Some(at), Some(now)) => now.saturating_sub(at) <= PTS_LIMIT, + (None, Some(_)) => false, + } + }) + .map(|(pid, _)| *pid) + .collect(); + if marks.pes.is_some() && self.pids.contains_key(&pid) { + pids.insert(pid); + } + Some(Ahead { + at, + stepped, + pcr: !marks.flagged && last_pcr.is_some(), + pids, + heard: HashSet::new(), + headers: self.headers.clone(), + back: None, + since: None, + scan, + framer, + confirmed: None, + }) + } + + /// Record the readings of a packet routed. + fn observe(&mut self, pid: u16, marks: Marks, routes: &Routes) { + if let (Some(value), Some((_, last))) = (marks.pcr, &mut self.clock) { + *last = Some(value); + } + if let Some(ticks) = marks.pes { + self.pids.insert(pid, Seen { ticks, at: routes.now }); + } + self.headers.apply(pid, marks.chain); + } +} + +impl Ahead { + /// Read on until every clock expected to step has, `Some(Ok)`, or one did not, `Some(Err)` + /// with the offset judged through. `None` waits for more input. + fn scan( + &mut self, + demux: &Demux, + routes: &Routes, + clock: &Option<(u16, Option)>, + pids: &HashMap, + ) -> Option> { + let last_pcr = clock.and_then(|(_, last)| last); + loop { + // Held no longer than the import stages bytes awaiting a timestamp. + if self.scan - self.at > STAGING_BYTES_MAX as u64 { + return Some(Err(self.scan)); + } + let (at, pkt) = demux.peek(&mut self.scan, &mut self.framer)?; + let end = at + TsPacket::SIZE as u64; + let pid = (u16::from(pkt[1] & 0x1f) << 8) | u16::from(pkt[2]); + let marks = Marks::read(pkt, routes, &self.headers); + + if self.pcr + && let (Some(value), Some(last)) = (marks.pcr, last_pcr) + { + let reading = Reading::new(last, value, PCR_WRAP, PCR_HZ); + if reading.way() != self.stepped.map(|(way, _)| way) { + return Some(Err(end)); + } + self.pcr = false; + } + // A cadenced stream speaks within TR 101 290's PTS interval, so one still silent + // that long after the step dropped out with the old source: those that spoke decide. + if let Some(value) = marks.pcr { + let since = *self.since.get_or_insert(value); + if let Delta::Forward(elapsed) = Delta::between(since, value, PCR_WRAP) + && elapsed >= PTS_LIMIT + && !self.pcr + { + return Some(self.verdict(end, !self.heard.is_empty())); + } + } + // Every PID with a timestamp to step from has its say, once: a sparse one too, which + // only the cadenced ones are waited for. + if let (Some(value), Some(last)) = (marks.pes, pids.get(&pid)) + && self.heard.insert(pid) + { + self.pids.remove(&pid); + let reading = Reading::new(last.ticks, value, PTS_WRAP, PTS_HZ); + let wanted = match self.stepped { + Some((way, _)) => way, + None => Way::Back, + }; + let Reading::Stepped(way, by) = reading else { + return Some(Err(end)); + }; + if way != wanted { + return Some(Err(end)); + } + self.back.get_or_insert(by); + } + self.headers.apply(pid, marks.chain); + + // A clock alone, such as one corrupt PCR, needs a stream to step with it. + if !self.pcr && self.pids.is_empty() && (self.stepped.is_none() || !self.heard.is_empty()) { + return Some(self.verdict(end, true)); + } + } + } + + /// The step every clock heard from agreed on, `stepped` when that is enough to end the + /// instance, or the offset judged through when it is not. + fn verdict(&self, end: u64, stepped: bool) -> Result { + match self.stepped { + Some((Way::Back, by)) if stepped => Ok(Step::Rewind { by, flagged: false }), + Some((Way::Leap, by)) if stepped => Ok(Step::Leap { by }), + Some(_) => Err(end), + // A flagged break no PID has stepped back from is a splice. + None => self.back.map(|by| Step::Rewind { by, flagged: true }).ok_or(end), + } + } +} + +#[cfg(test)] +mod tests { + use super::super::fixture::{Flags, Mux}; + use super::*; + + const CLOCK: u16 = 0x1ff; + const VIDEO: u16 = 0x100; + /// 40 ms of program clock, in 27 MHz ticks. + const FRAME: u64 = PCR_HZ / 25; + /// The step the issue's contribution took, in 27 MHz ticks. + const SWITCH: u64 = 16_844 * PCR_HZ + PCR_HZ / 2; + + /// Judge each packet of `ts` and route it, as an import does, with `timed` the PIDs that + /// carry timestamped PES. Returns the first restart. + fn judge(ts: &[u8], timed: &[u16]) -> Option { + let mut demux = Demux::default(); + demux.push(ts); + let mut steps = Steps::default(); + let timed = |pid| timed.contains(&pid); + let routes = Routes { + pcr_pid: Some(CLOCK), + timed: &timed, + now: None, + }; + loop { + match steps.judge(&demux, &routes) { + Verdict::Restart(step) => return Some(step), + Verdict::Hold => return None, + Verdict::Route => demux.next()?, + }; + } + } + + /// A video PES at `pts` (90 kHz), with a short body. + fn pes_bytes(pts: u64) -> Vec { + let mut pes = vec![0, 0, 1, 0xe0, 0, 0, 0x80, 0x80, 5]; + pes.extend_from_slice(&[ + 0x21 | ((pts >> 29) & 0x0e) as u8, + (pts >> 22) as u8, + 0x01 | ((pts >> 14) & 0xfe) as u8, + (pts >> 7) as u8, + 0x01 | ((pts << 1) & 0xfe) as u8, + ]); + pes.extend_from_slice(&[0xaa; 32]); + pes + } + + /// One packet on `pid` carrying `payload`, padded with adaptation stuffing as a muxer does. + fn ts(cc: &mut u8, pid: u16, start: bool, payload: &[u8]) -> [u8; 188] { + let mut pkt = [0xff; 188]; + pkt[..3].copy_from_slice(&[0x47, u8::from(start) << 6 | (pid >> 8) as u8, pid as u8]); + pkt[3] = 0x30 | *cc; + pkt[4] = (183 - payload.len()) as u8; + pkt[5] = 0x00; + pkt[188 - payload.len()..].copy_from_slice(payload); + *cc = (*cc + 1) & 0x0f; + pkt + } + + /// One PES start on `pid` with `pts` (90 kHz), its header split at each of `splits` across + /// packets. The packet holding fragment `repeat` is sent twice, as ISO 13818-1 allows a + /// retransmission. + fn pes(cc: &mut u8, pid: u16, pts: u64, splits: &[usize], repeat: Option) -> Vec { + let pes = pes_bytes(pts); + let bounds: Vec = std::iter::once(0) + .chain(splits.iter().copied()) + .chain(std::iter::once(pes.len())) + .collect(); + let mut out = Vec::new(); + for (i, range) in bounds.windows(2).enumerate() { + let pkt = ts(cc, pid, i == 0, &pes[range[0]..range[1]]); + out.extend_from_slice(&pkt); + if repeat == Some(i) { + out.extend_from_slice(&pkt); + } + } + out + } + + /// A second of the clock and the video on it, from `start` (27 MHz). + fn second(mux: &mut Mux, cc: &mut u8, start: u64, video: bool) { + for frame in 0..25 { + let clock = (start + frame * FRAME) % PCR_WRAP; + mux.packet(CLOCK, Flags::pcr(clock)); + if video { + mux.out + .extend(pes(cc, VIDEO, (clock / 300 + 9_000) % PTS_WRAP, &[7], None)); + } + } + } + + /// A program clock stepping with no stream to step with it, such as one corrupt PCR in a + /// program whose streams are idle, is not another source. + #[test] + fn a_clock_alone_does_not_restart() { + let (mut mux, mut cc) = (Mux::default(), 0); + second(&mut mux, &mut cc, PCR_HZ, false); + second(&mut mux, &mut cc, PCR_WRAP + 2 * PCR_HZ - SWITCH, false); + assert_eq!(judge(&mux.out, &[VIDEO]), None); + } + + /// A retransmitted fragment of a split PES header is ignored, as the import ignores it, so + /// it cannot read as the stream stepping with a lone PCR glitch. + #[test] + fn a_repeated_header_fragment_does_not_step() { + let (mut mux, mut cc) = (Mux::default(), 0); + mux.packet(CLOCK, Flags::pcr(PCR_HZ)); + mux.out.extend(pes(&mut cc, VIDEO, 90_000, &[7], None)); + // One corrupt PCR, then the next frame on time, its PTS 93,600 split after its first byte + // and the next two. Joining the repeat would read `21 00 05 00 05`, PTS 65,538: back. + mux.packet(CLOCK, Flags::pcr(PCR_WRAP + PCR_HZ + FRAME - SWITCH)); + mux.out.extend(pes(&mut cc, VIDEO, 93_600, &[10, 12], Some(1))); + second(&mut mux, &mut cc, PCR_HZ + 2 * FRAME, true); + assert_eq!(judge(&mux.out, &[VIDEO]), None); + } + + /// A PES start abandons the header begun before it, as the import's does, so the new PES's + /// body cannot complete the old header and read as a stream stepping with a lone PCR glitch. + #[test] + fn a_pes_start_abandons_the_header_begun() { + let (mut mux, mut cc) = (Mux::default(), 0); + mux.packet(CLOCK, Flags::pcr(PCR_HZ)); + mux.out.extend(pes(&mut cc, VIDEO, 90_000, &[7], None)); + // A PES start cut short after nine bytes, then a whole one at PTS 93,600. + mux.out.extend(ts(&mut cc, VIDEO, true, &pes_bytes(90_000)[..9])); + mux.out.extend(ts(&mut cc, VIDEO, true, &pes_bytes(93_600))); + // One corrupt PCR, then a body that, joined to the cut-short header, reads PTS 65,538: back. + mux.packet(CLOCK, Flags::pcr(PCR_WRAP + PCR_HZ + FRAME - SWITCH)); + mux.out + .extend(ts(&mut cc, VIDEO, false, &[0x21, 0x00, 0x05, 0x00, 0x05])); + second(&mut mux, &mut cc, PCR_HZ + 2 * FRAME, true); + assert_eq!(judge(&mux.out, &[VIDEO]), None); + } + + /// A PES header split across packets still times its stream, so the switch is seen. + #[test] + fn a_split_pes_header_steps_with_the_clock() { + let (mut mux, mut cc) = (Mux::default(), 0); + second(&mut mux, &mut cc, PCR_HZ, true); + second(&mut mux, &mut cc, PCR_WRAP + 2 * PCR_HZ - SWITCH, true); + let Some(Step::Rewind { by, flagged: false }) = judge(&mux.out, &[VIDEO]) else { + panic!("the switch was missed"); + }; + assert!( + by.abs_diff(Duration::from_millis(16_844_460)) < Duration::from_millis(100), + "{by:?}" + ); + } +} diff --git a/rs/moq-mux/src/container/ts/step_test.rs b/rs/moq-mux/src/container/ts/step_test.rs new file mode 100644 index 00000000000..287677312d1 --- /dev/null +++ b/rs/moq-mux/src/container/ts/step_test.rs @@ -0,0 +1,255 @@ +//! A step of the source's clock onto another timeline ends the import's instance. +//! +//! The stimulus is a real capture played twice, the second pass on another clock: the whole +//! multiplex steps at one packet, with every continuity counter broken once and no +//! `discontinuity_indicator`, as a changeover between two unsynchronised encoders does. + +use std::time::Duration; + +use super::{Ext, Import, Restart, Step}; + +/// 5 s of H.264, AAC and SCTE-35, with a PCR. +const CLIP: &[u8] = include_bytes!("test_data/scte35/bbb5s.ts"); +const HZ: i64 = 90_000; +const WRAP: i64 = 1 << 33; +/// The step the issue's contribution took: two encoders started about 4 h 40 m apart. +const SWITCH: i64 = 16_844 * HZ + HZ / 2; + +/// Which clock a timestamp belongs to. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Clock { + Pcr, + Pes(u16), +} + +fn pid(pkt: &[u8]) -> u16 { + (u16::from(pkt[1] & 0x1f) << 8) | u16::from(pkt[2]) +} + +/// The PCR base (90 kHz) a packet carries, and where it sits. +fn pcr_at(pkt: &[u8]) -> Option<(usize, i64)> { + if pkt[3] & 0x20 == 0 || pkt[4] < 7 || pkt[5] & 0x10 == 0 { + return None; + } + let base = (i64::from(pkt[6]) << 25) + | (i64::from(pkt[7]) << 17) + | (i64::from(pkt[8]) << 9) + | (i64::from(pkt[9]) << 1) + | (i64::from(pkt[10]) >> 7); + Some((6, base)) +} + +/// Where a PES start's PTS and DTS fields sit in its packet, when its header is whole. +fn pes_fields(pkt: &[u8]) -> Vec { + if pkt[1] & 0x40 == 0 || pkt[3] & 0x10 == 0 { + return Vec::new(); + } + let start = if pkt[3] & 0x20 != 0 { 5 + usize::from(pkt[4]) } else { 4 }; + let Some(pes) = pkt.get(start..start + 19) else { + return Vec::new(); + }; + if pes[..3] != [0, 0, 1] { + return Vec::new(); + } + match pes[7] >> 6 { + 0b10 => vec![start + 9], + 0b11 => vec![start + 9, start + 14], + _ => Vec::new(), + } +} + +fn read_timestamp(field: &[u8]) -> i64 { + (i64::from(field[0] >> 1 & 0x07) << 30) + | (i64::from(field[1]) << 22) + | (i64::from(field[2] >> 1) << 15) + | (i64::from(field[3]) << 7) + | i64::from(field[4] >> 1) +} + +fn write_timestamp(field: &mut [u8], t: i64) { + field[0] = (field[0] & 0xf1) | ((t >> 29) as u8 & 0x0e); + field[1] = (t >> 22) as u8; + field[2] = ((t >> 14) as u8 & 0xfe) | 1; + field[3] = (t >> 7) as u8; + field[4] = ((t << 1) as u8 & 0xfe) | 1; +} + +/// `ts` with each PCR and PES timestamp moved by `by(clock)` 90 kHz ticks, modulo the wrap. +fn shift(ts: &[u8], by: impl Fn(Clock) -> i64) -> Vec { + let mut out = ts.to_vec(); + for pkt in out.as_chunks_mut::<188>().0 { + let pid = pid(pkt); + if let Some((at, base)) = pcr_at(pkt) { + let base = (base + by(Clock::Pcr)).rem_euclid(WRAP); + pkt[at] = (base >> 25) as u8; + pkt[at + 1] = (base >> 17) as u8; + pkt[at + 2] = (base >> 9) as u8; + pkt[at + 3] = (base >> 1) as u8; + pkt[at + 4] = (pkt[at + 4] & 0x7f) | ((base as u8 & 1) << 7); + } + for at in pes_fields(pkt) { + let t = (read_timestamp(&pkt[at..at + 5]) + by(Clock::Pes(pid))).rem_euclid(WRAP); + write_timestamp(&mut pkt[at..at + 5], t); + } + } + out +} + +/// The program clock the clip spans, first PCR to last, in 90 kHz ticks. +fn span(ts: &[u8]) -> i64 { + let pcrs: Vec = ts + .as_chunks::<188>() + .0 + .iter() + .filter_map(|pkt| pcr_at(pkt).map(|(_, base)| base)) + .collect(); + pcrs.last().unwrap() - pcrs.first().unwrap() +} + +/// The clip, then the clip again on a clock `gap` past where the first pass ended, each clock +/// moved by `by` on top. Returns where the second pass starts. +fn replay(gap: i64, by: impl Fn(Clock) -> i64) -> (Vec, usize) { + let mut data = CLIP.to_vec(); + let second = span(CLIP) + gap; + data.extend(shift(CLIP, |clock| second + by(clock))); + (data, CLIP.len()) +} + +/// The importer, its catalog kept alive, fed `data` in `chunk`-byte pieces until it fails. +/// Returns the failure, if any, and the input it did not take: a restart's remainder followed +/// by the pieces never fed. +fn feed(data: &[u8], chunk: usize) -> (Option, Vec) { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = crate::catalog::Producer::new( + &mut broadcast, + crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::::default()), + ) + .unwrap(); + let mut import = Import::new(broadcast, catalog.reserve()); + for (index, piece) in data.chunks(chunk).enumerate() { + if let Err(err) = import.decode(piece) { + let Some(restart) = err.downcast_ref::() else { + return (Some(err), Vec::new()); + }; + let mut rest = restart.remainder.to_vec(); + rest.extend(data.chunks(chunk).skip(index + 1).flatten()); + // The instance before the step ends cleanly. + import.finish().unwrap(); + return (Some(err), rest); + } + } + (import.finish().err(), Vec::new()) +} + +/// The input from the first packet of `pass` that carries a timestamp. +fn first_timed(pass: &[u8]) -> &[u8] { + let at = pass + .as_chunks::<188>() + .0 + .iter() + .position(|pkt| pcr_at(pkt).is_some() || !pes_fields(pkt).is_empty()) + .unwrap(); + &pass[at * 188..] +} + +/// A switch to an encoder whose clock started hours away restarts the import either way, at +/// the switch, however the input is split. +#[test] +fn an_unflagged_switch_restarts_either_way() { + for (gap, expected) in [(-SWITCH, "rewind"), (SWITCH, "leap")] { + let (data, second) = replay(gap, |_| 0); + for chunk in [data.len(), 188 * 7, 1_000] { + let (err, rest) = feed(&data, chunk); + let err = err.expect("the switch ends the instance"); + let restart = err.downcast_ref::().expect("a restart"); + let by = match (restart.step, expected) { + (Step::Rewind { by, flagged: false }, "rewind") | (Step::Leap { by }, "leap") => by, + (step, _) => panic!("{expected}: {step:?}"), + }; + let switch = Duration::from_micros((SWITCH * 1_000_000 / HZ) as u64); + assert!(by.abs_diff(switch) < Duration::from_secs(1), "{expected} by {by:?}"); + + // The new instance takes everything from the second pass's first timestamp, which + // is where the old one stopped reading: none of it is published twice or lost. + assert!(data[second..].ends_with(&rest), "{expected} {chunk}"); + assert!(rest.len() >= first_timed(&data[second..]).len(), "{expected} {chunk}"); + + // What follows imports as a broadcast of its own. + let (err, _) = feed(&rest, chunk); + assert!(err.is_none(), "{expected}: {err:?}"); + } + } +} + +/// A clock that steps forward by less than the bound, such as across lost packets, carries +/// the same instance on. +#[test] +fn a_step_under_the_bound_carries_on() { + let (data, _) = replay(9 * HZ, |_| 0); + let (err, _) = feed(&data, 188 * 7); + assert!(err.is_none(), "{err:?}"); + + // Two seconds of the clip lost in the middle. + let packets = CLIP.len() / 188; + let mut lossy = CLIP[..packets / 3 * 188].to_vec(); + lossy.extend_from_slice(&CLIP[packets * 2 / 3 * 188..]); + let (err, _) = feed(&lossy, 188 * 7); + assert!(err.is_none(), "{err:?}"); +} + +/// A stream the new source does not carry is waited for only as long as a cadenced stream may +/// stay silent: the clocks that stepped decide, and the instance does not stall. +#[test] +fn a_stream_the_new_source_lacks_is_not_waited_for() { + let (mut data, second) = replay(-SWITCH, |_| 0); + let audio = audio_pid(CLIP); + let without: Vec = data[second..] + .as_chunks::<188>() + .0 + .iter() + .filter(|pkt| pid(&pkt[..]) != audio) + .flatten() + .copied() + .collect(); + data.truncate(second); + data.extend(&without); + + let (err, rest) = feed(&data, 188 * 7); + let err = err.expect("the switch ends the instance"); + let restart = err.downcast_ref::().expect("a restart"); + assert!( + matches!(restart.step, Step::Rewind { flagged: false, .. }), + "{restart:?}" + ); + assert!(without.ends_with(&rest) && rest.len() >= first_timed(&without).len()); +} + +/// One clock stepping without the rest is not another source, so the import judges it as +/// before: a PCR alone, or the streams with a stream left behind on the old timeline. +#[test] +fn a_clock_stepping_alone_is_no_switch() { + let (data, _) = replay(HZ, |clock| if clock == Clock::Pcr { -SWITCH } else { 0 }); + let (err, _) = feed(&data, 188 * 7); + assert!(err.as_ref().is_none_or(|err| !err.is::()), "{err:?}"); + + // The audio carries on a second after the first pass while the rest rewinds. + let audio = audio_pid(CLIP); + let (data, _) = replay( + -SWITCH, + |clock| if clock == Clock::Pes(audio) { SWITCH + HZ } else { 0 }, + ); + let (err, _) = feed(&data, 188 * 7); + assert!(err.as_ref().is_none_or(|err| !err.is::()), "{err:?}"); +} + +/// The PID of the clip's audio: the PES that are not video. +fn audio_pid(ts: &[u8]) -> u16 { + ts.as_chunks::<188>() + .0 + .iter() + .find_map(|pkt| { + let at = *pes_fields(pkt).first()?; + (!(0xe0..=0xef).contains(&pkt[at - 9 + 3])).then(|| pid(pkt)) + }) + .unwrap() +} diff --git a/rs/moq-srt/src/ts.rs b/rs/moq-srt/src/ts.rs index 79e38e0cc0b..2d496d0a47b 100644 --- a/rs/moq-srt/src/ts.rs +++ b/rs/moq-srt/src/ts.rs @@ -36,6 +36,11 @@ pub enum Program { /// next call (the same pattern `moq-cli import ... stdin ts` uses against stdin). /// [`Self::finish`] keeps the finished broadcasts up for a linger; dropping the /// publisher ends them and unannounces their paths. +/// +/// A step of the source's clock onto another timeline ([`ts::Restart`]), such as a +/// changeover between two encoders whose clocks were never locked, finishes the broadcast +/// and republishes the rest of the input on the same path under a new epoch, on the same +/// connection. [`Program::All`] still ends the ingest there. pub struct Publisher { importer: Importer, // The importer's per-stream counters, logged as `moq import ts` logs them, under a @@ -46,21 +51,69 @@ pub struct Publisher { } enum Importer { - One { - // TS carries undecoded elementary streams (SCTE-35, teletext, DVB AC-3, ...) - // verbatim, so the importer uses the `mpegts` catalog extension rather than the - // media-only `()`, which would route those PIDs to `Stream::Ignored` and drop them. - import: Box>, - // Finished at a clean end, so the finished broadcast keeps serving the catalog. - catalog: Box>, - // Controls publication while the importer fills it, including finish or close at EOF. - broadcast: moq_net::origin::Publication, - }, + One(Box), /// Each program's broadcast is created once the first PAT names it, and announced once its PMT /// declares its tracks. All(Box), } +/// One program's broadcast, and what the next instance publishes with when the source's +/// clock steps. +struct One { + instance: Instance, + origin: origin::Producer, + path: String, + config: moq_mux::catalog::Config, + program: Option, +} + +/// One program's broadcast, under one epoch. +struct Instance { + // TS carries undecoded elementary streams (SCTE-35, teletext, DVB AC-3, ...) + // verbatim, so the importer uses the `mpegts` catalog extension rather than the + // media-only `()`, which would route those PIDs to `Stream::Ignored` and drop them. + import: ts::Import, + // Finished at a clean end, so the finished broadcast keeps serving the catalog. + catalog: moq_mux::catalog::Producer, + // Controls publication while the importer fills it, including finish or close at EOF. + broadcast: moq_net::origin::Publication, +} + +impl Instance { + /// Publish `path` under a new epoch, importing `program` of the input, or its only one. + /// + /// Each instance is its own publisher, so an encoder reconnecting under the same stream id, + /// or a source whose clock stepped, replaces the stale broadcast instead of resuming into it. + fn publish( + origin: &origin::Producer, + path: &str, + config: moq_mux::catalog::Config, + program: Option, + ) -> Result<(Self, moq_net::Epoch)> { + let epoch = moq_net::Epoch::mint(); + let broadcast = origin.publish(path, moq_net::origin::Route::default().with_epoch(epoch.clone()))?; + let config = config.with_catalog(moq_mux::catalog::hang::Catalog::::default()); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast.broadcast().clone(), config)?; + let mut import = ts::Import::new(broadcast.broadcast().clone(), catalog.reserve()); + if let Some(program) = program { + import = import.with_program(program); + } + let instance = Self { + import, + catalog, + broadcast, + }; + Ok((instance, epoch)) + } + + /// Flush the buffered media and finish every track, the catalog included. + fn finish(&mut self) -> Result<()> { + self.import.finish().map_err(moq_mux::Error::from)?; + self.catalog.finish()?; + Ok(()) + } +} + impl Publisher { /// Wire up the TS importer and catalog for `path` on `origin`, announcing the broadcast /// now, or each program's once its PMT declares its tracks for [`Program::All`]. @@ -74,33 +127,32 @@ impl Publisher { config: moq_mux::catalog::Config, program: Option, ) -> Result { - // Each connection is its own publisher instance, so an encoder reconnecting - // under the same stream id replaces the stale broadcast instead of resuming into - // it. `ts::Programs` mints one per program likewise. - let mut epoch = None; let importer = match program { - Some(Program::All) => Importer::All(Box::new(ts::Programs::new( - origin.clone(), - moq_net::Path::new(path).map_err(moq_net::Error::from)?, - config, - ))), + // `ts::Programs` mints an epoch per program. + Some(Program::All) => { + tracing::info!(%path, ?program, "publishing ingest broadcasts"); + Importer::All(Box::new(ts::Programs::new( + origin.clone(), + moq_net::Path::new(path).map_err(moq_net::Error::from)?, + config, + ))) + } Some(Program::One(_)) | None => { - let route = moq_net::origin::Route::default().with_epoch(epoch.insert(moq_net::Epoch::mint()).clone()); - let broadcast = origin.publish(path, route)?; - let config = config.with_catalog(moq_mux::catalog::hang::Catalog::::default()); - let catalog = moq_mux::catalog::Producer::new(&mut broadcast.broadcast().clone(), config)?; - let mut import = ts::Import::new(broadcast.broadcast().clone(), catalog.reserve()); - if let Some(Program::One(program)) = program { - import = import.with_program(program); - } - Importer::One { - import: Box::new(import), - catalog: Box::new(catalog), - broadcast, - } + let program = match program { + Some(Program::One(program)) => Some(program), + _ => None, + }; + let (instance, epoch) = Instance::publish(origin, path, config.clone(), program)?; + tracing::info!(%path, ?program, %epoch, "publishing ingest broadcast"); + Importer::One(Box::new(One { + instance, + origin: origin.clone(), + path: path.to_string(), + config, + program, + })) } }; - tracing::info!(%path, ?program, epoch = epoch.as_ref().map(tracing::field::display), "publishing ingest broadcast"); Ok(Self { importer, @@ -118,10 +170,31 @@ impl Publisher { /// per-stream counters. pub fn feed(&mut self, data: Bytes) -> Result<()> { match &mut self.importer { - Importer::One { import, .. } => import.decode(&data), - Importer::All(programs) => programs.decode(&data), + Importer::One(one) => { + let One { + instance, + origin, + path, + config, + program, + } = &mut **one; + let mut data = data; + while let Err(err) = instance.import.decode(&data) { + let restart = err.downcast::().map_err(moq_mux::Error::from)?; + // The instance before the step finishes; the rest of the input republishes + // the path under a new epoch. + instance.finish()?; + self.span.in_scope(|| self.log.finish(&instance.import.stats())); + let (next, epoch) = Instance::publish(origin, path, config.clone(), *program)?; + tracing::warn!(%path, step = %restart.step, %epoch, "the source's clock stepped; republishing"); + // The finished broadcast ends with its publication. + *instance = next; + self.log = ts::stats::Log::default(); + data = restart.remainder; + } + } + Importer::All(programs) => programs.decode(&data).map_err(moq_mux::Error::from)?, } - .map_err(moq_mux::Error::from)?; if self.sampled.elapsed() >= ts::stats::Log::INTERVAL { self.sampled = moq_net::time::Instant::now(); let _span = self.span.enter(); @@ -132,7 +205,7 @@ impl Publisher { fn stats(&self) -> ts::stats::Snapshot { match &self.importer { - Importer::One { import, .. } => import.stats(), + Importer::One(one) => one.instance.import.stats(), Importer::All(programs) => programs.stats(), } } @@ -141,14 +214,9 @@ impl Publisher { /// finished broadcast up for `linger`. pub fn finish(mut self, linger: Duration) -> Result> { let finished = match &mut self.importer { - Importer::One { - import, - catalog, - broadcast, - } => { - import.finish().map_err(moq_mux::Error::from)?; - catalog.finish()?; - vec![broadcast.broadcast().clone()] + Importer::One(one) => { + one.instance.finish()?; + vec![one.instance.broadcast.broadcast().clone()] } Importer::All(programs) => programs.finish().map_err(moq_mux::Error::from)?, }; @@ -163,7 +231,8 @@ impl Publisher { /// Consumes the publisher and closes the broadcasts. pub fn abort(self, err: moq_net::Error) { match self.importer { - Importer::One { import, broadcast, .. } => { + Importer::One(one) => { + let Instance { import, broadcast, .. } = one.instance; import.abort(err); broadcast.close(); } @@ -359,21 +428,106 @@ mod tests { out } - /// Publish `ts` on `path` one SRT payload (7 packets) at a time, each delivered when the - /// program clock says it is due. - async fn ingest(origin: &moq_net::origin::Producer, path: &str, ts: &[u8]) { + /// Publish each of `passes` on `path` through one connection, one SRT payload (7 packets) at + /// a time, each delivered when its pass's program clock says it is due. A pass starts as the + /// one before ends, whatever its clock reads. + async fn ingest(origin: &moq_net::origin::Producer, path: &str, passes: &[&[u8]]) { let mut publisher = Publisher::new(origin, path, Default::default(), None).unwrap(); - let mut clock = Duration::ZERO; - for payload in timed(ts).collect::>().chunks(7) { - let (due, _) = payload[0]; - tokio::time::advance(due.saturating_sub(clock)).await; - clock = clock.max(due); - let bytes: Vec = payload.iter().flat_map(|(_, pkt)| pkt.iter().copied()).collect(); - publisher.feed(bytes.into()).unwrap(); + let (mut clock, mut start) = (Duration::ZERO, Duration::ZERO); + for ts in passes { + for payload in timed(ts).collect::>().chunks(7) { + let due = start + payload[0].0; + tokio::time::advance(due.saturating_sub(clock)).await; + clock = clock.max(due); + let bytes: Vec = payload.iter().flat_map(|(_, pkt)| pkt.iter().copied()).collect(); + publisher.feed(bytes.into()).unwrap(); + } + start = clock + Duration::from_millis(40); } drop(publisher.finish(Duration::ZERO).unwrap()); } + /// `ts` with every PCR, PTS and DTS moved by `by` 90 kHz ticks, modulo the 33-bit wrap, + /// and no `discontinuity_indicator`: the same programme from another encoder's clock. + fn shift(ts: &[u8], by: i64) -> Vec { + const WRAP: i64 = 1 << 33; + let mut out = ts.to_vec(); + for pkt in out.as_chunks_mut::<188>().0 { + if let Some(pcr) = pcr(pkt) { + let base = (pcr as i64 / 300 + by).rem_euclid(WRAP); + pkt[6] = (base >> 25) as u8; + pkt[7] = (base >> 17) as u8; + pkt[8] = (base >> 9) as u8; + pkt[9] = (base >> 1) as u8; + pkt[10] = (pkt[10] & 0x7f) | ((base as u8 & 1) << 7); + } + // A PES start's PTS, and DTS after it, when its header fits the packet. + if pkt[1] & 0x40 == 0 || pkt[3] & 0x10 == 0 { + continue; + } + let start = if pkt[3] & 0x20 != 0 { 5 + usize::from(pkt[4]) } else { 4 }; + if pkt.len() < start + 19 || pkt[start..start + 3] != [0, 0, 1] { + continue; + } + let fields = match pkt[start + 7] >> 6 { + 0b10 => 1, + 0b11 => 2, + _ => 0, + }; + for field in 0..fields { + let f = &mut pkt[start + 9 + 5 * field..start + 14 + 5 * field]; + let t = (i64::from(f[0] >> 1 & 0x07) << 30) + | (i64::from(f[1]) << 22) + | (i64::from(f[2] >> 1) << 15) + | (i64::from(f[3]) << 7) + | i64::from(f[4] >> 1); + let t = (t + by).rem_euclid(WRAP); + f[0] = (f[0] & 0xf1) | ((t >> 29) as u8 & 0x0e); + f[1] = (t >> 22) as u8; + f[2] = ((t >> 14) as u8 & 0xfe) | 1; + f[3] = (t >> 7) as u8; + f[4] = ((t << 1) as u8 & 0xfe) | 1; + } + } + out + } + + /// A source switched between two encoders whose clocks started hours apart, without a + /// `discontinuity_indicator`, either way: the ingest republishes under a new epoch on the + /// same connection, and an egress that stitches replacements carries on into it. + #[tokio::test(start_paused = true)] + async fn a_clock_step_republishes_under_a_new_epoch() { + // The issue's step: two encoders started about 4 h 40 m apart. + const SWITCH: i64 = 16_844 * 90_000 + 45_000; + for step in [-SWITCH, SWITCH] { + let origin = produce_origin(); + let options = Options { + latency: Duration::from_secs(1), + follow: Duration::from_secs(30), + stitch: true, + }; + let consumer = origin.consume(); + let egress = tokio::spawn(async move { + let mut subscriber = Subscriber::new(&consumer, "ingest", options).await.unwrap().unwrap(); + // The frames each program clock generation carried. + let mut generations = std::collections::BTreeMap::::new(); + while let Some(frame) = subscriber.next().await.expect("the egress follows the restart") { + if !frame.payload.is_empty() { + *generations.entry(subscriber.discontinuity()).or_default() += 1; + } + } + generations + }); + let second = shift(BBB5S, step); + ingest(&origin, "ingest", &[BBB5S, &second]).await; + let generations = egress.await.unwrap(); + assert!( + generations.len() >= 2 && generations.values().all(|&frames| frames > 0), + "{step}: the egress switched programs: {generations:?}" + ); + } + } + /// One payload-only TS packet carrying a complete PSI section (PUSI + pointer_field /// 0), padded to 188 with stuffing. fn psi_packet(pid: u16, section: &[u8]) -> Vec { @@ -623,9 +777,9 @@ mod tests { async fn publisher_reports_a_silent_pid() { const VIDEO: u16 = 0x100; let origin = produce_origin(); - ingest(&origin, "intact", BBB5S).await; + ingest(&origin, "intact", &[BBB5S]).await; let stimulus = suppress(BBB5S, VIDEO, Duration::from_millis(1500), Duration::from_millis(3500)); - ingest(&origin, "silent", &stimulus).await; + ingest(&origin, "silent", &[&stimulus]).await; logs_assert(|lines: &[&str]| { let stopped: Vec<_> = lines