diff --git a/drafts/draft-lcurley-moq-mpegts.md b/drafts/draft-lcurley-moq-mpegts.md index f070598e9c2..e5a1c299197 100644 --- a/drafts/draft-lcurley-moq-mpegts.md +++ b/drafts/draft-lcurley-moq-mpegts.md @@ -159,6 +159,7 @@ type Program = { `transportStreamId`, `programNumber` (the DVB service id), and `pmtPid` come from the PAT. They are the only service-layer fields this document parses; everything else about the program stays inside the opaque `si` tables. +A PAT that moves the PMT to another PID changes `pmtPid` in place. A publisher MUST include `program` when the broadcast came from a transport stream, and omit it otherwise. A publisher MUST NOT change `transportStreamId` or `programNumber` in place: the carried `si` describes that service, so a source whose PAT names another ends the broadcast. @@ -373,7 +374,7 @@ A consumer rebuilding a transport stream: - MUST place each track on its recorded `pid`, and MUST assign an unused PID to a track with none. - MUST put the PMT on `program.pmtPid` and build the PAT and PMT from `program`. When it carries a NIT (the `si` entry for `table_id` 0x40 on PID 0x0010), the PAT MUST also list program 0 with that `network_PID`, so a receiver can find it. - MUST choose a PCR PID and emit PCRs on it at the program clock's repetition interval. With no media stream it MUST use an unused PID with adaptation-only PCR packets and no PMT elementary-stream entry. It MUST NOT refuse a program for lacking video. -- MUST follow the catalog as it changes: when a track joins or leaves `tracks` or `programDescriptors` changes, it MUST emit a PMT at the next `version_number` (modulo 32) before any packet of a stream it adds, and MUST NOT emit a stream's packets after a PMT that no longer lists it. Each version it emits MUST differ from the one before, and no older version may follow a newer one. The PCR PID MUST NOT move until the stream it rides ends. +- MUST follow the catalog as it changes: when a track joins or leaves `tracks` or `programDescriptors` changes, it MUST emit a PMT at the next `version_number` (modulo 32) before any packet of a stream it adds, and MUST NOT emit a stream's packets after a PMT that no longer lists it. Each version it emits MUST differ from the one before, and no older version may follow a newer one. The PCR PID MUST NOT move until the stream it rides ends. When `program.pmtPid` changes, it MUST emit a PAT at the next `version_number` naming the new PID, and MUST NOT emit the PMT on the new PID before that PAT or on the old one after it. - MUST re-emit each track's `descriptors` as its ES-level descriptors, and `programDescriptors` as the PMT's `program_info`. - MUST re-emit each `si` entry's sections byte-for-byte on that entry's PID, reading them from its track ({{si-track}}), at least as often as its `interval` when declared. - MUST packetize each `verbatim` frame as one section on its track's PID. @@ -462,6 +463,7 @@ A broadcast demultiplexed from a DVB transport stream: video and audio described - An `m2ts` track unwraps PCRs from its history, clamps extrapolated object times at a forward discontinuity, and cuts a fallback group after one second. `randomAccess` covers every published group; `muxRate` follows the measured source rate. - A `tracks` entry is never changed in place: a stream the PMT revises continues under a new name, and one it drops leaves the section. `programDescriptors` follows the PMT; `program`'s `transportStreamId` and `programNumber` do not change. - A consumer rebuilding a stream follows the catalog's revisions at the next PMT `version_number`, and carries a stream only while a PMT lists it. +- `pmtPid` follows a PAT that moves the PMT, and a consumer rebuilding a stream moves the PMT behind a PAT at the next `version_number`. ## draft-lcurley-moq-mpegts-00 {:numbered="false"} diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index e8bb1e074b2..7b3c893d7ca 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -73,7 +73,7 @@ refused means an error or a counted drop. | DVB SI on 0x0010, 0x0011, 0x0012, 0x0014 | Preserved: snapshot track per PID and `table_id` | landed | | Program identity, program and ES descriptors | Preserved: descriptors follow each PMT revision; an identity change refuses | landed | | PMT add, remove, retype; PID reuse | Normalized without a new epoch: a changed or reused PID ends its track and continues under a fresh name, and the export rebuilds the PMT at the next `version_number` | landed | -| PAT changes: moved PMT PID, programs added or removed | Normalized: followed; a removed program retires, and returns as a new epoch. Today a running export keeps the PMT on its first PID | [PAT changes](/quest/m1/ts-reliability/pat-lifecycle.md) | +| PAT changes: moved PMT PID, programs added or removed | Normalized: followed; a removed program retires, and returns as a new epoch, and a running export moves the PMT behind a PAT at the next `version_number` | landed | | Programs without video | Preserved when the PMT names a PCR PID; refused without one | landed | | PCR | Normalized: rebased, carried on the first video PID, else audio, else a dedicated PCR-only PID | landed | | Mux rate, null stuffing | Normalized: re-padded to `muxRate` while the source is CBR | landed | @@ -101,7 +101,6 @@ catalog linger (#5305). - [A timebase step restarts the TS import](/quest/m1/ts-reliability/step-restart.md) - an unflagged step or a rewind republishes under a new epoch instead of ending the import and every subscriber's export - [A restart keeps its catalog](/quest/m1/ts-reliability/restart-catalog.md) - a restarted import publishes its catalog with its first media -- [PAT changes](/quest/m1/ts-reliability/pat-lifecycle.md) - a running export follows a moved PMT PID - [A small step back keeps the TS export's video](/quest/m1/ts-reliability/step-back.md) - the export re-acquires on a sub-group step back from any source instead of dropping its video - [Decide whether a stitched export demotes its old leg](/quest/m1/ts-reliability/stitch-catch-up.md) - measure on the bufferbloat fixture whether demoting the old leg while the replacement acquires shrinks the switch's output hole, and land it if so diff --git a/quest/m1/ts-reliability/pat-lifecycle.md b/quest/m1/ts-reliability/pat-lifecycle.md deleted file mode 100644 index e649d3d8778..00000000000 --- a/quest/m1/ts-reliability/pat-lifecycle.md +++ /dev/null @@ -1,42 +0,0 @@ -# [S] A live export follows a moved PMT PID - -## Goal - -A running `export ts` whose catalog moves `program.pmtPid` follows it: a PAT -at the next `version_number` names the new PID before any PMT goes out -there, and nothing goes out on the old PID after it. The output stays one -program throughout, and no two PATs at one version name different PIDs. - -## Plan - -The import follows the PAT: a moved PMT PID moves its reassembler and -`program.pmtPid`, and `Programs` publishes, retires and republishes programs -as the PAT adds, drops and restores them. A running export still keeps the -PMT on the PID its tables first went out on, whatever `program.pmtPid` later -records: its output stays consistent, but a move is followed only by an -export started after it. - -The maintainer's plan, from closing -[moq-dev/moq#6301](https://github.com/moq-dev/moq/pull/6301): the export -follows a new `program.pmtPid` with a new PAT, built on the table rework -that lays the PAT and PMT from the current table state when a copy of them -begins to go out, so a move is one more state change rather than a rewrite -of queued copies. That rework has landed: the export's `Psi` holds the -tables as they stand, `Export::restate` hands them to -`Schedule::set_psi`, which every copy not yet begun lays, and a copy -already on its way finishes as it began. A move then updates `Psi::pmt_pid`, -calls `Schedule::set_tables` for the new PID, and restates; `Psi::current` -gives the PAT the next version when the PID differs from the last laid. -This quest keeps only the PMT move and its tests. - -The draft then says that `pmtPid` can change in place, and when a consumer -rebuilding the stream moves the PMT. - -Rust: `rs/moq-mux` TS export (`pmt_pid`, `build_psi`, the PSI schedule). - -Tests: a live export whose catalog moves `pmtPid` emits PATs at versions 0 -then 1, with nothing on the new PID before the moving PAT and nothing on the -old one after it, and a move while every copy is still queued goes out as -one PAT naming the new PID. `revision_test`'s -`a_track_recorded_on_the_pmt_pid_after_the_catalog_moves_it_takes_another` -then pins the move instead of the PMT staying put. diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index 843ca561702..7a71f4d280e 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -149,9 +149,11 @@ where /// Tracks [`Export::follow`] left on the ended broadcast, resubscribed as the returned /// catalog lists them. stale: HashSet, - /// The PAT and PMT `version_number` the program's first tables go out at, advanced by each - /// switch to another instance so a demux that caches tables by version reads the new ones. + /// The PMT `version_number` the program's first tables go out at, advanced by each switch to + /// another instance so a demux that caches tables by version reads the new ones. version: VersionNumber, + /// The PAT's, advanced by each switch alike. + pat_version: VersionNumber, /// After a switch to another instance, the PIDs whose first packet is queued flagging the /// break. `None` for an export that never switched. flagged: Option>, @@ -974,10 +976,10 @@ impl Kind { } } -/// The program tables: the PAT, fixed for the program's life but for its network entry, and the -/// PMT, which lists the streams on the air. The schedule lays them as they stand when each copy -/// begins to go out ([`Schedule::set_psi`]), so a change is one update of this state, and a -/// revision overtaken before any copy of it began takes no version. +/// The program tables: the PAT, fixed for the program's life but for its network entry and the +/// PID it names for the PMT, and the PMT, which lists the streams on the air. The schedule lays +/// them as they stand when each copy begins to go out ([`Schedule::set_psi`]), so a change is one +/// update of this state, and a revision overtaken before any copy of it began takes no version. struct Psi { /// The source's `transport_stream_id`, so the rebuilt PAT stays consistent with the carried SI. transport_stream_id: u16, @@ -986,6 +988,12 @@ struct Psi { /// PID the PMT rides on: the source's original (preserved from the service /// record) or the synthesized [`PMT_PID`] for a media-only source. pmt_pid: u16, + /// The PID the catalog moved the PMT to, which it takes once no copy of the tables is on its + /// way ([`Export::follow_pmt`]). + moving: Option, + /// The PIDs the PMT rode before, which no stream takes: a receiver may still read them as + /// the PMT's. + left: Vec, /// The PMT the last tables handed to the schedule listed: the streams the next copy adds. led: Option, /// The tables last set on the schedule, and the PIDs they flag a break on. @@ -998,13 +1006,16 @@ struct Psi { out: Option, } -/// Program tables as a copy of them goes out: the PMT at its version, and the PAT. +/// Program tables as a copy of them goes out: the PMT and the PAT, each at its version. #[derive(Clone)] struct Laid { version: VersionNumber, pmt: psi::Pmt, + /// The PID the PAT names for the PMT, which the PMT rides. + pmt_pid: u16, /// Whether the PAT lists the network entry, pointing at the NIT the export carries. network: bool, + pat_version: VersionNumber, pat: Bytes, } @@ -1020,18 +1031,26 @@ impl Psi { .filter(|&pcr| self.pmt.streams.iter().all(|es| es.pid.as_u16() != pcr)) } - /// The tables as they stand, listing the network entry when `network`. A PMT that differs - /// from the last laid goes out at the next version, `first` before any has; the PAT keeps its - /// version until its network entry changes, so it never runs ahead of the PMT's. - fn current(&self, network: bool, first: VersionNumber) -> anyhow::Result { - let (version, pat) = match &self.laid { - Some(laid) if laid.pmt == self.pmt && laid.network == network => return Ok(laid.clone()), + /// The tables as they stand, listing the network entry when `network`. Each table that + /// differs from the last laid goes out at the version after that one's own, `first` (the PMT's + /// and the PAT's) before any has: the PAT changes only with its network entry or PMT PID. + fn current(&self, network: bool, first: (VersionNumber, VersionNumber)) -> anyhow::Result { + let next = |mut version: VersionNumber| { + version.increment(); + version + }; + let (version, pat_version, pat) = match &self.laid { Some(laid) => { - let mut version = laid.version; - version.increment(); - (version, (laid.network == network).then(|| laid.pat.clone())) + let version = match laid.pmt == self.pmt { + true => laid.version, + false => next(laid.version), + }; + match laid.network == network && laid.pmt_pid == self.pmt_pid { + true => (version, laid.pat_version, Some(laid.pat.clone())), + false => (version, next(laid.pat_version), None), + } } - None => (first, None), + None => (first.0, first.1, None), }; let pat = match pat { Some(pat) => pat, @@ -1040,13 +1059,15 @@ impl Psi { self.pmt.program_number, self.pmt_pid, network, - version, + pat_version, )?, }; Ok(Laid { version, pmt: self.pmt.clone(), + pmt_pid: self.pmt_pid, network, + pat_version, pat, }) } @@ -1329,6 +1350,7 @@ where instance, stale: HashSet::new(), version: VersionNumber::default(), + pat_version: VersionNumber::default(), flagged: None, tables: Vec::new(), psi_due: false, @@ -2004,12 +2026,13 @@ where .psi .as_ref() .and_then(|psi| psi.out.as_ref()) - .map(|laid| laid.version); - next.version = out.unwrap_or(self.program.version); + .map(|laid| (laid.version, laid.pat_version)); + (next.version, next.pat_version) = out.unwrap_or((self.program.version, self.program.pat_version)); next.epoch = self.program.epoch; next.emitted_epoch = self.program.emitted_epoch; if out.is_some() { next.version.increment(); + next.pat_version.increment(); next.epoch += 1; next.flagged = Some(HashSet::new()); next.pcr_discontinuity = true; @@ -2173,6 +2196,7 @@ where } } self.revise()?; + self.move_pmt(&mpegts)?; } // Assign a PID to every desired track the PMT does not list yet: prefer the original @@ -2182,7 +2206,8 @@ where // the first (partial) snapshot. A recorded PID may take over one a stream the catalog // dropped still drains on ([`Self::list`]); any other PID the PMT lists is taken. let pmt_pid = self.pmt_pid(); - let mut taken: Vec = vec![0x0000, pmt_pid, 0x1FFF]; + let mut taken: Vec = vec![0x0000, 0x1FFF]; + taken.extend(self.pmt_pids()); taken.extend(self.program.psi.as_ref().and_then(|psi| psi.own_clock())); taken.extend( self.program @@ -2801,7 +2826,8 @@ where /// PID the PMT rides on: the source's original (preserved in the service record), /// or the synthesized [`PMT_PID`] for a media-only source or an invalid (zero) value. - /// Once the tables are built they keep the PID they went out on. + /// Once the tables are built they move only behind a PAT naming the new PID + /// ([`Self::move_pmt`]). fn pmt_pid(&self) -> u16 { if let Some(psi) = &self.program.psi { return psi.pmt_pid; @@ -2814,6 +2840,80 @@ where .unwrap_or(PMT_PID) } + /// The PIDs no stream takes for the PMT's sake: the one it rides, the one it is moving to, + /// and the ones it rode before. + fn pmt_pids(&self) -> Vec { + let mut pids = vec![self.pmt_pid()]; + if let Some(psi) = &self.program.psi { + pids.extend(psi.moving); + pids.extend(&psi.left); + } + pids + } + + /// Follow a catalog that moves the PMT to another PID: the PAT of the next tables names it, + /// at the PAT's next version ([`Psi::current`]). A move onto a PID a stream the catalog + /// dropped still drains on waits for it ([`Self::follow_pmt`]); one onto a PID the catalog + /// gives a stream or SI, the clock's own PID, or no PID at all is refused, as is a catalog + /// that gives the PID to a stream while the move waits. A catalog that records no PID keeps + /// the PMT where it is. + fn move_pmt(&mut self, mpegts: &catalog::Mpegts) -> anyhow::Result<()> { + let Some(pid) = self.program.program.as_ref().map(|s| s.pmt_pid).filter(|&pid| pid != 0) else { + // The catalog no longer asks for a move, so drop one that is still waiting. + if let Some(psi) = self.program.psi.as_mut() { + psi.moving = None; + } + return Ok(()); + }; + let Some(psi) = self.program.psi.as_mut() else { + return Ok(()); + }; + if pid == psi.pmt_pid { + psi.moving = None; + return Ok(()); + } + let psi = &*psi; + anyhow::ensure!(pid < 0x1fff, "TS export cannot move the PMT to PID {pid:#06x}"); + let taken = psi.own_clock() == Some(pid) + || mpegts.si.contains_key(&pid) + || self + .program + .tracks + .values() + .any(|track| track.cataloged && track.pid == pid); + anyhow::ensure!( + !taken, + "TS export cannot move the PMT from PID {:#06x} to {pid:#06x}, which another stream rides", + psi.pmt_pid + ); + self.program.psi.as_mut().expect("checked above").moving = Some(pid); + self.follow_pmt() + } + + /// Move the PMT to the PID the catalog moved it to once no copy of the tables is on its way, + /// so each copy goes out whole on the PID its PAT names, and no stream is left on the PID, + /// and lead the next frame with them. Every copy not yet begun takes the move. + fn follow_pmt(&mut self) -> anyhow::Result<()> { + let Some(psi) = self.program.psi.as_mut() else { + return Ok(()); + }; + let Some(pid) = psi.moving else { + return Ok(()); + }; + if !psi.sending.is_empty() || self.program.tracks.values().any(|track| track.pid == pid) { + return Ok(()); + } + psi.moving = None; + let old = std::mem::replace(&mut psi.pmt_pid, pid); + psi.left.retain(|&left| left != pid); + psi.left.push(old); + tracing::debug!(from = old, to = pid, "moving the PMT"); + self.program.schedule.set_tables(pid); + self.program.schedule.forget(old); + self.program.revised = true; + self.restate() + } + /// Build the PAT/PMT once every track's PID and codec is known. fn build_psi(&mut self) -> anyhow::Result<()> { for track in self.program.tracks.values_mut() { @@ -2846,6 +2946,8 @@ where transport_stream_id, pmt, pmt_pid, + moving: None, + left: Vec::new(), led: None, stated: None, laid: None, @@ -2879,8 +2981,9 @@ where }) .map(|t| t.pid) .or_else(|| { + let pmt_pids = self.pmt_pids(); (FIRST_ES_PID..0x1fff).find(|pid| { - *pid != self.pmt_pid() + !pmt_pids.contains(pid) && self.program.tracks.values().all(|track| track.pid != *pid) && self.program.si.keys().all(|(si, _)| si != pid) }) @@ -3066,6 +3169,8 @@ where } if changed { self.revise()?; + // A move may have waited for a stream that just retired. + self.follow_pmt()?; } for name in free { let parked = std::mem::take(&mut self.program.tracks.get_mut(&name).expect("collected above").parked); @@ -3081,6 +3186,8 @@ where if !std::mem::take(&mut self.program.psi_due) { return Ok(()); } + // A rewind may have dropped the copy a move waited for. + self.follow_pmt()?; self.restate()?; let psi = self.program.psi.as_mut().context("PSI not built")?; let added: Vec = psi @@ -3100,12 +3207,12 @@ where /// Set the program tables as they stand on the schedule, which every copy of them still to /// begin lays. A NIT acquired or lost after the tables went out revises the PAT's network - /// entry, at the next version beside the PMT. The first copy after a switch to another + /// entry, at the PAT's next version. The first copy after a switch to another /// instance flags the break on each table PID that owes one ([`Self::breaks`]). fn restate(&mut self) -> anyhow::Result<()> { let network = self.carries_nit(); let psi = self.program.psi.as_ref().context("PSI not built")?; - let tables = psi.current(network, self.program.version)?; + let tables = psi.current(network, (self.program.version, self.program.pat_version))?; let pmt_pid = psi.pmt_pid; let clock = psi.pcr_pid().context("PMT without a clock")?; let section = super::psi::encode_pmt(&tables.pmt, tables.version)?; @@ -3570,9 +3677,9 @@ where /// grid instead: slots on the media timeline, shared by every exporter of the /// broadcast, like [`due`]. fn lay(&mut self, known: Option) -> anyhow::Result<()> { - let Some(pmt_pid) = self.program.psi.as_ref().map(|psi| psi.pmt_pid) else { + if self.program.psi.is_none() { return Ok(()); - }; + } if known.is_none() { self.flush_packs(|_, _| true)?; } @@ -3589,6 +3696,7 @@ where break; }; let clock = pcr_packet(slot.clock.context("a slot without a clock PID")?, slot.pcr)?; + let pmt_pid = self.pmt_pid(); let mut payload = slot.layout(&clock, pmt_pid, &NULL_PACKET); // The slot opens with its PCR, in the clock packet or the clock PID's packet that // carries it, whose adaptation field flags the break. @@ -3625,6 +3733,8 @@ where } self.restate()?; } + // The copies a move waited for may have finished. + self.follow_pmt()?; } Ok(()) } @@ -4630,13 +4740,13 @@ mod tests { use moq_net::Timestamp; use super::{ - Buffer, DecodeClock, Dolby, Kind, PSI_INTERVAL, PesPacket, Psi, Settle, dolby_removals, due, + Buffer, DecodeClock, Dolby, Kind, Laid, PSI_INTERVAL, PesPacket, Psi, Settle, dolby_removals, due, is_complete_section, si_due, slot, unwrap_pts, verbatim_buffer, }; /// The tables as they stand take the version after the last laid, whatever was set on the /// schedule meanwhile, and the last laid's own when they come back to it. The PAT keeps its - /// version until its network entry changes. + /// own version until its network entry or PMT PID changes. #[test] fn a_revision_takes_the_version_after_the_last_laid() { use mpeg2ts::ts::{Pid, VersionNumber}; @@ -4658,28 +4768,45 @@ mod tests { transport_stream_id: 1, pmt: pmt(&[0x100]), pmt_pid: 0x20, + moving: None, + left: Vec::new(), led: None, stated: None, laid: None, sending: Default::default(), out: None, }; - let first = VersionNumber::from_u8(3).unwrap(); + let version = |version| VersionNumber::from_u8(version).unwrap(); + let first = (version(3), version(7)); + let versions = |laid: &Laid| (laid.version.as_u8(), laid.pat_version.as_u8()); let laid = psi.current(false, first).unwrap(); - assert_eq!(laid.version.as_u8(), 3); + assert_eq!(versions(&laid), (3, 7)); psi.laid = Some(laid.clone()); psi.pmt = pmt(&[0x100, 0x101]); let revised = psi.current(false, first).unwrap(); - assert_eq!(revised.version.as_u8(), 4); + assert_eq!(versions(&revised), (4, 7)); assert_eq!(revised.pat, laid.pat); psi.pmt = pmt(&[0x100]); - assert_eq!(psi.current(false, first).unwrap().version.as_u8(), 3); + assert_eq!(versions(&psi.current(false, first).unwrap()), (3, 7)); let network = psi.current(true, first).unwrap(); - assert_eq!(network.version.as_u8(), 4); + assert_eq!(versions(&network), (3, 8)); assert_ne!(network.pat, laid.pat); + + // A moved PMT re-encodes the PAT at its own next version, naming the new PID: after 31 + // PMT revisions, the PMT's next is the PAT's last. + psi.laid = Some(Laid { + version: version(31), + pat_version: version(0), + ..laid + }); + psi.pmt_pid = 0x30; + let moved = psi.current(false, first).unwrap(); + assert_eq!(versions(&moved), (31, 1)); + let pat = super::psi::PatAssembler::default().section(&moved.pat).unwrap(); + assert_eq!(pat.programs[0].pmt_pid.as_u16(), 0x30); } /// The same available video/audio backlog must choose the same start in either fill order. diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 53e91af2c96..05b1cf52421 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -11264,6 +11264,55 @@ async fn a_switch_follows_the_last_pmt_out() { drop((old, old_catalog)); } +/// A PMT move no slot laid before a switch reached no receiver: the replacement's PAT follows +/// the last one out by one version. +#[tokio::test(start_paused = true)] +async fn a_switch_follows_the_last_pat_out() { + let origin = crate::source::produce_origin(); + let route = || moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()); + + let (old, mut old_catalog) = publish_live::(&origin, route()); + let mut audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); + let source = crate::source::resolved(&origin, "live").await; + let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang) + .await + .unwrap() + .with_delay(RECORDING_MAX_AGE) + .with_replay(); + for ms in (0..200).step_by(20) { + write_aac(&mut audio, ms); + } + let mut frames = drain_frames(&mut export).await; + old_catalog.modify().unwrap().ext.mpegts.program = Some(tscat::Program { + transport_stream_id: 1, + program_number: 1, + pmt_pid: 0x30, + }); + frames.extend(drain_frames(&mut export).await); + + let (new, mut new_catalog) = publish_live::(&origin, route()); + let mut audio = aac_rendition(&mut new.broadcast().clone(), &mut new_catalog, "b.aac"); + let replacement = origin.consume().request("live", None).await.unwrap(); + let stream = crate::catalog::Consumer::new(&replacement, Default::default()) + .await + .unwrap(); + let mut export = export.follow(replacement, stream).unwrap(); + for ms in (0..200).step_by(20) { + write_aac(&mut audio, ms); + } + frames.extend(drain_frames(&mut export).await); + audio.finish().unwrap(); + new_catalog.finish().unwrap(); + let (rest, end) = drain_to_end(&mut export).await; + frames.extend(rest); + end.unwrap(); + + let mut versions: Vec = pats(&frames).iter().map(|&(version, _)| version).collect(); + versions.dedup(); + assert_eq!(versions, [0, 1]); + drop((old, old_catalog)); +} + /// A switch onto an instance replaced after it built its tables but before a slot laid them /// numbers the next program's from the last PMT out, not from the one that never went. #[tokio::test(start_paused = true)] diff --git a/rs/moq-mux/src/container/ts/revision_test.rs b/rs/moq-mux/src/container/ts/revision_test.rs index dbaadccf4b1..c5a535cb142 100644 --- a/rs/moq-mux/src/container/ts/revision_test.rs +++ b/rs/moq-mux/src/container/ts/revision_test.rs @@ -50,6 +50,13 @@ struct Span<'a> { /// as a live one keeps up with the catalog, and runs to its end after the last. Returns /// every PMT it wrote, each run of repeats once. async fn reexport(spans: &[Span<'_>], intervals: u64) -> Vec { + let spans: Vec<_> = spans.iter().map(|span| (PMT_PID, span)).collect(); + tables(&reexport_frames(&spans, intervals).await, PMT_PID) +} + +/// [`reexport`], each span's PAT naming its PMT on the PID beside it. Returns every frame +/// the export wrote. +async fn reexport_frames(spans: &[(u16, &Span<'_>)], intervals: u64) -> Vec { let mut broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); let mut catalog = crate::catalog::Producer::new( @@ -62,10 +69,10 @@ async fn reexport(spans: &[Span<'_>], intervals: u64) -> Vec
{ let mut ticks = 1_000 * MS; let mut export = None; let mut frames = Vec::new(); - for (version, span) in spans.iter().enumerate() { + for (version, &(pmt_pid, span)) in spans.iter().enumerate() { let mut mux = Mux::default(); - mux.pat(&[(1, PMT_PID)]) - .pmt_at(version as u8 % 32, PMT_PID, 1, CLOCK_PID, span.streams, 0); + mux.pat(&[(1, pmt_pid)]) + .pmt_at(version as u8 % 32, pmt_pid, 1, CLOCK_PID, span.streams, 0); // The import publishes its catalog once it has measured the multiplex rate. let intervals = if version == 0 { 75 } else { intervals }; for _ in 0..intervals { @@ -104,7 +111,7 @@ async fn reexport(spans: &[Span<'_>], intervals: u64) -> Vec
{ catalog.finish().unwrap(); drain(&mut export.unwrap(), &mut frames).await; - tables(&frames, PMT_PID) + frames } /// Run `export` until it waits on its source, as a live one keeps up with each catalog @@ -235,6 +242,33 @@ fn tables_at(frames: &[Frame], pmt_pid: u16) -> Vec<(usize, Table)> { out } +/// Every PAT in `frames` as its `version_number` and PMT PID, each run of repeats once. +fn pats(frames: &[Frame]) -> Vec<(u8, u16)> { + let mut out = Vec::new(); + for pat in packets(frames).filter_map(pat) { + if out.last() != Some(&pat) { + out.push(pat); + } + } + out +} + +/// The `version_number` and PMT PID of the PAT `packet` starts, if it starts one. +fn pat(packet: &[u8; 188]) -> Option<(u8, u16)> { + if pid(packet) != 0 || packet[1] & 0x40 == 0 { + return None; + } + let mut at = 4; + if packet[3] & 0x20 != 0 { + at += 1 + usize::from(packet[4]); + } + let section = &packet[at + 1 + usize::from(packet[at])..]; + Some(( + (section[5] >> 1) & 0x1f, + u16::from(section[10] & 0x1f) << 8 | u16::from(section[11]), + )) +} + /// Each table's version and streams as `(stream_type, PID)`. fn layouts(tables: &[Table]) -> Vec<(u8, Vec<(u8, u16)>)> { tables @@ -1079,9 +1113,8 @@ async fn a_pid_taken_from_a_stream_that_never_ends_cuts_it() { drop(old); } -/// The PMT stays on the PID and program its tables went out with, whatever a later catalog -/// records, so a track a later catalog records on that PID takes another, and each revision is -/// still the program the PAT names. +/// A track the catalog records on the PMT's PID as it moves the PMT takes another, and the +/// PMT moves behind a new PAT, still the program the first named. #[tokio::test(start_paused = true)] async fn a_track_recorded_on_the_pmt_pid_after_the_catalog_moves_it_takes_another() { let program = |pmt_pid: u16, program_number: u16| super::catalog::Program { @@ -1118,14 +1151,301 @@ async fn a_track_recorded_on_the_pmt_pid_after_the_catalog_moves_it_takes_anothe catalog.finish().unwrap(); drain(&mut export, &mut frames).await; - assert!(tables(&frames, OWN_PMT_PID).is_empty(), "the PMT stays on its PID"); - let tables = tables(&frames, PMT_PID); + assert_eq!(pats(&frames), [(0, PMT_PID), (1, OWN_PMT_PID)]); + let tables = tables(&frames, OWN_PMT_PID); let pids: Vec = tables.last().unwrap().streams.iter().map(|es| es.1).collect(); assert_eq!(pids.len(), 2, "{tables:?}"); - assert!(!pids.contains(&PMT_PID), "{tables:?}"); + assert!(!pids.contains(&PMT_PID) && !pids.contains(&OWN_PMT_PID), "{tables:?}"); assert!(tables.iter().all(|table| table.program == 1), "{tables:?}"); } +/// The program a catalog records, its PMT on `pmt_pid`. +fn program(pmt_pid: u16) -> super::catalog::Program { + super::catalog::Program { + transport_stream_id: 1, + program_number: 1, + pmt_pid, + } +} + +/// A catalog that moves the PMT to another PID is followed by a PAT at the next version +/// naming it, and from then on the PMT rides the new PID alone. +#[tokio::test(start_paused = true)] +async fn a_moved_pmt_moves_behind_a_new_pat() { + const MOVED_PMT_PID: u16 = 0x30; + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new( + &mut broadcast, + crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::::default()), + ) + .unwrap(); + catalog.modify().unwrap().ext.mpegts.program = Some(program(PMT_PID)); + let (_, mut audio) = aac_track(&mut broadcast, &mut catalog, Some(AUDIO_PID)); + let mut frames = Vec::new(); + let start = tokio::time::Instant::now(); + let mut export = export_of(&consumer).await; + live(&mut export, &mut frames, start, &mut [(&mut audio, 0x21)], 0, 3_000).await; + catalog.modify().unwrap().ext.mpegts.program = Some(program(MOVED_PMT_PID)); + live(&mut export, &mut frames, start, &mut [(&mut audio, 0x21)], 3_000, 6_000).await; + audio.finish().unwrap(); + catalog.finish().unwrap(); + drain(&mut export, &mut frames).await; + + assert_eq!(pats(&frames), [(0, PMT_PID), (1, MOVED_PMT_PID)]); + assert_eq!( + layouts(&tables(&frames, MOVED_PMT_PID)), + [(0, vec![(AUDIO, AUDIO_PID)])] + ); + let moved = packets(&frames) + .position(|packet| pat(packet) == Some((1, MOVED_PMT_PID))) + .expect("the PAT naming the new PID"); + assert!( + packets(&frames).take(moved).all(|packet| pid(packet) != MOVED_PMT_PID), + "no PMT on the new PID before the PAT naming it" + ); + assert!( + packets(&frames).skip(moved).all(|packet| pid(packet) != PMT_PID), + "no PMT on the old PID after the PAT that moved it" + ); +} + +/// A catalog that moves the PMT onto the PID a stream it lists rides fails the export: the +/// PMT cannot follow it there. +#[tokio::test(start_paused = true)] +async fn a_pmt_moved_onto_a_stream_fails_the_export() { + let (mut broadcast, consumer, mut catalog) = broadcast(); + catalog.modify().unwrap().ext.mpegts.program = Some(program(PMT_PID)); + let (_, mut audio) = aac_track(&mut broadcast, &mut catalog, Some(AUDIO_PID)); + let mut frames = Vec::new(); + let start = tokio::time::Instant::now(); + let mut export = export_of(&consumer).await; + live(&mut export, &mut frames, start, &mut [(&mut audio, 0x21)], 0, 3_000).await; + catalog.modify().unwrap().ext.mpegts.program = Some(program(AUDIO_PID)); + let err = tokio::time::timeout(DELAY * 2, async { + loop { + if let Err(err) = export.next().await { + return err; + } + } + }) + .await + .expect("the export fails"); + assert!(err.to_string().contains("another stream rides"), "{err}"); + assert_eq!(pats(&frames), [(0, PMT_PID)]); +} + +/// A catalog that moves the PMT onto the PID of a stream it dropped moves it once that stream +/// has drained: the stream's packets all go out ahead of the PAT naming its PID, and only the +/// PMT rides it after. +#[tokio::test(start_paused = true)] +async fn a_pmt_moved_onto_a_draining_stream_waits_for_it() { + let (mut broadcast, consumer, mut catalog) = broadcast(); + catalog.modify().unwrap().ext.mpegts.program = Some(program(PMT_PID)); + let (_, mut audio) = aac_track(&mut broadcast, &mut catalog, Some(AUDIO_PID)); + let (name, mut leaving) = aac_track(&mut broadcast, &mut catalog, Some(OTHER_AUDIO_PID)); + let mut frames = Vec::new(); + let start = tokio::time::Instant::now(); + let mut export = export_of(&consumer).await; + live( + &mut export, + &mut frames, + start, + &mut [(&mut audio, 0x21), (&mut leaving, 0x22)], + 0, + 3_000, + ) + .await; + remove(&mut catalog, &name); + catalog.modify().unwrap().ext.mpegts.program = Some(program(OTHER_AUDIO_PID)); + live( + &mut export, + &mut frames, + start, + &mut [(&mut audio, 0x21), (&mut leaving, 0x22)], + 3_000, + 4_000, + ) + .await; + leaving.finish().unwrap(); + live(&mut export, &mut frames, start, &mut [(&mut audio, 0x21)], 4_000, 6_000).await; + audio.finish().unwrap(); + catalog.finish().unwrap(); + drain(&mut export, &mut frames).await; + + assert_eq!(pats(&frames), [(0, PMT_PID), (1, OTHER_AUDIO_PID)]); + let moved = packets(&frames) + .position(|packet| pat(packet) == Some((1, OTHER_AUDIO_PID))) + .expect("the PAT naming the new PID"); + // What each unit starting on the PID opens: a PES, or a section behind its pointer field. + let starts = |packets: Vec<&[u8; 188]>| -> Vec { + packets + .into_iter() + .filter(|packet| pid(packet) == OTHER_AUDIO_PID && packet[1] & 0x40 != 0) + .map(|packet| { + let at = match packet[3] & 0x20 != 0 { + true => 5 + usize::from(packet[4]), + false => 4, + }; + match packet[at..at + 3] == [0, 0, 1] { + true => 0xff, + false => packet[at + 1 + usize::from(packet[at])], + } + }) + .collect() + }; + let before = starts(packets(&frames).take(moved).collect()); + assert!( + !before.is_empty() && before.iter().all(|&start| start == 0xff), + "{before:?}" + ); + let after = starts(packets(&frames).skip(moved).collect()); + assert!( + !after.is_empty() && after.iter().all(|&start| start == 0x02), + "{after:?}" + ); +} + +/// A catalog that stops recording a PMT PID while a move waits for a draining stream cancels +/// the move: the PMT stays where it is once the stream has drained. +#[tokio::test(start_paused = true)] +async fn a_catalog_dropping_the_pmt_pid_cancels_a_waiting_move() { + let (mut broadcast, consumer, mut catalog) = broadcast(); + catalog.modify().unwrap().ext.mpegts.program = Some(program(PMT_PID)); + let (_, mut audio) = aac_track(&mut broadcast, &mut catalog, Some(AUDIO_PID)); + let (name, mut leaving) = aac_track(&mut broadcast, &mut catalog, Some(OTHER_AUDIO_PID)); + let mut frames = Vec::new(); + let start = tokio::time::Instant::now(); + let mut export = export_of(&consumer).await; + let mut tracks = [(&mut audio, 0x21), (&mut leaving, 0x22)]; + live(&mut export, &mut frames, start, &mut tracks, 0, 3_000).await; + remove(&mut catalog, &name); + catalog.modify().unwrap().ext.mpegts.program = Some(program(OTHER_AUDIO_PID)); + live(&mut export, &mut frames, start, &mut tracks, 3_000, 3_500).await; + catalog.modify().unwrap().ext.mpegts.program = Some(program(0)); + live(&mut export, &mut frames, start, &mut tracks, 3_500, 4_000).await; + leaving.finish().unwrap(); + live(&mut export, &mut frames, start, &mut [(&mut audio, 0x21)], 4_000, 6_000).await; + audio.finish().unwrap(); + catalog.finish().unwrap(); + drain(&mut export, &mut frames).await; + + assert_eq!(pats(&frames), [(0, PMT_PID)]); +} + +/// A catalog that gives a stream back the PID the PMT waits to move onto fails the export. +#[tokio::test(start_paused = true)] +async fn a_stream_returning_to_the_pid_the_pmt_waits_for_fails_the_export() { + let (mut broadcast, consumer, mut catalog) = broadcast(); + catalog.modify().unwrap().ext.mpegts.program = Some(program(PMT_PID)); + let (_, mut audio) = aac_track(&mut broadcast, &mut catalog, Some(AUDIO_PID)); + let (name, mut leaving) = aac_track(&mut broadcast, &mut catalog, Some(OTHER_AUDIO_PID)); + let config = catalog.snapshot().audio.renditions[&name].clone(); + let mut frames = Vec::new(); + let start = tokio::time::Instant::now(); + let mut export = export_of(&consumer).await; + let mut tracks = [(&mut audio, 0x21), (&mut leaving, 0x22)]; + live(&mut export, &mut frames, start, &mut tracks, 0, 3_000).await; + remove(&mut catalog, &name); + catalog.modify().unwrap().ext.mpegts.program = Some(program(OTHER_AUDIO_PID)); + live(&mut export, &mut frames, start, &mut tracks, 3_000, 3_500).await; + { + let mut guard = catalog.modify().unwrap(); + guard.audio.renditions.insert(name.clone(), config); + guard + .ext + .mpegts + .tracks + .insert(name, super::catalog::Track::new(OTHER_AUDIO_PID)); + } + let err = tokio::time::timeout(DELAY * 2, async { + loop { + if let Err(err) = export.next().await { + return err; + } + } + }) + .await + .expect("the export fails"); + assert!(err.to_string().contains("another stream rides"), "{err}"); + assert_eq!(pats(&frames), [(0, PMT_PID)]); +} + +/// A move landing while a copy of the tables is on its way waits for that copy: at a rate low +/// enough that a copy spans slots, every PMT packet rides the PID the PAT before it names, and +/// no two PATs at one version name different PIDs. +#[tokio::test(start_paused = true)] +async fn a_pmt_moved_while_its_tables_go_out_moves_behind_them() { + const MOVED_PMT_PID: u16 = 0x30; + let (mut broadcast, consumer, mut catalog) = broadcast(); + catalog.modify().unwrap().ext.mpegts.program = Some(program(PMT_PID)); + let (_, mut audio) = aac_track(&mut broadcast, &mut catalog, Some(AUDIO_PID)); + let mut frames = Vec::new(); + let start = tokio::time::Instant::now(); + let mut export = export_of(&consumer).await.with_mux_rate(rate(4)); + for millis in (0..6_000).step_by(50) { + tokio::time::sleep_until(start + Duration::from_millis(millis)).await; + match millis { + // Three packets of PMT from here, in slots of four. + 2_500 => { + catalog.modify().unwrap().ext.mpegts.program_descriptors = (0x80..0x82) + .map(|tag| super::catalog::Descriptor { + tag, + data: bytes::Bytes::from(vec![tag; 250]), + }) + .collect(); + } + // Back and forth, landing at each point of the copies going out. + 3_000..4_000 => { + let pmt_pid = match millis / 50 % 2 { + 0 => MOVED_PMT_PID, + _ => PMT_PID, + }; + catalog.modify().unwrap().ext.mpegts.program = Some(program(pmt_pid)); + } + _ => {} + } + write_marked(&mut audio, millis, 0x21); + settle(&mut export, &mut frames).await; + } + audio.finish().unwrap(); + catalog.finish().unwrap(); + drain(&mut export, &mut frames).await; + + let pats = pats(&frames); + assert!(pats.len() > 2, "{pats:?}"); + let mut named = std::collections::HashMap::new(); + for &(version, pmt_pid) in &pats { + assert_eq!(*named.entry(version).or_insert(pmt_pid), pmt_pid, "{pats:?}"); + } + let mut pmt_pid = None; + for packet in packets(&frames) { + if let Some((_, pid)) = pat(packet) { + pmt_pid = Some(pid); + } + if [PMT_PID, MOVED_PMT_PID].contains(&pid(packet)) { + assert_eq!(Some(pid(packet)), pmt_pid, "a PMT packet off the PID its PAT names"); + } + } +} + +/// A PAT that moves the PMT before any table has gone out takes every copy queued with it: +/// the PAT names the new PID at the version the first had, and nothing goes out on the old. +#[tokio::test(start_paused = true)] +async fn a_pmt_moved_before_its_tables_go_out_moves_every_copy() { + const MOVED_PMT_PID: u16 = 0x30; + let span = Span { + streams: &[(AUDIO, AUDIO_PID, &[])], + }; + let frames = reexport_frames(&[(PMT_PID, &span), (MOVED_PMT_PID, &span)], 25).await; + assert_eq!(pats(&frames), [(0, MOVED_PMT_PID)]); + assert_eq!( + layouts(&tables(&frames, MOVED_PMT_PID)), + [(0, vec![(AUDIO, AUDIO_PID)])] + ); + assert!(packets(&frames).all(|packet| pid(packet) != PMT_PID)); +} + /// An SCTE-35 `splice_null` section. const CUE: &[u8] = &[ 0xfc, 0x30, 0x1b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xff, 0xf0, 0x0a, 0x05, 0x00, 0x00, 0x2b, 0xb4, 0x7f,