diff --git a/doc/concept/hang.md b/doc/concept/hang.md index a676fcaba06..01dd4a27aad 100644 --- a/doc/concept/hang.md +++ b/doc/concept/hang.md @@ -170,9 +170,10 @@ service. The root `m2ts` section names the one track holding it: Each frame is a run of whole 188-byte packets in input order, timestamped on the multiplex's own PCR clock, which never steps back: a source whose clock -steps back is a new broadcast. `randomAccess` says whether every group begins -where a receiver can start decoding, and `muxRate` is the constant multiplex -rate in bits per second, absent for a variable-rate source. A player has +steps back is a new broadcast. `randomAccess` says whether every group of the +latest two-second window began where a receiver can start decoding, and +`muxRate` is that window's constant multiplex rate in bits per second, absent +for a variable-rate source. A player has nothing to decode here. `moq import ts --passthrough` writes this section (see [moq-cli](/bin/cli)); the encoding is in [draft-lcurley-moq-mpegts](/draft/moq-mpegts). diff --git a/drafts/draft-lcurley-moq-mpegts.md b/drafts/draft-lcurley-moq-mpegts.md index d8429b75c25..b15555bc053 100644 --- a/drafts/draft-lcurley-moq-mpegts.md +++ b/drafts/draft-lcurley-moq-mpegts.md @@ -278,10 +278,10 @@ type M2ts = { `track` names the one track carrying the multiplex, in the same broadcast as the catalog. A second multiplex is a second broadcast. -`randomAccess` says that every group starts at a random access point ({{m2ts-groups}}). +`randomAccess` says that every group of the latest window the section describes started at a random access point ({{m2ts-groups}}). A publisher MUST write it, and a consumer MUST read an absent `randomAccess` as false. -`muxRate` follows {{field-mux-rate}}, measured over every packet of the track. +`muxRate` follows {{field-mux-rate}}, measured over every packet of that window. The section records no program, PID, descriptor, or table: they all reach the consumer inside the packets, as the publisher received them. A track named here MUST NOT also be described as a rendition or in an `mpegts` section. @@ -302,11 +302,15 @@ A group starts at: - a packet with `random_access_indicator` set on the video PID of the clock's program (the first PID its PMT declares with a video `stream_type`), or on the clock PID when that program has no video; - a PCR on the clock PID with `discontinuity_indicator` set that steps forward ({{m2ts-timestamps}}), since a group MUST NOT contain a PCR discontinuity; -- otherwise, the first PCR at least one second of PCR time after the group's start; +- the first PCR at or past a whole second of the PCR count ({{m2ts-timestamps}}), unless a random access point or a forward jump fell in the second before it, each counted at the PCR before it (its own, if its packet carries one); - the frame that would take a group past a size bound the publisher chooses below what a relay caches of one group; frames are never split to meet it. -`randomAccess` is true only while the PAT lists one program, its PMT declares at most one video stream, and the clock PID is that PMT's `PCR_PID`, and only if every group so far has started at a `random_access_indicator`. -The first group that starts any other way, or that carries packets while that layout does not hold, republishes the section with `randomAccess` false for the rest of the track. +Before its first group, a publisher starts one at a second only once it has watched from a PCR before the second before it, with the PMT of the clock's program read, so it never cuts where a publisher running longer saw a random access point or a jump (the first PCR it saw may carry a flag it missed). + +The section describes two-second windows of the PCR count, each holding the PCR intervals that start in it. +As the count passes each window, from the first that starts after the PCR the publisher's first group follows, the section takes that window's values; a window whose values match the last one's leaves the catalog unchanged. +An input that ends before its first such window takes `randomAccess` from every group it carried and `muxRate` from the window in progress. +For a window, `randomAccess` is true only if the PAT listed one program, its PMT declared at most one video stream, and the clock PID was that PMT's `PCR_PID` throughout it, and every group it started began at a `random_access_indicator`. With more programs or video streams the random access points can stagger, so a group start is not one for all of them. Both indicators sit in the adaptation field, which TS-level scrambling leaves in the clear, so a scrambled multiplex groups like its clear twin. @@ -318,11 +322,11 @@ A PCR gives the time of the byte holding the last bit of its `program_clock_refe A byte between two PCRs is placed at the rate between them; a byte past the last PCR of a timebase takes the rate of the interval before it, and one before the first PCR of a timebase takes the rate of the interval after it. PCRs are unwrapped into a 64-bit count of 27 MHz ticks, and timestamps never decrease. -A step forward of at most half the 2^33 × 300 tick period adds to the count, across the wrap included, so a wrap is not a discontinuity. -A PCR with `discontinuity_indicator` set whose step is forward by that measure adds its step too, after the bytes before it are timed at the rate before it, and the group it starts MUST follow hang's discontinuity marker ({{hang}}). +Each PCR takes the number of 2^33 × 300 tick wraps that lands it nearest the current time on the catalog clock ({{hang}}), so the count runs on across the wrap, which is not a discontinuity, and a PCR more than half a wrap ahead of that time lands a wrap back. +A PCR with `discontinuity_indicator` set that lands at or past the one before it adds its step too, after the bytes before it are timed at the rate before it, and the group it starts MUST follow hang's discontinuity marker ({{hang}}). A PCR that steps backwards MUST end the track with an error, flagged or not: a flagged one is a restart, which is new content, so a publisher that carries on publishes it as a new broadcast. -The timeline depends only on the bytes since the publisher's first PCR, so two publishers that start at the same packet of one input publish identical frames. +The count therefore depends only on the bytes and the catalog clock, so publishers of one input whose catalog clocks agree publish identical frames, group boundaries, and sections wherever each started, from the later one's first group on. ## Mapping onto MSFTS {#m2ts-msfts} Stripping each frame's timestamp leaves an {{msfts}} object of whole source packets, and the groups carry over unchanged. @@ -443,6 +447,7 @@ A broadcast demultiplexed from a DVB transport stream: video and audio described - Added the `m2ts` section, a transport stream carried whole, with its mapping onto MSFTS; the comparison uses MSFTS's `mpeg2ts` names. - Added the `pesPacket` record: each frame one whole PES packet, timestamped with its placement. A consumer that predates it drops the track rather than refusing the catalog. - A flagged PCR jump on an `m2ts` track follows hang's discontinuity marker rather than a skipped group sequence. +- An `m2ts` track's fallback cut falls on whole seconds of a PCR count whose wraps follow the catalog clock, and `randomAccess` and `muxRate` describe the latest two-second window rather than the whole track, so publishers sharing a clock cut and describe the multiplex alike. ## draft-lcurley-moq-mpegts-00 {:numbered="false"} diff --git a/quest/m2/README.md b/quest/m2/README.md index 0b4c6a04840..6b39f3e6ec9 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -26,7 +26,6 @@ ranked just below deterministic TS legs, which it requires. - [Release execution](/quest/m2/release/README.md) - publication, installed-artifact proof, cleanup, and downstream adoption after m1 - [P2P](/quest/m2/p2p/README.md) - application-selected peer sessions use shared priorities, routing, and same-epoch handover - [Epoch anchor](/quest/m2/epoch-anchor.md) - same-epoch importers each derive the catalog clock and timestamp era from the epoch and stay live, so they agree across a 33-bit wrap, a forward clock break, and a restart -- [Raw TS cuts on the epoch timeline](/quest/m2/ts-passthrough-epoch-cuts.md) - passthrough fallback cuts fall on the epoch's grid and the M2ts section derives from the epoch and content, so instances started apart publish identical groups and catalogs - [Same-epoch importers](/quest/m2/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives - [Hard-fork the binding generator](/quest/m2/uniffi-fork.md) - one moq-dev repo holds the chosen generator's core and every language backend, released under one tag and one pin - [Benchmark trend site](/quest/m2/bench-site.md) - a GitHub Pages site charts the binding, Rust, and JS nightly benchmarks over time diff --git a/quest/m2/epoch-anchor.md b/quest/m2/epoch-anchor.md index d6db4404c9e..efc119070ee 100644 --- a/quest/m2/epoch-anchor.md +++ b/quest/m2/epoch-anchor.md @@ -80,10 +80,8 @@ precondition. `Clock::wall_clock` never enters it. - Every catalog field an importer derives from its history rather than the content at hand derives from the epoch and the content instead (maintainer, - superseding #5412's adoption from the live catalog). In TS passthrough that - is the M2ts section's `random_access` and recorded `mux_rate`, owned by - [raw TS cuts](/quest/m2/ts-passthrough-epoch-cuts.md); audit the other - importers for more. + superseding #5412's adoption from the live catalog). TS passthrough's M2ts + section already does; audit the other importers for more. - A restart under the same epoch re-derives the same coordinate, including after a total outage, so the epoch keeps naming one timeline and resuming across the outage is a gap, not a stitch (maintainer, reversing #5412's diff --git a/quest/m2/hop-aligned-import.md b/quest/m2/hop-aligned-import.md index 81e686c5795..7b68bb78af2 100644 --- a/quest/m2/hop-aligned-import.md +++ b/quest/m2/hop-aligned-import.md @@ -76,7 +76,6 @@ stopping and restarting; repeat it with `--passthrough`. ## Required - [Epoch anchor](/quest/m2/epoch-anchor.md) - every importer derives one catalog clock and timestamp era from the epoch -- [Raw TS cuts on the epoch timeline](/quest/m2/ts-passthrough-epoch-cuts.md) - passthrough groups start where every instance cuts ## Related diff --git a/quest/m2/ts-passthrough-epoch-cuts.md b/quest/m2/ts-passthrough-epoch-cuts.md deleted file mode 100644 index 3f109700691..00000000000 --- a/quest/m2/ts-passthrough-epoch-cuts.md +++ /dev/null @@ -1,67 +0,0 @@ -# [S] Raw TS groups and their catalog section follow the epoch timeline - -## Goal - -Two `moq import ts --passthrough` instances sharing one `--epoch` and fed -one stream cut identical groups wherever each started, including on a -source that never marks random access and on a multi-program stream. -Objects already run from PCR to PCR, so only group starts differ: the -fallback cut counts `FALLBACK` from the current group's start, and before -the first group from when the importer began waiting, so on a source with no -random access points each importer keeps its own group phase for good. -Their catalogs' M2ts sections match too, though both fields depend on -history today: one non-random-access cut clears `random_access` for good, -and the recorded `mux_rate` is judged over windows that start at the -importer's first PCR. - -## Plan - -Decided 2026-10-13 by the maintainer: every group start is a function of -the content and the [epoch's timeline](/quest/m2/epoch-anchor.md), never of -when the importer started. - -- Fallback cuts fall on a grid of the epoch timeline, so an importer started - later first cuts where its partner did. A grid on the raw 27 MHz count will - not do: a 33-bit wrap is not a whole number of seconds, so such a grid - shifts by about 0.72 s at each wrap. -- An importer's first group starts only where a partner running since - before it would also cut: a random access point, a flagged forward break, - or the grid. -- Settle the grid rule so that a source marking random access more often - than `FALLBACK` is never cut away from its random access points, since any - other start publishes `random_access` false. -- Size cuts count from the group's start, so they follow once group starts - do. -- A cut is a position in the content, so under a shared epoch it follows - media time alone. Decided 2026-10-13 by the maintainer as an explicit - exception to timers advancing on max(wall, pts), because a wall-clock cut - would split a stalled instance from its partner. Rejected: keeping a wall - path. -- The M2ts section derives from the epoch and the content, never from the - live catalog or what came before the importer started (maintainer, - superseding #5412's adoption by a joiner). Today both fields are sticky - history. Settle the rule; recommended: each is a function of a window on - the epoch's grid (the rate measured over it, `random_access` true while its - cuts all fell on random access points), and an importer publishes its - catalog only once it has a whole window, so it never publishes a section - its partner did not. Rejected in #5412 and still out: conservative - constants under a shared epoch, which lose `random_access` and the rate - `--sync` uses. - -Tests: two instances started at different offsets into one capture publish -identical groups and objects from the later one's first group, and a -restarted instance rejoins identically. Cover a -source with no random access indicator, a multi-program stream, groups that -hit the size cap, the later instance started after a 33-bit wrap the -earlier crossed, and one started after the earlier made a fallback cut and -recorded a rate; their catalogs match. - -Public API: none. - -## Required - -- [Epoch anchor](/quest/m2/epoch-anchor.md) - the timeline the grid sits on - -## Related - -- [Same-epoch importers](/quest/m2/hop-aligned-import.md) - group sequences and the end-to-end pair diff --git a/rs/hang/src/catalog/m2ts.rs b/rs/hang/src/catalog/m2ts.rs index 6857f9ed319..de558c02888 100644 --- a/rs/hang/src/catalog/m2ts.rs +++ b/rs/hang/src/catalog/m2ts.rs @@ -18,8 +18,9 @@ pub struct M2ts { /// The track carrying the packets, published in the same broadcast as the catalog. pub track: String, - /// Whether every group starts at a random access point of the multiplex's one video stream, - /// so a subscriber can decode from the first object of any group. + /// Whether every group the publisher's latest two-second window started began at a random + /// access point of the multiplex's video stream, or of its clock PID when it has none, so a + /// subscriber can decode from the first object of any of them. /// /// Only ever true for a single program with at most one video stream: programs or video /// streams sharing a clock can stagger their GOPs, leaving no packet that starts all of them. @@ -27,8 +28,8 @@ pub struct M2ts { #[serde(default)] pub random_access: bool, - /// The rate the source's PCR paced the whole multiplex at, in bits per second, null packets - /// included. Present only while the source holds a constant rate. + /// The rate the source's PCR paced the whole multiplex at over that window, in bits per + /// second, null packets included. Present only while the source holds a constant rate. #[serde(default)] pub mux_rate: Option, } diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index 44c18fa97af..b26cc9b653a 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -1187,24 +1187,32 @@ pub(crate) mod tests { #[tokio::test] async fn ts_passthrough_crosses_a_relay_through_a_flagged_jump() { const KYRION: &[u8] = include_bytes!("../../moq-mux/src/container/ts/test_data/kyrion_mpeg2av_ac3.ts"); - // The capture twice, the second copy's clock twenty seconds ahead of the first, its first - // PCR flagged as the discontinuity. Inside the track's max age, so the recording keeps - // every group. - const STEP: u64 = 20 * 90_000; - let mut second = KYRION.to_vec(); - let mut pcrs = second - .as_chunks_mut::<188>() - .0 - .iter_mut() - .filter(|pkt| pkt[3] & 0x20 != 0 && pkt[4] > 0 && pkt[5] & 0x10 != 0) - .peekable(); - pcrs.peek_mut().expect("a PCR")[5] |= 0x80; - for pkt in pcrs { - let base = pkt[6..11].iter().fold(0u64, |base, byte| base << 8 | u64::from(*byte)) >> 7; - let base = ((base + STEP) & ((1 << 33) - 1)) << 7 | u64::from(pkt[10] & 0x7f); - pkt[6..11].copy_from_slice(&base.to_be_bytes()[3..]); - } - let input = [KYRION, &second].concat(); + // The capture with its clock `seconds` ahead, its first PCR flagged as a discontinuity + // when `flagged`. + let shifted = |seconds: u64, flagged: bool| { + let mut copy = KYRION.to_vec(); + let mut pcrs = copy + .as_chunks_mut::<188>() + .0 + .iter_mut() + .filter(|pkt| pkt[3] & 0x20 != 0 && pkt[4] > 0 && pkt[5] & 0x10 != 0) + .peekable(); + if flagged { + pcrs.peek_mut().expect("a PCR")[5] |= 0x80; + } + for pkt in pcrs { + let base = pkt[6..11].iter().fold(0u64, |base, byte| base << 8 | u64::from(*byte)) >> 7; + let base = ((base + seconds * 90_000) & ((1 << 33) - 1)) << 7 | u64::from(pkt[10] & 0x7f); + pkt[6..11].copy_from_slice(&base.to_be_bytes()[3..]); + } + copy + }; + // The capture, carried straight on by a copy four seconds ahead so a whole catalog window + // follows the first group, then a copy twenty seconds past that one behind a flagged + // discontinuity. Inside the track's max age, so the recording keeps every group. + let lead = [KYRION, &shifted(4, false)].concat(); + let second = shifted(24, true); + let input = [&lead[..], &second].concat(); let _env = crate::test_env::EnvGuard::clear(&["MOQ_CONNECT", "MOQ_HOP"]); let _ = moq_tokio::crypto::install_default(); @@ -1263,10 +1271,10 @@ pub(crate) mod tests { .await .unwrap(); - // The catalog names the track once the first group is out, which the first copy holds; - // the jump is in the second. + // The catalog names the track once a whole window has followed the first group, which + // the lead holds; the jump is in the copy after it. const WAIT: Duration = Duration::from_secs(10); - decoder.decode_chunk(KYRION).unwrap(); + decoder.decode_chunk(&lead).unwrap(); let consumer = tokio::time::timeout(WAIT, remote.consume().routed_broadcast("pt.hang")) .await .unwrap() diff --git a/rs/moq-mux/src/container/ts/mux_rate.rs b/rs/moq-mux/src/container/ts/mux_rate.rs index 84d37401451..2760052e69e 100644 --- a/rs/moq-mux/src/container/ts/mux_rate.rs +++ b/rs/moq-mux/src/container/ts/mux_rate.rs @@ -7,6 +7,10 @@ //! whose samples agree on one value publishes it, and a window that no longer //! agrees clears it, so a VBR source (whose samples never agree) records nothing //! and a rate change lands once rather than as churn. +//! +//! [`Windows`] measures the same rate over each window of a grid on the unwrapped +//! PCR timeline instead, carrying nothing from one window to the next, so importers +//! whose PCRs share a timeline measure the same rate for every window each saw whole. /// The PCR clock, in Hz. const PCR_HZ: u64 = 27_000_000; @@ -15,7 +19,7 @@ const PACKET_BITS: u64 = 188 * 8; /// The 33-bit base times 300 plus the 9-bit extension: the PCR wraps here. pub(super) const PCR_WRAP: u64 = (1 << 33) * 300; /// How much PCR time one window spans before it is judged. -const WINDOW: u64 = 2 * PCR_HZ; +pub(super) const WINDOW: u64 = 2 * PCR_HZ; /// Media time after which a source that has not completed a window publishes without a rate: /// twice the window, so a usable PCR clock always completes one first. pub(super) const UNMEASURED: std::time::Duration = std::time::Duration::from_secs(2 * WINDOW / PCR_HZ); @@ -35,6 +39,10 @@ const AGREEMENT: u64 = 20; /// How far a stable window must move before the published rate follows it, in /// thousandths, so measurement noise never touches the catalog. const DRIFT: u64 = 10; +/// The bits per second a [`Windows`] rate is rounded to: far coarser than a PCR +/// clock's jitter over a window, so a constant-rate source measures one value +/// window after window. +const STEP: u64 = 1_000; /// Packets over the PCR ticks they were paced across. #[derive(Clone, Copy, Default)] @@ -169,6 +177,111 @@ impl Meter { } } +/// The multiplex rate over each [`WINDOW`] of a grid on the unwrapped PCR timeline. +/// +/// Window `k` holds the PCR intervals that start in `[k * WINDOW, (k + 1) * WINDOW)`, +/// pooled into samples on a grid of [`SAMPLE`]. It is measured only if the intervals +/// since its start all arrived unbroken: no flagged jump, and none empty or longer +/// than [`MAX_INTERVAL`]. Its rate, rounded to [`STEP`], stands when every sample +/// agrees with it. +#[derive(Default)] +pub(super) struct Windows { + /// Packets since the last PCR, every PID and null stuffing included. + packets: u64, + /// The last PCR, unwrapped. + last: Option, + /// Whether the window in progress missed an interval: one started in it broken, or + /// the run of intervals began after its start. + spoiled: bool, + /// The window the last PCR falls in. + window: u64, + /// The window's samples, each with its index on the sample grid. + samples: Vec<(u64, Interval)>, +} + +/// A window the PCR clock has passed. +pub(super) struct Ended { + /// The window's index on the grid: it starts at `index * WINDOW` ticks. + pub index: u64, + /// The window's rate, if it was measured whole and its samples agree. + pub rate: Option, +} + +impl Windows { + /// One transport packet went by. + pub fn packet(&mut self) { + self.packets += 1; + } + + /// A PCR at `ticks` on the unwrapped timeline, carried by the packet just counted, + /// `broken` when a flagged jump brought it. Returns the window it ended, if it falls + /// in a later one; the windows a jump skips held nothing and end unreported. + pub fn pcr(&mut self, ticks: u64, broken: bool) -> Option { + let packets = std::mem::take(&mut self.packets); + let Some(last) = self.last.replace(ticks) else { + self.spoiled = !ticks.is_multiple_of(WINDOW); + self.window = ticks / WINDOW; + return None; + }; + let interval = Interval { + packets, + ticks: ticks.saturating_sub(last), + }; + let bad = broken || interval.ticks == 0 || interval.ticks > MAX_INTERVAL; + if bad { + self.spoiled = true; + } else { + let index = last / SAMPLE; + match self.samples.last_mut() { + Some((at, sample)) if *at == index => sample.add(interval), + _ => self.samples.push((index, interval)), + } + } + if ticks / WINDOW == self.window { + return None; + } + let ended = Ended { + index: self.window, + rate: self.rate(&self.samples), + }; + self.window = ticks / WINDOW; + // A broken interval reaching into this window covered its start. + self.spoiled = bad && !ticks.is_multiple_of(WINDOW); + self.samples.clear(); + Some(ended) + } + + /// The rate of the window in progress, over its completed samples, if there are at + /// least two: for an input that ends before its first window does. + pub fn partial(&self) -> Option { + let in_progress = self.last.map(|last| last / SAMPLE); + let completed = match self.samples.last() { + Some((at, _)) if Some(*at) == in_progress => &self.samples[..self.samples.len() - 1], + _ => &self.samples[..], + }; + if completed.len() < 2 { + return None; + } + self.rate(completed) + } + + fn rate(&self, samples: &[(u64, Interval)]) -> Option { + if self.spoiled { + return None; + } + let mut whole = Interval::default(); + samples.iter().for_each(|(_, s)| whole.add(*s)); + if whole.ticks == 0 { + return None; + } + let rate = whole.rate(); + samples + .iter() + .all(|(_, s)| s.agrees(rate)) + .then(|| (rate + STEP / 2) / STEP * STEP) + } +} + #[cfg(test)] mod test { use super::*; @@ -277,6 +390,106 @@ mod test { assert_eq!(meter.published(), None); } + /// Feed `windows` the PCRs `intervals` of (packets, ticks) lead to from `pcrs`'s last, + /// returning each window ended as (index, rate). + fn feed_windows( + windows: &mut Windows, + pcrs: &mut Vec, + intervals: impl IntoIterator, + ) -> Vec<(u64, Option)> { + let mut ended = Vec::new(); + for (packets, ticks) in intervals { + for _ in 0..packets { + windows.packet(); + } + let pcr = pcrs.last().map_or(0, |last| last + ticks); + pcrs.push(pcr); + ended.extend(windows.pcr(pcr, false).map(|window| (window.index, window.rate))); + } + ended + } + + #[test] + fn a_window_is_measured_alone() { + let mut first = Windows::default(); + let mut pcrs = Vec::new(); + let ended = feed_windows(&mut first, &mut pcrs, std::iter::repeat_n(CBR, PER_WINDOW * 5)); + assert_eq!(ended.len(), 5); + assert!(ended.iter().all(|&(_, rate)| rate == Some(2_500_000)), "{ended:?}"); + + // A meter started part-way through the second window measures the windows after it the same. + let mut second = Windows::default(); + let join = PER_WINDOW * 3 / 2; + let mut replay = vec![pcrs[join]]; + second.packet(); + assert!(second.pcr(pcrs[join], false).is_none()); + let joined = feed_windows( + &mut second, + &mut replay, + std::iter::repeat_n(CBR, PER_WINDOW * 5 - join - 1), + ); + assert_eq!(joined[0], (1, None), "the window it joined part-way through"); + assert_eq!(joined[1..], ended[2..]); + + // A file-paced source's samples never agree. + let mut vbr = Windows::default(); + let intervals = (0..PER_WINDOW as u64 * 3).map(|i| (if (i / 7) % 2 == 0 { 10 } else { 60 }, 714_701)); + let ended = feed_windows(&mut vbr, &mut Vec::new(), intervals); + assert!(ended.iter().all(|&(_, rate)| rate.is_none()), "{ended:?}"); + } + + #[test] + fn a_break_spoils_only_its_own_window() { + let mut windows = Windows::default(); + let mut pcrs = Vec::new(); + // Half a window, then a clock stalled past the longest interval, still inside window 0. + let stall = (100, MAX_INTERVAL + 1); + let intervals = std::iter::repeat_n(CBR, PER_WINDOW / 2).chain(std::iter::once(stall)); + let mut ended = feed_windows(&mut windows, &mut pcrs, intervals); + ended.extend(feed_windows( + &mut windows, + &mut pcrs, + std::iter::repeat_n(CBR, PER_WINDOW * 2), + )); + // A flagged jump ten windows forward, landing part-way into window 13. + windows.packet(); + let jump = pcrs.last().unwrap() + 10 * WINDOW; + ended.extend(windows.pcr(jump, true).map(|window| (window.index, window.rate))); + pcrs.push(jump); + ended.extend(feed_windows( + &mut windows, + &mut pcrs, + std::iter::repeat_n(CBR, PER_WINDOW * 2), + )); + let measured = Some(2_500_000); + assert_eq!( + ended, + vec![ + (0, None), + (1, measured), + (2, measured), + (3, None), + (13, None), + (14, measured) + ], + "only the windows the run broke in, or began part-way through, go unmeasured" + ); + } + + #[test] + fn an_empty_interval_on_a_line_spoils_its_window() { + let mut windows = Windows::default(); + let mut pcrs = Vec::new(); + // Fifty intervals a window, so a PCR lands on every line, then that PCR repeated. + let paced = (44, WINDOW / 50); + let mut ended = feed_windows(&mut windows, &mut pcrs, std::iter::repeat_n(paced, 51)); + ended.extend(feed_windows(&mut windows, &mut pcrs, [(44, 0)])); + ended.extend(feed_windows(&mut windows, &mut pcrs, std::iter::repeat_n(paced, 100))); + let measured = ended[0].1; + assert!(measured.is_some()); + assert_eq!(ended, vec![(0, measured), (1, None), (2, measured)]); + } + #[test] fn survives_the_pcr_wrap() { let mut meter = Meter { diff --git a/rs/moq-mux/src/container/ts/passthrough.rs b/rs/moq-mux/src/container/ts/passthrough.rs index 0743e08e1cc..c07b7421626 100644 --- a/rs/moq-mux/src/container/ts/passthrough.rs +++ b/rs/moq-mux/src/container/ts/passthrough.rs @@ -14,21 +14,39 @@ //! program has no video; //! - a PCR with `discontinuity_indicator` that jumps forward, since no group may hold a PCR //! discontinuity; -//! - the first PCR at least [`FALLBACK`] after the group's start, for a source that never marks -//! random access; +//! - the first PCR at or past a line of a grid every [`FALLBACK`] on the PCR timeline, when no +//! random access point or flagged jump fell in the [`FALLBACK`] before the line, for a source +//! that marks random access rarely or never; //! - the first object that would take the group past [`GROUP_BYTES_MAX`] or //! [`GROUP_OBJECTS_MAX`], so a long GOP cannot outgrow what a relay caches. Objects are never //! split to meet the bound. //! -//! Only the first of these makes a group a random access point, so any other start publishes -//! [`random_access`](hang::catalog::M2ts::random_access) false from then on, as does any group -//! carried while the layout allows no random access (more than one program or video stream). A +//! A random access point or a jump counts against the grid at the PCR before it (its own PCR, if +//! its packet carries one). Only the first kind of start makes a group a random access point. A //! group the PCR discontinuity starts is behind a break even when it is also a random access //! point. //! //! Bytes before the first group are dropped: until the PMT, a PCR, and a group start have all -//! been seen there is no group to put them in. The track is byte-identical to the input from its -//! first group on, and only from there. Bytes off the 188-byte packet grid are dropped too. +//! been seen there is no group to put them in. A jump before the first group starts none, and a +//! line of the grid starts it only once the importer has watched for random access from a PCR +//! before the [`FALLBACK`] before the line, since a jump on its first PCR may be one it cannot see +//! (a retransmitted flag, or one in an earlier packet). The track is byte-identical to the input +//! from its first group on, and only from there. Bytes off the 188-byte packet grid are dropped +//! too. +//! +//! # The catalog section +//! +//! The section is written for each window of a grid every two seconds on the PCR timeline, once +//! the clock passes it; a window holds the PCR intervals that start in it, so the packets up to +//! the first PCR past a line, and any layout change among them, count in the window before the +//! line. It is written only for windows that start after the PCR the importer's first group +//! follows: the catalog lists the track from the first such window on, or at the end of the input +//! if that comes first. [`random_access`](hang::catalog::M2ts::random_access) is true when every group +//! the window started is a random access point and the layout allowed random access throughout it +//! (one program, at most one video stream); [`mux_rate`](hang::catalog::M2ts::mux_rate) is the rate the +//! window was paced at, when the PCRs arrived unbroken across it and agree on one. An input that +//! ends before its first window publishes the random access of everything it carried and the rate +//! of the window in progress. //! //! # Timestamps //! @@ -48,10 +66,11 @@ //! one before ends the import: [`PcrRewindError`] without the flag, [`PcrRestartError`] with it, //! since a signalled restart is new content and needs a new broadcast. So does a jump that carries //! the PCR more than half a wrap (about 13.25 hours) from the clock's reading, since it lands a -//! wrap back. The timeline therefore depends only on the bytes and the clock, and two importers of -//! one feed whose timebases one shared epoch placed (see -//! [`Timebase::place`](crate::catalog::Timebase::place)) publish identical objects wherever each -//! started. Objects timed past the last PCR before a flagged jump smaller than their lead can land +//! wrap back. The timeline therefore depends only on the bytes and the clock, and the grids above +//! lie on it, so two importers of one feed whose timebases one shared epoch placed (see +//! [`Timebase::place`](crate::catalog::Timebase::place)) publish identical objects, groups and +//! sections wherever each started, from the later one's first group on (group sequence numbers +//! aside). Objects timed past the last PCR before a flagged jump smaller than their lead can land //! after the first ones behind it; if one of them starts a group, the group behind the break would //! start before it, and the import ends there. @@ -61,7 +80,7 @@ use bytes::BytesMut; use mpeg2ts::ts::{Pid, TsPacket}; use super::import::{Framer, PatReader, SectionReassembler, adaptation_valid, discontinuity_indicator, pcr}; -use super::mux_rate::{Meter, PCR_WRAP}; +use super::mux_rate::{PCR_WRAP, WINDOW, Windows}; use super::psi::{self, Pmt}; /// The PCR clock, in Hz. @@ -73,7 +92,8 @@ const PCR: crate::catalog::Wrap = crate::catalog::Wrap { period: PCR_WRAP, }; -/// The longest a group runs without a random access point before a PCR starts the next. +/// The spacing of the grid that cuts a source without frequent random access points, so no group +/// runs much past two spacings. pub const FALLBACK: std::time::Duration = std::time::Duration::from_secs(1); const FALLBACK_TICKS: u64 = PCR_HZ; @@ -179,8 +199,8 @@ pub struct PcrGapError { /// timestamped. pub struct Passthrough { catalog: crate::catalog::Producer, - /// Held until the first group is written, so the first catalog already names the track with - /// its final `randomAccess`. + /// Held until the section is first written, so the first catalog already describes a whole + /// window. reservation: Option>, /// Puts every PCR in its era and shifts every object's PCR time onto the catalog clock, /// anchored by the first PCR. @@ -219,19 +239,26 @@ pub struct Passthrough { /// A `discontinuity_indicator` on the clock PID, waiting for the PCR that it flags (normally /// in the same packet). pending_break: bool, - /// Before the first group: the PCR time the importer started waiting for a group start at. - waiting_since: Option, + /// The first PCR at which the clock's program was mapped: every random access point since + /// has been seen. + watching_since: Option, + /// The PCR the latest random access point or flagged jump counts against the grid at. + last_mark: Option, /// The object being filled, from its first packet. open: Option, /// Closed objects waiting on the next PCR for their timestamps. queue: VecDeque, group: Option, + /// The PCR the first group started at. + first_mark: Option, /// Whether every group so far started at a random access point, and the layout allowed random - /// access throughout. + /// access throughout: the section of an input that ends before its first window. all_random_access: bool, + /// The same, for the window in progress. + window_random_access: bool, - mux_rate: Meter, + windows: Windows, } /// Owns the `m2ts` catalog section once written, removing it however the track ends, as the @@ -273,6 +300,8 @@ struct Sample { struct Object { /// Where the object's first byte sits in the input. at: u64, + /// The PCR at or before the object's first packet, which places a group it starts on the grids. + mark: u64, payload: BytesMut, /// Why this object starts a group, if it does. start: Option, @@ -290,9 +319,8 @@ enum Start { Full, } +#[derive(Default)] struct Group { - /// The PCR time of the group's first byte. - ticks: u64, bytes: u64, objects: usize, } @@ -329,12 +357,15 @@ impl Passthrough { mapped: false, timebase: None, pending_break: false, - waiting_since: None, + watching_since: None, + last_mark: None, open: None, queue: VecDeque::new(), group: None, + first_mark: None, all_random_access: true, - mux_rate: Meter::default(), + window_random_access: false, + windows: Windows::default(), }) } @@ -375,6 +406,9 @@ impl Passthrough { } self.close_object(); self.drain_queue()?; + if self.section.is_none() && self.group.is_some() { + self.record(self.single && self.all_random_access, self.windows.partial())?; + } self.reservation = None; self.media.finish()?; Ok(()) @@ -461,7 +495,7 @@ impl Passthrough { { self.pmt_packet(pmt_pid, pkt)?; } - self.mux_rate.packet(); + self.windows.packet(); // A packet the demodulator flagged corrupt, or whose adaptation field overruns it, is // carried like any other, but nothing is read from it. @@ -481,6 +515,7 @@ impl Passthrough { } if readable && self.access_pid == Some(pid) && random_access_indicator(pkt) && self.can_start() { start = Some(Start::RandomAccess); + self.last_mark = self.last_pcr(); } if start.is_some() || (clock && self.started()) { @@ -491,6 +526,7 @@ impl Passthrough { payload.extend_from_slice(pkt); self.open = Some(Object { at, + mark: self.last_pcr().expect("an object follows a PCR"), payload, start, discontinuity, @@ -525,10 +561,9 @@ impl Passthrough { raw, samples: VecDeque::from([Sample { at, ticks }]), }); - self.waiting_since = self.mapped.then_some(ticks); - if self.mux_rate.pcr(raw) { - self.record()?; - } + self.watching_since = self.mapped.then_some(ticks); + self.windows.pcr(ticks, false); + self.window_random_access = self.single; return Ok(None); }; @@ -551,16 +586,10 @@ impl Passthrough { raw, samples: VecDeque::from([Sample { at, ticks }]), }); - if self.mux_rate.discontinuity() { - self.record()?; - } - if self.mux_rate.pcr(raw) { - self.record()?; - } - // The jump is not elapsed time, so the wait for the first group starts over. - if self.waiting_since.is_some() { - self.waiting_since = Some(ticks); - } + // An importer already running cuts here, so the jump counts against the grid even + // before this one's first group. + self.last_mark = Some(ticks); + self.end_window(ticks, true)?; return Ok(self.started().then_some(Start::Discontinuity)); } @@ -570,18 +599,41 @@ impl Passthrough { timebase.samples.pop_front(); } self.drain_queue()?; - if self.mux_rate.pcr(raw) { - self.record()?; + self.end_window(ticks, false)?; + if self.mapped { + self.watching_since.get_or_insert(ticks); } - let since = match &self.group { - Some(group) => Some(group.ticks), - None if self.mapped => Some(*self.waiting_since.get_or_insert(ticks)), - None => None, + // The first PCR at or past a line of the grid cuts there, unless a random access point or + // a jump fell in the spacing before the line. Before the first group, only a line whose + // spacing was watched from a PCR before it may: an importer running since before would cut + // there too, and a jump on the first PCR watched may be one this importer could not see. + let line = ticks / FALLBACK_TICKS * FALLBACK_TICKS; + let crossed = last < line; + let marked = self.last_mark.is_some_and(|mark| mark + FALLBACK_TICKS >= line); + let watched = self.started() || self.watching_since.is_some_and(|since| since + FALLBACK_TICKS < line); + Ok((crossed && !marked && watched && self.can_start()).then_some(Start::Fallback)) + } + + /// The section for the window a PCR at `ticks` ended, if it ended one: written once the window + /// starts after the PCR the first group follows, since every group start and layout change in + /// it is then one an importer running since before saw the same way. A window starting on that + /// PCR is not: the partner may have cut earlier in the same PCR interval. + fn end_window(&mut self, ticks: u64, broken: bool) -> anyhow::Result<()> { + let Some(ended) = self.windows.pcr(ticks, broken) else { + return Ok(()); }; - Ok(since - .filter(|since| ticks.saturating_sub(*since) >= FALLBACK_TICKS && self.can_start()) - .map(|_| Start::Fallback)) + let random_access = std::mem::replace(&mut self.window_random_access, self.single); + if self.first_mark.is_some_and(|mark| mark < ended.index * WINDOW) { + self.record(random_access, ended.rate)?; + } + Ok(()) + } + + /// The PCR the last packet followed, on the unwrapped timeline. + fn last_pcr(&self) -> Option { + let timebase = self.timebase.as_ref()?; + timebase.samples.back().map(|sample| sample.ticks) } /// Whether a group may start here: once the first one has, always; before it, only once the @@ -628,12 +680,10 @@ impl Passthrough { } if start != Start::RandomAccess || !self.single { self.all_random_access = false; + self.window_random_access = false; } - self.group = Some(Group { - ticks, - bytes: 0, - objects: 0, - }); + self.first_mark.get_or_insert(object.mark); + self.group = Some(Group::default()); } // Only an object that starts a group is ever opened before the first one. @@ -646,25 +696,19 @@ impl Passthrough { payload: object.payload.freeze(), keyframe: start.is_some(), })?; - if start.is_some() { - self.record()?; - self.reservation = None; - } Ok(()) } - /// Write the section to the catalog if it changed. Nothing is written before the first group. - fn record(&mut self) -> anyhow::Result<()> { - if self.group.is_none() && self.section.is_none() { - return Ok(()); - } + /// Write the section to the catalog if it changed, and let the catalog publish. + fn record(&mut self, random_access: bool, mux_rate: Option) -> anyhow::Result<()> { let mut section = hang::catalog::M2ts::new(self.media.name()); - section.random_access = self.single && self.all_random_access; - section.mux_rate = self.mux_rate.published(); + section.random_access = random_access; + section.mux_rate = mux_rate; if self.section.as_ref().is_some_and(|written| written.written == section) { return Ok(()); } self.catalog.mutate(|catalog| catalog.m2ts = Some(section.clone()))?; + self.reservation = None; match &mut self.section { Some(written) => written.written = section, None => { @@ -777,11 +821,19 @@ impl Passthrough { self.access_pid = access_pid; self.mapped = mapped; self.single = single; - // The group in progress now carries what no random access point covers, and it stays cached. - if !single && self.started() { - self.all_random_access = false; + if !single { + self.window_random_access = false; + // The group in progress now carries what no random access point covers. + if self.started() { + self.all_random_access = false; + } + } + // Unmapped, an importer that has not started misses random access points one that has + // sees, so its watch starts over. + if !mapped && !self.started() { + self.watching_since = None; } - self.record() + Ok(()) } } diff --git a/rs/moq-mux/src/container/ts/passthrough_test.rs b/rs/moq-mux/src/container/ts/passthrough_test.rs index 88d7afa5034..f12742b81ad 100644 --- a/rs/moq-mux/src/container/ts/passthrough_test.rs +++ b/rs/moq-mux/src/container/ts/passthrough_test.rs @@ -81,21 +81,6 @@ async fn import_onto( import_on(broadcast, catalog, timebase, data, chunk, pcr_pid).await } -/// [`import`] on a host whose wall clock reads `host`, its clock placed by a shared epoch at -/// `epoch`: PCR zero one wrap before it. -async fn import_placed( - data: &[u8], - epoch: std::time::SystemTime, - host: std::time::SystemTime, -) -> anyhow::Result { - crate::clock::fake_host_wall(host); - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let catalog = crate::catalog::Producer::new(&mut broadcast, Default::default())?; - let timebase = catalog.timebase(); - timebase.place(moq_net::Timestamp::ZERO, epoch - Duration::from_micros(WRAP_MICROS))?; - import_on(broadcast, catalog, timebase, data, data.len(), None).await -} - /// [`import_onto`] on `timebase`, one of `catalog`'s. async fn import_on( broadcast: moq_net::broadcast::Producer, @@ -115,9 +100,17 @@ async fn import_on( passthrough.decode(chunk)?; } passthrough.finish()?; + read_back(&consumer, &name, &catalog).await +} +/// Every object of the finished track `name`, and the catalog as it stands. +async fn read_back( + consumer: &moq_net::broadcast::Consumer, + name: &str, + catalog: &crate::catalog::Producer, +) -> anyhow::Result { let mut track = consumer - .track(&name)? + .track(name)? .subscribe(moq_net::track::Subscription::default().with_max_delay(RECORDING_MAX_DELAY)) .await?; let mut objects = Vec::new(); @@ -361,21 +354,71 @@ async fn two_programs_on_one_clock_publish_no_random_access() { } } -/// Groups started while a second program was on air stay cached, so dropping back to one program -/// does not win random access back. +/// The window a second program was on air in publishes no random access, and the first window +/// carried whole on one program wins it back. #[tokio::test(start_paused = true)] -async fn random_access_lost_to_a_second_program_stays_lost() { +async fn random_access_returns_once_a_window_carries_one_program() { let mut mux = Mux::default(); mux.pat(&[(1, 0x20), (2, 0x30)]) .pmt(0x20, 1, 0x100, &[(0x1b, 0x100)], 0) .pmt(0x30, 2, 0x100, &[(0x1b, 0x200)], 0); paced(&mut mux, 0x100, 0, 30, 10, 4); program(&mut mux, 0x100); - paced(&mut mux, 0x100, 1_200 * MS, 30, 10, 4); + paced(&mut mux, 0x100, 1_200 * MS, 100, 10, 4); let data = mux.bytes(); + let cut = pcr_at(&data, 3_000 * MS); + let early = import(&data[..cut], cut, None).await.unwrap(); + assert!(!early.section().random_access, "the window with two programs"); let imported = import(&data, data.len(), None).await.unwrap(); - assert!(imported.groups().len() >= 6); - assert!(!imported.section().random_access); + assert!(imported.groups().len() >= 12); + assert!(imported.section().random_access, "the window after it"); +} + +/// A window holds the PCR intervals that start in it: a second program carried past a line, up to +/// the first PCR after it, counts in the window before the line, and the window from the line on +/// carries one program throughout. +#[tokio::test(start_paused = true)] +async fn a_layout_change_before_the_first_pcr_past_a_line_counts_before_it() { + let mut mux = Mux::default(); + mux.pat(&[(1, 0x20), (2, 0x30)]) + .pmt(0x20, 1, 0x100, &[(0x1b, 0x100)], 0) + .pmt(0x30, 2, 0x100, &[(0x1b, 0x200)], 0); + // The last PCR before the 2 s line is at 1.98 s; most of the interval after it, program 2's + // packets included, lies past the line, and the PAT drops program 2 before the 2.02 s PCR. + paced(&mut mux, 0x100, 20 * MS, 50, 10, 4); + mux.fill(0x200, 20); + program(&mut mux, 0x100); + paced(&mut mux, 0x100, 2_020 * MS, 60, 10, 4); + let data = mux.bytes(); + let end = pcr_at(&data, 4_020 * MS); + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = crate::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let mut passthrough = Passthrough::new(broadcast.clone(), catalog.reserve()).unwrap(); + passthrough.decode(&data[..end + 188]).unwrap(); + let section = catalog.snapshot().m2ts.expect("the window from 2 s has passed"); + assert!(section.random_access, "the window from the line carried one program"); +} + +/// The section waits for the first window that starts at or after the first group, so the catalog +/// first lists the track with a whole window behind it. +#[tokio::test(start_paused = true)] +async fn the_catalog_waits_for_a_whole_window() { + let mut mux = Mux::default(); + program(&mut mux, 0x100); + // The first group starts at 0.52 s, so the window from 2 s is the first after it. + paced(&mut mux, 0x100, 520 * MS, 120, 10, 4); + let data = mux.bytes(); + let end = pcr_at(&data, 4_000 * MS); + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = crate::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let mut passthrough = Passthrough::new(broadcast.clone(), catalog.reserve()).unwrap(); + passthrough.decode(&data[..end]).unwrap(); + assert!(catalog.snapshot().m2ts.is_none(), "listed before a whole window"); + passthrough.decode(&data[end..end + 188]).unwrap(); + let section = catalog.snapshot().m2ts.expect("listed once the window passed"); + assert!(section.random_access); + // 250 packets a window, and 5 random access points: 255 * 1504 bits over two seconds. + assert_eq!(section.mux_rate, Some(192_000)); } #[tokio::test(start_paused = true)] @@ -448,8 +491,9 @@ async fn a_source_without_random_access_groups_every_second() { let imported = import(&data, data.len(), None).await.unwrap(); assert!(!imported.section().random_access); let groups = imported.groups(); - // The first group waits a second for a random access point that never comes. - assert_eq!(groups[0][0].micros, 1_000_000 - micros_before_pcr(&data, 1_000_000)); + // The first group waits for a random access point that never comes. The first PCR sits on a + // line, so the first line watched from a PCR before its second is at 2 s. + assert_eq!(groups[0][0].micros, 2_000_000 - micros_before_pcr(&data, 2_000_000)); for group in &groups { assert_eq!(group.len(), 25, "a second of 40 ms objects"); } @@ -528,11 +572,15 @@ async fn objects_after_the_last_pcr_extrapolate_its_rate() { /// The section describes the track for as long as the importer lives, however the track ends. #[tokio::test(start_paused = true)] async fn the_section_goes_with_the_importer() { + let mut mux = Mux::default(); + program(&mut mux, 0x100); + paced(&mut mux, 0x100, 0, 125, 10, 4); + let data = mux.bytes(); for abort in [false, true] { let mut broadcast = moq_net::broadcast::Info::new().produce(); let catalog = crate::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); let mut passthrough = Passthrough::new(broadcast.clone(), catalog.reserve()).unwrap(); - passthrough.decode(KYRION_H264).unwrap(); + passthrough.decode(&data).unwrap(); assert!(catalog.snapshot().m2ts.is_some(), "abort {abort}"); if abort { passthrough.abort(moq_net::Error::Cancel); @@ -613,30 +661,91 @@ fn pcr_at(data: &[u8], ticks: u64) -> usize { packet.expect("a packet carrying that PCR") * 188 } -/// Import `data` placed by `epoch` from its first byte, and again from the program and then the -/// packet carrying PCR `join`, three seconds later on a host clock 7 ms ahead. The second must -/// publish exactly the first's objects from its first group on. +/// A passthrough importer whose clock a shared epoch at `epoch` placed: PCR zero one wrap before it. +struct Placed { + broadcast: moq_net::broadcast::Consumer, + catalog: crate::catalog::Producer, + passthrough: Passthrough, + name: String, +} + +impl Placed { + fn new(epoch: std::time::SystemTime) -> Self { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = crate::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let timebase = catalog.timebase(); + timebase + .place(moq_net::Timestamp::ZERO, epoch - Duration::from_micros(WRAP_MICROS)) + .unwrap(); + let passthrough = Passthrough::new(broadcast.clone(), timebase.reserve()).unwrap(); + Self { + broadcast: broadcast.consume(), + catalog, + name: passthrough.track().to_string(), + passthrough, + } + } + + async fn finish(mut self) -> Imported { + self.passthrough.finish().unwrap(); + read_back(&self.broadcast, &self.name, &self.catalog).await.unwrap() + } +} + +/// Whether each object starts a group. +fn starts(imported: &Imported) -> Vec { + let objects = &imported.objects; + (0..objects.len()) + .map(|i| i == 0 || objects[i].group != objects[i - 1].group) + .collect() +} + +/// Import `data` placed by one epoch from its first byte, and again, on a host clock 7 ms ahead, +/// from the program and then the packet carrying PCR `join`, the two fed in step. From the second's +/// first group on, both must publish the same objects in the same groups, and from the second's +/// first section on, the same section after every PCR. async fn import_twice_placed(data: &[u8], prefix: usize, join: u64) -> (Imported, Imported) { + import_twice_placed_from(data, prefix, pcr_at(data, join)).await +} + +/// [`import_twice_placed`], the second importer joining at byte `split`. +async fn import_twice_placed_from(data: &[u8], prefix: usize, split: usize) -> (Imported, Imported) { let epoch = epoch_time(); - let start = epoch + Duration::from_secs(600); - let first = import_placed(data, epoch, start).await.unwrap(); - let mut joined = data[..prefix].to_vec(); - joined.extend_from_slice(&data[pcr_at(data, join)..]); - tokio::time::advance(Duration::from_secs(3)).await; - let host = start + Duration::from_millis(3_007); - let second = import_placed(&joined, epoch, host).await.unwrap(); + let hosts = [epoch + Duration::from_secs(600), epoch + Duration::from_millis(600_007)]; + crate::clock::fake_host_wall(hosts[0]); + let mut first = Placed::new(epoch); + first.passthrough.decode(&data[..split]).unwrap(); + crate::clock::fake_host_wall(hosts[1]); + let mut second = Placed::new(epoch); + second.passthrough.decode(&data[..prefix]).unwrap(); + let mut described = false; + for pkt in data[split..].chunks(188) { + crate::clock::fake_host_wall(hosts[0]); + first.passthrough.decode(pkt).unwrap(); + crate::clock::fake_host_wall(hosts[1]); + second.passthrough.decode(pkt).unwrap(); + let Some(pcr) = header(pkt).pcr else { continue }; + let section = second.catalog.snapshot().m2ts; + described |= section.is_some(); + if described { + assert_eq!(section, first.catalog.snapshot().m2ts, "the sections after PCR {pcr}"); + } + } + assert!(described, "the joiner described no window"); + let (first, second) = (first.finish().await, second.finish().await); assert_eq!(first.catalog.clock, Some(epoch_clock(epoch))); assert_eq!(second.catalog.clock, first.catalog.clock); assert!(!second.objects.is_empty(), "the joiner published nothing"); - let tail = &first.objects[first.objects.len() - second.objects.len()..]; - for (index, (first, second)) in tail.iter().zip(&second.objects).enumerate() { + let skip = first.objects.len() - second.objects.len(); + for (index, (first, second)) in first.objects[skip..].iter().zip(&second.objects).enumerate() { assert_eq!( (first.micros, &first.payload), (second.micros, &second.payload), "object {index} of the importer that joined" ); } + assert_eq!(starts(&first)[skip..], starts(&second), "group starts"); (first, second) } @@ -649,7 +758,7 @@ async fn importers_sharing_an_epoch_publish_identical_objects_across_a_wrap() { program(&mut mux, 0x100); let prefix = mux.out.len(); let start = WRAP - 2_000 * MS; - for i in 0..150 { + for i in 0..250 { mux.packet(0x100, Flags::pcr((start + i * 40 * MS) % WRAP)); if i % 10 == 0 { mux.packet(0x100, ACCESS); @@ -674,13 +783,164 @@ async fn importers_sharing_an_epoch_agree_across_a_flagged_jump() { // Flagged three seconds on from the PCR due a second short of the wrap, so the jump crosses it. mux.packet(0x100, Flags::pcr(2_000 * MS).discontinuity()); mux.fill(0x101, 4); - paced(&mut mux, 0x100, 2_040 * MS, 50, 10, 4); + paced(&mut mux, 0x100, 2_040 * MS, 150, 10, 4); let data = mux.bytes(); let (first, _) = import_twice_placed(&data, prefix, 3_000 * MS).await; let micros: Vec = first.objects.iter().map(|object| object.micros).collect(); assert!(micros.windows(2).all(|pair| pair[0] <= pair[1]), "{micros:?}"); } +/// Importers placed by one epoch cut the same groups and describe them alike wherever the second +/// joined, as a restarted one does: on random access points rarer than the grid, on none, and on a +/// clock two programs share. +/// Every join falls after a cut on the grid, and the later ones after the first importer recorded a +/// rate. +#[tokio::test(start_paused = true)] +async fn importers_sharing_an_epoch_cut_the_same_groups() { + // Random access every `gop` PCRs on program 1's video, and how many programs share the clock. + for (gop, programs) in [(36, 1), (0, 1), (36, 2)] { + let mut mux = Mux::default(); + match programs { + 1 => program(&mut mux, 0x100), + _ => { + mux.pat(&[(1, 0x20), (2, 0x30)]) + .pmt(0x20, 1, 0x100, &[(0x1b, 0x100)], 0) + .pmt(0x30, 2, 0x100, &[(0x1b, 0x200)], 0); + } + } + let prefix = mux.out.len(); + for i in 0..400 { + mux.packet(0x100, Flags::pcr(i * 40 * MS)); + if gop != 0 && i % gop == 0 { + mux.packet(0x100, ACCESS); + } + if programs == 2 && i % 36 == 18 { + mux.packet(0x200, ACCESS); + } + mux.fill(0x101, 4); + } + let data = mux.bytes(); + for join in [2_520, 4_720, 7_360] { + let (first, _) = import_twice_placed(&data, prefix, join * MS).await; + let fallback = first + .groups() + .iter() + .any(|group| !header(&group[0].payload).random_access); + assert!(fallback, "gop {gop}: the grid cut"); + if gop == 0 || programs == 2 { + assert!(!first.section().random_access, "gop {gop}, {programs} programs"); + } + assert!(first.section().mux_rate.is_some(), "gop {gop}: a rate was recorded"); + } + } +} + +/// Importers placed by one epoch split groups at the object ceiling alike, the ceiling counted +/// from a group start both made. +#[tokio::test(start_paused = true)] +async fn importers_sharing_an_epoch_split_full_groups_alike() { + let mut mux = Mux::default(); + program(&mut mux, 0x100); + let prefix = mux.out.len(); + mux.packet(0x100, Flags::pcr(0).random_access()); + // A PCR in every packet, 200 µs apart: the ceiling fills before each line of the grid. + for i in 1..60_000 { + mux.packet(0x100, Flags::pcr(i * 5_400)); + } + let data = mux.bytes(); + let (first, second) = import_twice_placed(&data, prefix, 3_300 * MS).await; + let full = second + .groups() + .iter() + .filter(|group| group.len() == GROUP_OBJECTS_MAX) + .count(); + assert!(full >= 3, "{full} groups at the ceiling"); + assert!(!first.section().random_access); +} + +/// A joiner whose first PCR lands exactly on a window's line misses the grid cut its partner made +/// there, so the first window it describes is the one after: describing that window would publish +/// random access the partner lost to the cut. +#[tokio::test(start_paused = true)] +async fn a_joiner_on_a_window_line_describes_the_window_after() { + const WINDOW: u64 = 2_000 * MS; + let mut mux = Mux::default(); + program(&mut mux, 0x100); + let prefix = mux.out.len(); + // The placed epoch puts these PCRs one wrap on, so `offset` lands every fiftieth on a line. + let offset = (WINDOW - WRAP % WINDOW) % WINDOW; + for i in 0..400 { + mux.packet(0x100, Flags::pcr(offset + i * 40 * MS)); + // No random access before 4 s, then one just after the PCR on each whole second. + if i >= 100 && i % 25 == 0 { + mux.packet(0x100, ACCESS); + } + mux.fill(0x101, 4); + } + let data = mux.bytes(); + let (first, second) = import_twice_placed(&data, prefix, offset + 4_000 * MS).await; + assert!( + first.section().random_access, + "every group from 6 s on is a random access point" + ); + assert_eq!(first.section(), second.section()); +} + +/// A joiner whose first PCR is a flagged jump on a line of the grid that it cannot see as one, the +/// retransmitted copy of the flagged PCR or a PCR whose flag came in the packet before, waits out +/// the spacing after it as its partner, which counted the jump there, does. +#[tokio::test(start_paused = true)] +async fn a_joiner_on_a_jump_it_cannot_see_waits_out_the_spacing() { + const LINE: u64 = 1_000 * MS; + for retransmitted in [true, false] { + let mut mux = Mux::default(); + program(&mut mux, 0x100); + let prefix = mux.out.len(); + // The placed epoch puts these PCRs one wrap on, so `offset` lands every 25th on a line. + let offset = (LINE - WRAP % LINE) % LINE; + paced(&mut mux, 0x100, offset, 100, 0, 4); + let jump = offset + 11 * LINE; + if retransmitted { + mux.packet(0x100, Flags::pcr(jump).discontinuity()); + mux.out.extend_from_within(mux.out.len() - 188..); + } else { + mux.packet(0x100, Flags::default().discontinuity()); + mux.packet(0x100, Flags::pcr(jump)); + } + mux.fill(0x101, 4); + paced(&mut mux, 0x100, jump + 40 * MS, 250, 0, 4); + let data = mux.bytes(); + let split = pcr_at(&data, jump) + if retransmitted { 188 } else { 0 }; + import_twice_placed_from(&data, prefix, split).await; + } +} + +/// The grid lies on the timeline the epoch places, so it runs on regularly across a PCR wrap that +/// would shift a grid of raw ticks by about 0.72 s, and an importer joining after the wrap cuts on +/// it too. +#[tokio::test(start_paused = true)] +async fn the_grid_runs_on_across_a_wrap() { + const WRAP: u64 = (1 << 33) * 300; + let mut mux = Mux::default(); + program(&mut mux, 0x100); + let prefix = mux.out.len(); + let start = WRAP - 3_000 * MS; + for i in 0..300 { + mux.packet(0x100, Flags::pcr((start + i * 40 * MS) % WRAP)); + mux.fill(0x101, 4); + } + let data = mux.bytes(); + let (first, _) = import_twice_placed(&data, prefix, 2_000 * MS).await; + let groups = first.groups(); + assert!(groups.len() >= 10, "{} groups", groups.len()); + for group in &groups { + let pcr = header(&group[0].payload).pcr.expect("a cut on the grid falls on a PCR"); + let unwrapped = if pcr >= start { pcr } else { pcr + WRAP }; + let past = unwrapped % (1_000 * MS); + assert!(past < 40 * MS, "a cut {} ms past the line", past / MS); + } +} + /// With every PCR in the era nearest the clock and nothing clamping it, a flagged jump forward by /// less than the object a random access point started already reached would start its group /// before that one: the import ends rather than shift the timeline off the source's clock.