diff --git a/doc/bin/cli.md b/doc/bin/cli.md index dcde2cd12f4..9e5a5de42d2 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -231,8 +231,10 @@ to `--delay` ahead of its decode time and no further than its decoder buffer holds. That send-ahead comes out of the delay, so the output trails a TS source by `--delay`. A frame that cannot arrive in time at the rate fails the export, as does video whose HRD declares a longer buffer delay than `--delay`; the -error names the `--delay` it needs. Skipped media within the same broadcast -flags the next PCR as discontinuous without changing the PMT version. Raw +error names the `--delay` it needs. A clock jump flags its first PCR without +changing the PMT version. Padding keeps PCR values on a continuous grid; after +an outage exhausts the buffered media, overdue padding can arrive in a catch-up +burst when media resumes. Raw `--passthrough` keeps surviving packets unchanged, including across loss. ## Play diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 0ac762f0bc8..b53913e6349 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -102,6 +102,9 @@ test, which failed two merge-queue batches and two loaded local runs. Added 2026-11-05 (maintainer, `/quest-audit`): Interop's P2P data-channel timeout, one child. +Added 2026-11-06: `js/net`'s lite-06 datagram end/close case, from `Check` on +[moq-dev/moq#7981](https://github.com/moq-dev/moq/pull/7981). + Added 2026-11-05 (maintainer, `/quest-complete` follow-ups): moq-net's pool-debt flight test, the Python audio playout-delay test, and the io_uring tests' shared locked-memory budget, one child each. The pool-debt test diff --git a/quest/m0/test-flakes-2/datagram-end-close.md b/quest/m0/test-flakes-2/datagram-end-close.md index 39259d35445..80b19d87b65 100644 --- a/quest/m0/test-flakes-2/datagram-end-close.md +++ b/quest/m0/test-flakes-2/datagram-end-close.md @@ -44,4 +44,32 @@ Reproduce first: temporarily remove the case's `test.skip`, then run the whole file beside CPU loops, as above. Keep the failing output before making other changes to the test or the end path. Restore the skip until the fix. +### Additional lite-06 investigation + +Planned 2026-11-06 under the maintainer's rule that a known flaky test is +quarantined immediately and its investigation planned. `Check` failed on this +case in +[run 42200239263](https://github.com/moq-dev/moq/actions/runs/42200239263/job/127833434287) +on the unrelated [moq-dev/moq#7981](https://github.com/moq-dev/moq/pull/7981), +where it was the only failing case in `js/net`. The unchanged module reproduced +it once in eight runs: `cd js/net && bun test src/lite/datagram-tracks.test.ts`. + +The case spins 1000 microtasks after `datagrams.finish()`, then aborts the +client, expecting the reader's `recv()` to return `undefined` because the end +already arrived. On failure the reader surfaces the session close instead +(`Session: remote error: 0`). The spin assumes the mock delivers the end in +microtasks, but the subscriber's response loop yields through `@moq/clock`'s +`yieldNow` (a host task) once its budget's 4 ms slice expires, which a loaded +runner can reach before the end is read. Whether that is only the fixture's +wait or a real gap between receiving the end and delivering it is not +established: diagnose before changing the wait or production code, and do not +keep the case passing by loosening the assertion. Base subsequently quarantined +all three protocol versions, as described above. + +Decided 2026-11-06 (maintainer, /quest-unblock): investigate the additional +[CodeRabbit finding](https://github.com/moq-dev/moq/pull/7981#discussion_r4491539836) +here: an accepted lite-06 `SUBSCRIBE_END` followed by duplicate SETUP can be +overridden by session failure before subscription finalization. Reproduce this +ordering deterministically and preserve the accepted end during shutdown. + Public API: none. Wire: none. diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index 49d55df2f2d..6cf2835ee20 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -1,4 +1,4 @@ -# TS fidelity and recovery +# [M] TS fidelity and recovery ## Goal @@ -117,25 +117,13 @@ Decided 2026-11-04 (maintainer, design review of ends, and the initial reservation releases at the first PTS. This is independent of the handoff. -Decided 2026-11-05 (maintainer, /quest-complete, for -[moq-dev/moq#7785](https://github.com/moq-dev/moq/pull/7785)): ✅ cross-track -loss coalescing keeps only each track's last loss. A slower track can flag -outage A again after a faster track resumed A and then B (three flags for -two outages); state stays bounded by the track count, and the extra flag -lands where loss happened. Not chosen: keeping every loss until all tracks -pass it (unbounded while a track stalls); collapsing losses older than the -release point minus the delay (more code, and assumes muxing in release -order). Reversed the same day: the planned -[PCR jump flag](/quest/m1/ts-reliability/pcr-jump-flag.md) removes the -per-track loss tracking. - PMT fragmentation (#5220), initial mux-rate measurement (#5287), and decoder-time admission (#5306) have landed, as have PSI runs (#5606) and catalog linger (#5305). ## Required -- [The TS export flags a PCR only where its clock jumps](/quest/m1/ts-reliability/pcr-jump-flag.md) - a padded skip goes out unflagged; a schedule that skips slots flags the jump +- [PCR jump validation](/quest/m1/ts-reliability/pcr-jump-flag.md) - validate the remaining TS recovery interop coverage ## Related diff --git a/quest/m1/ts-reliability/pcr-jump-flag.md b/quest/m1/ts-reliability/pcr-jump-flag.md index 199627d42f5..382181145c8 100644 --- a/quest/m1/ts-reliability/pcr-jump-flag.md +++ b/quest/m1/ts-reliability/pcr-jump-flag.md @@ -1,44 +1,13 @@ -# [S] The TS export flags a PCR only where its clock jumps +# [S] Validate PCR jumps across recovery ## Goal -`moq export ts` sets `discontinuity_indicator` on a PCR exactly when the -time base it asserts breaks: a declared switch, as today, or a schedule that -skips empty slots so the PCR jumps past the bytes between. A delivery skip the -schedule pads on a continuing clock goes out unflagged, a gap in the programme -and not a new time base. +Validate actual-PCR-jump flags in the remaining TS recovery interop coverage. ## Plan -Reported in [moq-dev/moq#7965](https://github.com/moq-dev/moq/issues/7965), -a correction to [moq-dev/moq#7741](https://github.com/moq-dev/moq/issues/7741): -in a live relay-kill run the export padded a 7 s outage with nulls on an -unbroken 25 ms PCR grid, and every PID's DTS kept its offset to the PCR. So -[moq-dev/moq#7785](https://github.com/moq-dev/moq/pull/7785), which flags -every delivery skip, declared a new time base equal to the old one there. -Verified against `main`: the claim holds for holes the schedule pads, but -`Schedule::index` (`rs/moq-mux/src/container/ts/schedule.rs`) skips the -empty slots of a gap longer than `BACKFILL` (1 s), and that PCR does jump. -`a_skipped_group_flags_the_clock` runs 6 s holes through that path. +Decided 2026-11-06 (maintainer, /quest-unblock): ✅ Continue the actual-PCR-jump approach. Not chosen: rethink; drop the PR. -Decided 2026-11-05 (maintainer, /quest-import of #7965): +Decided 2026-11-06 (maintainer, /quest-unblock): ✅ Reproduce and fix the Rust `with_delay` schedule replacement in this PR. Hand the unrelated JS decoded-end/session-failure finding to [datagram-end-close](/quest/m0/test-flakes-2/datagram-end-close.md). Not chosen: fixing both findings here. -- ✅ Flag real jumps only: derive the flag from the schedule, set on the first - PCR after it skips slots, whatever caused the gap, so a source stall with - no delivery skip is flagged too. Delete #7785's per-track loss tracking - (`Track::{gap, last_timestamp, lost}`, `coalesce`, `Queued::gap`, - `PesUnit::skipped`, `Schedule::{gap, skip}`), and the 2026-11-05 - coalescing decision in the epic README goes with it. Not chosen: keeping - the flag as a skip marker and rewording its docs (it misstates the time - base); reverting #7785 (a hole past the backfill would again carry an - unflagged PCR jump); won't fix. -- ✅ m1, in the TS reliability epic. - -Rework the tests to match: holes the schedule pads carry no flag and keep the -PCR's repetition health clean; holes past the backfill carry one flag each on -the first PCR after the jump. Fix `doc/bin/cli.md` and `test/ts/README.md`, -which describe the skip flag. - -## Closes - -- [#7965](https://github.com/moq-dev/moq/issues/7965) - `export ts`: the skip #7741 reported keeps its clock +Decided 2026-11-06 (maintainer, /quest-unblock): ✅ Construction-only delay supersedes runtime updates. TS Export and Release take delay when constructed, so retained state never changes its delay basis. Not chosen: full runtime retiming; landing without the fix. diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index 2f829c44f84..e80c6fd704b 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -1323,9 +1323,10 @@ pub(crate) mod tests { async fn open_export( source: moq_mux::Source, format: CatalogFormat, + delay: Duration, ) -> moq_mux::Result>> { let catalog = source.catalog::(format).await?; - Ok(Export::new(source, catalog)) + Ok(Export::new(source, catalog, delay)) } /// Drain an exporter, concatenating every frame's payload until output stops. The @@ -1431,10 +1432,9 @@ pub(crate) mod tests { // `catalog`, the producers, and `import` stay alive: the exporter subscribes to // the retained tracks. drain( - open_export(resolved(&origin, "cli").await, CatalogFormat::Hang) + open_export(resolved(&origin, "cli").await, CatalogFormat::Hang, RECORDING_MAX_AGE) .await - .unwrap() - .with_delay(RECORDING_MAX_AGE), + .unwrap(), ) .await } @@ -1500,10 +1500,9 @@ pub(crate) mod tests { // Subscribe side: the same catalog stream `run_ts` selects, re-emitting the // ancillary streams verbatim. let output = drain( - open_export(resolved(&origin, "cli").await, format) + open_export(resolved(&origin, "cli").await, format, RECORDING_MAX_AGE) .await - .unwrap() - .with_delay(RECORDING_MAX_AGE), + .unwrap(), ) .await; @@ -2380,10 +2379,13 @@ pub(crate) mod tests { instance.restart(&republish, restart, &mut log).unwrap(); } if follower.is_none() && instance.pending.is_none() { - let export = open_export(resolved(&origin, "demo").await, CatalogFormat::Hang) - .await - .unwrap() - .with_delay(Duration::from_secs(1)); + let export = open_export( + resolved(&origin, "demo").await, + CatalogFormat::Hang, + Duration::from_secs(1), + ) + .await + .unwrap(); follower = Some( ts::Follower::new(export) .unwrap() diff --git a/rs/moq-cli/src/subscribe.rs b/rs/moq-cli/src/subscribe.rs index e58c102ad74..ef387b7cd75 100644 --- a/rs/moq-cli/src/subscribe.rs +++ b/rs/moq-cli/src/subscribe.rs @@ -485,7 +485,7 @@ impl Subscribe { // selects the `mpegts` catalog extension so undecoded elementary streams // (SCTE-35, teletext, DVB AC-3, ...) are re-emitted verbatim on their PIDs. let catalog = self.source.catalog::(self.catalog).await?; - let mut ts = moq_mux::container::ts::Export::new(self.source, catalog).with_delay(self.args.max_delay); + let mut ts = moq_mux::container::ts::Export::new(self.source, catalog, self.args.max_delay); if let Some(mux_rate) = self.args.mux_rate { ts = ts.with_mux_rate(mux_rate); } @@ -550,9 +550,7 @@ impl Subscribe { /// time on the source's PCR comes round. async fn run_ts_passthrough(self) -> anyhow::Result<()> { let mut stdout = tokio::io::stdout(); - let mut release = moq_mux::container::ts::Release::new(self.source, self.catalog) - .await? - .with_delay(self.args.max_delay); + let mut release = moq_mux::container::ts::Release::new(self.source, self.catalog, self.args.max_delay).await?; let report = |stats: &moq_mux::container::ts::stats::Release| { if let Some(sections) = &self.sections { sections.set(|report| report.release = Some(stats.clone())); diff --git a/rs/moq-mux/benches/export.rs b/rs/moq-mux/benches/export.rs index 3efe19e8fdf..7e62a0c48a6 100644 --- a/rs/moq-mux/benches/export.rs +++ b/rs/moq-mux/benches/export.rs @@ -101,7 +101,11 @@ fn open(format: &str, source: moq_mux::Source, stream: moq_mux::catalog::Consume "flv" => Export::Flv(Box::new( moq_mux::container::flv::Export::new(source, stream).with_multitrack(true), )), - "ts" => Export::Ts(Box::new(moq_mux::container::ts::Export::new(source, stream))), + "ts" => Export::Ts(Box::new(moq_mux::container::ts::Export::new( + source, + stream, + std::time::Duration::ZERO, + ))), _ => Export::Mkv(Box::new( moq_mux::container::mkv::Export::new(source, stream).with_fragment_duration(Duration::ZERO), )), @@ -330,7 +334,8 @@ fn video_reorder(c: &mut Criterion) { |(source, stream, fixture)| { let refused = runtime.block_on(async { if format == "ts" { - let mut opened = moq_mux::container::ts::Export::new(source, stream); + let mut opened = + moq_mux::container::ts::Export::new(source, stream, std::time::Duration::ZERO); let mut chunks = 0; while let Some(frame) = opened.next().await.unwrap() { std::hint::black_box(frame); diff --git a/rs/moq-mux/benches/ts_schedule.rs b/rs/moq-mux/benches/ts_schedule.rs index c0115bc8dbe..28105772372 100644 --- a/rs/moq-mux/benches/ts_schedule.rs +++ b/rs/moq-mux/benches/ts_schedule.rs @@ -70,9 +70,6 @@ fn admission_case(c: &mut Criterion, skipped: bool) { for frame in 0..frames { for pid in 1..=pids { let gap = if skipped && frame >= frames / 2 { 4_000 } else { 0 }; - if skipped && frame == frames / 2 { - schedule.skip(); - } let at = (999 + gap + frame * 40) * 1_000_000; schedule .push_frames( diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index 67065dab5be..34449e7dde8 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -12,7 +12,7 @@ //! the instant it asserts. Video is carried as Annex-B, audio as ADTS AAC. //! //! Every frame is muxed a fixed delay after its release time, its decode time less its -//! send-ahead ([`Export::with_delay`]), so tracks interleave in one order whatever the +//! send-ahead ([`Export::new`]), so tracks interleave in one order whatever the //! arrival skew between them, and the output keeps the source's pace. //! //! Video flows through `ExportSource`, which normalizes every H.264/H.265 @@ -240,7 +240,8 @@ where /// What a queued output frame tells [`Export::stats`], applied only once the frame is /// returned: a rewind drops the queue unwritten. struct Tally { - /// The frame's PCR in 27 MHz ticks, and whether it flags a new time base. + /// The frame's PCR in 27 MHz ticks, and whether a declared marker, rewind, or program + /// switch started a new time base. pcr: (u64, bool), /// The PID of each access unit whose packets begin in the frame. units: Vec, @@ -272,8 +273,6 @@ struct Pending { /// A frame waiting in the jitter buffer, with what it needs from the moment it was read. struct Queued { frame: Frame, - /// Delivery skips before this frame, excluding declared timeline boundaries. - gap: u64, /// When the frame decodes: its DTS, else its PTS. decode: Timestamp, /// Where the jitter buffer orders it ([`Track::release_key`]). @@ -319,14 +318,6 @@ struct Track { /// The marker, skip and cut counts of the sources the track read before this one, so the /// counts keep rising across a resubscription. base: (u64, u64, u64), - /// Delivery skips before the last frame muxed, so read-ahead cannot flag earlier output. - gap: u64, - /// Furthest presentation time muxed, so overlapping track losses share one break. - last_timestamp: Option, - /// The last delivery loss the track signalled or joined ([`coalesce`]). Only the last, so - /// state stays bounded by the track count: a slower track can flag an outage again after a - /// faster one moved past it, an extra flag where loss did happen. - lost: Option, /// How many unmarked steps back the track's timeline took ([`Self::steps_back`]), each /// counted as a marker of its own. steps: u64, @@ -409,9 +400,6 @@ impl Track { latest: kio::clock::Instant::ZERO, restart: 0, base: (0, 0, 0), - gap: 0, - last_timestamp: None, - lost: None, steps: 0, high: None, finished: false, @@ -853,7 +841,6 @@ impl Track { let Pending { frame, restart, - skip, arrived, read, .. @@ -902,7 +889,6 @@ impl Track { bytes: frame.payload.len() + self.key.name.len(), item: Queued { frame, - gap: skip.saturating_sub(restart), decode, key, dts, @@ -1239,8 +1225,6 @@ struct PesUnit { pid: u16, is_video: bool, keyframe: bool, - /// Delivery loss precedes this frame, so its scheduled PCR flags the gap. - skipped: bool, timestamp: Timestamp, /// Authored decode timestamp for a reordered (B-frame) video frame, in continuous /// (unwrapped) 90 kHz ticks (wrapped to the wire field in `write_pes`). `Some` only when @@ -1566,7 +1550,7 @@ impl Program where S::Ext: catalog::Catalog, { - fn new(mut source: crate::Source, catalog: S) -> Self { + fn new(mut source: crate::Source, catalog: S, delay: Duration) -> Self { source.isolate(); let instance = source.broadcast().info().instance.clone(); Self { @@ -1587,7 +1571,7 @@ where tables: Vec::new(), psi_due: false, revised: false, - delay: Duration::ZERO, + delay, generation: 0, tracks: HashMap::new(), counters: FxHashMap::default(), @@ -1601,7 +1585,7 @@ where epoch: 0, emitted_epoch: 0, pcr_discontinuity: false, - schedule: Schedule::new(Duration::ZERO), + schedule: Schedule::new(delay), slot_timer: None, replay: false, queue: VecDeque::new(), @@ -1623,10 +1607,42 @@ where /// The caller chooses the catalog format and rendition selection on any [`Stream`]. /// The stream remains owned until the export is dropped or follows another broadcast; /// concrete catalog consumers release their subscription and decoder when they end. - pub fn new(source: crate::Source, catalog: S) -> Self { + /// + /// Mux each frame `delay` after its release time, like an SRT receiver's TSBPD: its + /// decode time less its send-ahead. + /// + /// The send-ahead comes out of the same budget. A frame may go out up to the delay + /// ahead of its decode time, as far as its PID's decoder buffer holds and as early as the + /// receiver's buffers admit, so a heavy passage rides the slots before it. Its release + /// time is that far before its decode time, so it is muxed just as the first slot it may + /// ride is laid, and each slot goes out the delay after its own time on the source's + /// clock rather than a delay later still. + /// + /// The clock is acquired before anything goes out: frames are held until every expected + /// track has delivered (two delays at most) and a track starts a new group, or for a delay + /// at most once all have. The track sent latest against its release time sets the clock, + /// so a joiner runs at the delay from its first output and every frame has the whole delay + /// to arrive in. A source that sends its video its send-ahead + /// early, as a TS mux does, is trailed by the delay; one that sends every track at its + /// decode time, by the delay and the video's send-ahead. The output then keeps the + /// source's pace and muxes every track in `(release time, PID)` order whatever the + /// arrival skew between them. A frame that arrives after its deadline is dropped, and a + /// video track that dropped one resumes at its next keyframe. A source whose clock runs + /// further off ours than 13818-1 lets the PCR follow fails the export once it has used half + /// the delay, where a brief network stall only drops what it made late. A stalled group is + /// skipped once it falls half the delay behind the newest content (see + /// [`Consumer`](crate::container::Consumer)), so the group after it still arrives with + /// the other half to spare. + /// + /// With a multiplex rate, a frame that cannot arrive by its decode time fails the export, + /// as does video whose HRD declares a buffer delay past the delay. + /// + /// A zero `delay` holds nothing and sends each frame by its decode time. + pub fn new(source: crate::Source, catalog: S, delay: Duration) -> Self { + let program = Program::new(source, catalog, delay); Self { - program: Program::new(source, catalog), - jitter: jitter::Buffer::new(Duration::ZERO), + program, + jitter: jitter::Buffer::new(delay), held: None, pending: None, next_program: 1, @@ -1662,52 +1678,11 @@ where self } - /// Mux each frame this long after its release time, like an SRT receiver's TSBPD: its - /// decode time less its send-ahead. - /// - /// The send-ahead comes out of the same budget. A frame may go out up to the delay - /// ahead of its decode time, as far as its PID's decoder buffer holds and as early as the - /// receiver's buffers admit, so a heavy passage rides the slots before it. Its release - /// time is that far before its decode time, so it is muxed just as the first slot it may - /// ride is laid, and each slot goes out the delay after its own time on the source's - /// clock rather than a delay later still. - /// - /// The clock is acquired before anything goes out: frames are held until every expected - /// track has delivered (two delays at most) and a track starts a new group, or for a delay - /// at most once all have. The track sent latest against its release time sets the clock, - /// so a joiner runs at the delay from its first output and every frame has the whole delay - /// to arrive in. A source that sends its video its send-ahead - /// early, as a TS mux does, is trailed by the delay; one that sends every track at its - /// decode time, by the delay and the video's send-ahead. The output then keeps the - /// source's pace and muxes every track in `(release time, PID)` order whatever the - /// arrival skew between them. A frame that arrives after its deadline is dropped, and a - /// video track that dropped one resumes at its next keyframe. A source whose clock runs - /// further off ours than 13818-1 lets the PCR follow fails the export once it has used half - /// the delay, where a brief network stall only drops what it made late. A stalled group is - /// skipped once it falls half the delay behind the newest content (see - /// [`Consumer`](crate::container::Consumer)), so the group after it still arrives with - /// the other half to spare. - /// - /// With a multiplex rate, a frame that cannot arrive by its decode time fails the export, - /// as does video whose HRD declares a buffer delay past the delay. Defaults to - /// [`Duration::ZERO`], which holds nothing and sends each frame by its decode time. - pub fn with_delay(mut self, delay: Duration) -> Self { - self.program.delay = delay; - self.jitter = jitter::Buffer::new(delay); - #[cfg(test)] - if self.program.replay { - self.jitter = jitter::Buffer::new(delay).replay(); - } - self.program.schedule = Schedule::new(delay); - self.program.schedule.set_rate(self.program.mux_rate); - self - } - /// Get the next muxed frame. /// /// Each [`Frame`] carries one slice of the PCR grid in `payload`: the clock /// packets that slice opens with, followed by the muxed bytes belonging to it. - /// With a [delay](Self::with_delay), each slice is handed over at its time on the + /// With a [delay](Self::new), each slice is handed over at its time on the /// jitter buffer's clock, which follows the source's, so a sink writes it as it /// comes. It is stamped with the media time that slice starts at, for a transport /// that stamps its own delivery, and `keyframe` marks the slice a video keyframe @@ -2923,13 +2898,11 @@ where } fn replacement(&mut self, source: crate::Source, catalog: S) -> Program { - let mut next = Program::new(source, catalog); + let mut next = Program::new(source, catalog, self.program.delay); next.id = self.next_program; self.next_program += 1; - next.delay = self.program.delay; next.replay = self.program.replay; next.mux_rate_override = self.program.mux_rate_override; - next.schedule = Schedule::new(next.delay); next } @@ -3029,8 +3002,6 @@ where psi.led = psi.out.as_ref().map(|laid| laid.pmt.clone()); } for track in self.program.tracks.values_mut() { - track.last_timestamp = None; - track.lost = None; track.pack = None; track.parked.clear(); } @@ -3509,27 +3480,12 @@ where fn mux(&mut self, name: &str, queued: Queued) -> anyhow::Result<()> { let Queued { frame, - gap, decode, dts, description, pes, .. } = queued; - let track = self.program.tracks.get_mut(name).context("missing track")?; - let skipped = gap > track.gap; - track.gap = gap; - // The track's missing interval runs from its furthest media muxed, or from the last - // reset when it has muxed none since, a start every such track shares (`None`). - let start = track.last_timestamp; - let end = frame.timestamp; - track.last_timestamp = Some(start.map_or(end, |start| start.max(end))); - let skipped = skipped && { - let others = self.program.tracks.values().filter_map(|track| track.lost); - let (lost, separate) = coalesce((start, end), others); - self.program.tracks.get_mut(name).context("missing track")?.lost = Some(lost); - separate - }; let track = self.program.tracks.get(name).context("missing track")?; let kind = &track.kind; let is_video = matches!(kind, Kind::Video(_)); @@ -3592,7 +3548,7 @@ where // The tables lead the unit that carries this frame, behind whatever went before it. if kind.placed() { self.program.psi_due |= psi; - return self.place(name, &frame, out, skipped); + return self.place(name, &frame, out); } match es_payload { // Section-framed verbatim (SCTE-35, ...) rides in private sections, not PES; @@ -3606,7 +3562,6 @@ where pid, is_video, keyframe, - skipped, timestamp: frame.timestamp, dts, stream_id: stream_id(&kind), @@ -3621,7 +3576,7 @@ where self.write_pes(&mut out, &unit, &es_payload)?; } } - self.push_unit(name, pid, decode, out, (keyframe, skipped), Vec::new()) + self.push_unit(name, pid, decode, out, keyframe, Vec::new()) } /// Queue a whole PES or section unchanged on `name`'s PID behind `tables`, its @@ -3632,7 +3587,7 @@ where /// unit that begins in it, so each one after counts on from there and leaves at its own time. /// The bytes ahead of the first end a frame an earlier PES began; they leave with it, /// which only overstates what the buffer holds. A PES without a PTS holds nothing there. - fn place(&mut self, name: &str, frame: &Frame, tables: Vec, skipped: bool) -> anyhow::Result<()> { + fn place(&mut self, name: &str, frame: &Frame, tables: Vec) -> anyhow::Result<()> { let track = self.program.tracks.get(name).context("missing track")?; let pes = match track.kind { Kind::PesPacket { stream_type } => Some( @@ -3672,9 +3627,6 @@ where self.program.schedule.set_buffer(pid, buffer); } self.lead_tables()?; - if skipped { - self.program.schedule.skip(); - } self.program.schedule.push_placed( pid, (place, decode.unwrap_or(place)), @@ -3712,8 +3664,7 @@ where let (size, fit) = track.fit(payload.len()); let place = place(unit.timestamp, samples, rate); // A PTS is a whole 90 kHz tick, so the frames run on within one of where they end. - let joins = !unit.skipped - && start.is_some() + let joins = start.is_some() && track.pack.as_ref().is_some_and(|pack| { pack.start == start && pack.rate == rate @@ -3760,7 +3711,7 @@ where } = pack; let mut out = Vec::new(); self.write_pes(&mut out, &unit, &payload)?; - self.push_unit(name, unit.pid, decode, out, (unit.keyframe, unit.skipped), frames) + self.push_unit(name, unit.pid, decode, out, unit.keyframe, frames) } /// Send on, in decode order, the packed frames of each track for which `done` holds, given @@ -3792,7 +3743,7 @@ where pid: u16, decode: Timestamp, out: Vec, - (keyframe, skipped): (bool, bool), + keyframe: bool, frames: Vec<(u128, usize)>, ) -> anyhow::Result<()> { let mut tables = std::mem::take(&mut self.program.tables); @@ -3822,9 +3773,6 @@ where } let by = decode.as_nanos().saturating_sub(track.drain().as_nanos()); self.lead_tables()?; - if skipped { - self.program.schedule.skip(); - } self.program.schedule.push_frames( pid, super::schedule::Timing { @@ -3959,8 +3907,8 @@ where let mut payload = slot.layout(&clock, pmt_pid, &NULL_PACKET); // The slot opens with its PCR, in the clock packet or the clock PID's packet that // carries it, whose adaptation field flags the break. - let discontinuity = std::mem::take(&mut self.program.pcr_discontinuity) || slot.discontinuity; - if discontinuity { + let discontinuity = std::mem::take(&mut self.program.pcr_discontinuity); + if discontinuity || slot.discontinuity { payload[5] |= 0x80; } self.number(&mut payload); @@ -3980,6 +3928,7 @@ where spent = breaks; } let tables = (0..slot.tables).filter_map(|_| psi.sending.pop_front()).last(); + // Skipped slots retain the source timeline, so silence statistics include the jump. let tally = Tally { pcr: (slot.pcr, discontinuity), units: slot.units, @@ -4331,30 +4280,6 @@ fn dolby_removals(codec: Dolby, payload: &[u8], decode: u128) -> Vec<(u128, usiz frames } -/// A delivery loss's presentation interval: from the furthest media its track muxed before, -/// `None` when it had muxed none since the last reset, to the frame it resumed at. -type Lost = (Option, Timestamp); - -/// Merge `lost` with every interval in `others` it overlaps, and whether it overlapped none, -/// so it needs a PCR flag of its own. -/// -/// Tracks resume the same outage in different release slots, and in any order. One flag covers -/// their overlapping intervals, including a later read on another track that extends one -/// already signalled. Presentation time shares the content's timeline across their different -/// decode reserves, and every track that muxed nothing since the last reset starts its loss -/// there. -fn coalesce((start, end): Lost, others: impl IntoIterator) -> (Lost, bool) { - let shared: Vec = others - .into_iter() - .filter(|&(first, last)| start.is_none_or(|start| start < last) && first.is_none_or(|first| first < end)) - .collect(); - // `None` sorts first, so a merge keeps an unbounded start. - let lost = shared.iter().fold((start, end), |(start, end), &(first, last)| { - (start.min(first), end.max(last)) - }); - (lost, shared.is_empty()) -} - /// Whether `timestamp` has crossed into a later repetition slot than `last`. /// /// Slots are absolute on the media timeline (`floor(timestamp / interval)`) rather than @@ -5110,29 +5035,6 @@ mod tests { due, is_complete_section, si_due, slot, unwrap_pts, verbatim_buffer, }; - /// Losses resumed out of presentation order flag each outage once, however they interleave. - #[test] - fn coalesce_flags_each_outage_once() { - let ms = |ms: u64| Timestamp::from_millis(ms).unwrap(); - let a = (Some(ms(10_200)), ms(10_400)); - let b = (Some(ms(9_800)), ms(10_000)); - assert_eq!(super::coalesce(a, []), (a, true)); - assert_eq!(super::coalesce(b, [a]), (b, true), "a disjoint earlier outage"); - assert_eq!(super::coalesce(a, [a, b]), (a, false), "the first outage again"); - let wider = (Some(ms(9_900)), ms(10_300)); - assert_eq!( - super::coalesce(wider, [a, b]), - ((Some(ms(9_800)), ms(10_400)), false), - "spanning both" - ); - let reset = (None, ms(24_280)); - assert_eq!( - super::coalesce((None, ms(24_216)), [reset]), - ((None, ms(24_280)), false), - "both since a reset" - ); - } - /// The tables as they stand take the version after the last laid, whatever was set on the /// schedule meanwhile, and the last laid's own when they come back to it. The PAT keeps its /// own version until its network entry or PMT PID changes. diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 1be3d98e020..b6a47a44adf 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -23,18 +23,22 @@ use crate::container::{Frame, Kind, Producer}; use moq_net::Timestamp; /// Open a media-only export on the default catalog stream. -pub(super) async fn open_export(source: crate::Source) -> crate::Result> { +pub(super) async fn open_export( + source: crate::Source, + delay: Duration, +) -> crate::Result> { let catalog = source.catalog::<()>(Default::default()).await?; - Ok(Export::new(source, catalog)) + Ok(Export::new(source, catalog, delay)) } /// Open an export with the MPEG-TS catalog extension and explicit encoding. pub(super) async fn open_ts( source: crate::Source, format: crate::catalog::CatalogFormat, + delay: Duration, ) -> crate::Result>> { let catalog = source.catalog::(format).await?; - Ok(Export::new(source, catalog)) + Ok(Export::new(source, catalog, delay)) } /// A millisecond track: the frames written here carry timestamps. @@ -98,7 +102,12 @@ const DRAIN: std::time::Duration = RECORDING_MAX_AGE .saturating_add(std::time::Duration::from_secs(1)); async fn drain(consumer: moq_net::broadcast::Consumer) -> BytesMut { - drain_with(open_export(crate::source::announced(&consumer).await).await.unwrap()).await + drain_with( + open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) + .await + .unwrap(), + ) + .await } /// `drain` for an exporter built with an explicit catalog extension. @@ -114,7 +123,7 @@ async fn drain_stats(exporter: Export) -> (BytesMu where S::Ext: tscat::Catalog, { - let mut exporter = exporter.with_delay(RECORDING_MAX_AGE).with_replay(); + let mut exporter = exporter.with_replay(); let mut out = BytesMut::new(); // `while let Ok` stops on the first timeout (`Pending`: no more output). while let Ok(res) = kio::clock::timeout(DRAIN, exporter.next()).await { @@ -190,7 +199,7 @@ async fn custom_selected_catalog_drives_export_and_follow() { .select(crate::select::Broadcast::default().audio(crate::select::Audio::default().name("keep"))); let source = crate::source::resolved(&origin, "live").await; let mut current = match export.take() { - None => Export::new(source, stream).with_delay(RECORDING_MAX_AGE).with_replay(), + None => Export::new(source, stream, RECORDING_MAX_AGE).with_replay(), Some(previous) => previous.follow(source.broadcast(), stream).unwrap(), }; let (frames, end) = drain_to_end(&mut current).await; @@ -242,7 +251,7 @@ async fn a_follower_keeps_msf_after_catalog_eof() { track.finish().unwrap(); let source = crate::source::resolved(&origin, "live").await; let stream = source.catalog::<()>(CatalogFormat::Msf).await.unwrap(); - let mut export = Export::new(source, stream); + let mut export = Export::new(source, stream, std::time::Duration::ZERO); assert!(export.next().await.unwrap().is_none()); assert_eq!(export.catalog_format(), CatalogFormat::Msf); // The routed subscription is released when the origin driver observes the local drop. @@ -634,7 +643,9 @@ async fn export_lead_audio() -> BytesMut { video.finish().unwrap(); audio.finish().unwrap(); - let exporter = open_export(crate::source::announced(&consumer).await).await.unwrap(); + let exporter = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) + .await + .unwrap(); // The producers stay alive through the drain so the retained tracks are readable. drain_with(exporter).await } @@ -1006,10 +1017,9 @@ async fn export_trims_leading_pictures_after_each_delivery_break() { group.finish().unwrap(); } track.finish().unwrap(); - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); let bytes: Vec = drain_frames(&mut export) .await @@ -1053,10 +1063,9 @@ async fn export_trims_leading_pictures_after_a_late_drop() { }), ); let mut video = Producer::new(track, HangContainer::Legacy(Kind::Video)); - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, Duration::from_millis(500)) .await - .unwrap() - .with_delay(Duration::from_millis(500)); + .unwrap(); let start = kio::clock::now(); let mut out = Vec::new(); // (arrival, PTS) in milliseconds, in decode order. The second group acquires the clock. @@ -1140,10 +1149,10 @@ async fn export_resumes_whole_mpeg2_video_at_a_sequence_header() { let mut export = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + Duration::from_millis(500), ) .await - .unwrap() - .with_delay(Duration::from_millis(500)); + .unwrap(); // (arrival, PTS) in milliseconds, and whether a sequence header opens a group. The 120 ms // picture stalls past its deadline; the two pictures after it still reference it. @@ -1315,10 +1324,9 @@ async fn export_declared(reorder: u8, groups: &[&[u64]]) -> (Export, ms: std::ops::Range| { @@ -2763,10 +2773,9 @@ async fn aac_program_config_follows_a_revision() { } }; let mut a = add("a.aac"); - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); write(&mut a, 0, 0..58); let mut frames = poll_frames(&mut export); @@ -2883,7 +2892,9 @@ async fn description_less_aac_refuses_raw_payloads() { .unwrap(); producer.finish().unwrap(); - let mut exporter = open_export(crate::source::announced(&consumer).await).await.unwrap(); + let mut exporter = open_export(crate::source::announced(&consumer).await, std::time::Duration::ZERO) + .await + .unwrap(); let err = loop { match kio::clock::timeout(DRAIN, exporter.next()).await { Ok(Ok(Some(_))) => continue, @@ -3188,6 +3199,7 @@ async fn scte35_fixtures_survive_roundtrip() { open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap(), @@ -3425,6 +3437,7 @@ async fn service_layer_survives_roundtrip() { open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap(), @@ -3536,6 +3549,7 @@ async fn a_reimport_keeps_the_first_si() { let mut export = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap(); @@ -3667,6 +3681,7 @@ async fn a_reimport_keeps_the_first_si_across_a_gap() { let mut export = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + std::time::Duration::ZERO, ) .await .unwrap() @@ -3808,6 +3823,7 @@ async fn si_pids_are_re_emitted_on_their_own_interval() { open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap(), @@ -3897,6 +3913,7 @@ async fn inline_si_form_is_re_emitted() { open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap(), @@ -3990,10 +4007,10 @@ async fn export_rejects_si_pid_on_the_pmt() { let mut exporter = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await - .unwrap() - .with_delay(RECORDING_MAX_AGE); + .unwrap(); let err = exporter .next() .await @@ -4043,10 +4060,10 @@ async fn export_rejects_si_pid_on_an_elementary_stream() { let mut exporter = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await - .unwrap() - .with_delay(RECORDING_MAX_AGE); + .unwrap(); let err = exporter .next() .await @@ -4062,10 +4079,10 @@ async fn export_of(consumer: &moq_net::broadcast::Consumer) -> Export) -> Re .unwrap(); producer.finish().unwrap(); - let mut exporter = open_export(crate::source::announced(&consumer).await) + let mut exporter = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); let mut out = BytesMut::new(); loop { @@ -5694,7 +5712,13 @@ async fn export_twice(with_video: bool, late_clock: bool) -> (Vec, Vec Export { - let mut export = open_export(self.source.clone()).await.unwrap().with_delay(delay); + let mut export = open_export(self.source.clone(), delay).await.unwrap(); assert!(drain_frames(&mut export).await.is_empty()); export } @@ -6363,7 +6387,7 @@ async fn audio_dropped_ahead_of_a_moved_start_is_not_started() { rig.audio_frame(); } } - let mut export = open_export(rig.source.clone()).await.unwrap().with_delay(delay); + let mut export = open_export(rig.source.clone(), delay).await.unwrap(); let mut out = Vec::new(); for tick in join..3 * GOP { rig.at(tick * VIDEO_US).await; @@ -6411,7 +6435,7 @@ async fn a_stale_start_group_holds_the_audio_behind_the_next_keyframe() { rig.audio_frame(); } } - let mut export = open_export(rig.source.clone()).await.unwrap().with_delay(delay); + let mut export = open_export(rig.source.clone(), delay).await.unwrap(); let mut out = Vec::new(); for tick in join..3 * GOP { rig.at(tick * VIDEO_US).await; @@ -6542,10 +6566,10 @@ async fn repointed_si_entry_resubscribes() { let mut exporter = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await - .unwrap() - .with_delay(RECORDING_MAX_AGE); + .unwrap(); let mut before = BytesMut::new(); write_key(&mut producer, 0); write_key(&mut producer, 1); @@ -6671,6 +6695,7 @@ async fn si_revision_after_final_media_frame_is_flushed() { let mut exporter = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + std::time::Duration::ZERO, ) .await .unwrap(); @@ -6769,6 +6794,7 @@ async fn si_cadence_rig((pid, table_id): (u16, u8), interval: Duration, info: mo let exporter = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + std::time::Duration::ZERO, ) .await .unwrap(); @@ -6984,15 +7010,22 @@ async fn queued_clock_revisions_follow_the_exporters_freshness_budgets() { .next() .unwrap() .reorder = Some(0); - rig.exporter = rig.exporter.with_delay(delay).with_replay(); + rig.exporter = open_ts( + crate::source::announced(&rig._broadcast.consume()).await, + crate::catalog::CatalogFormat::Hang, + delay, + ) + .await + .unwrap() + .with_replay(); rig.si_track.write_frame(Timestamp::ZERO, section(0)).unwrap(); let mut joining = open_ts( crate::source::announced(&rig._broadcast.consume()).await, crate::catalog::CatalogFormat::Hang, + delay, ) .await .unwrap() - .with_delay(delay) .with_replay(); let (mut initial, mut initial_b) = tokio::join!(rig.media_frames(0), drain_frames(&mut joining)); let (a, b) = tokio::join!(rig.media_frames(1_000), drain_frames(&mut joining)); @@ -7372,11 +7405,10 @@ async fn export_cbr_video() -> Vec { } video.finish().unwrap(); - let mut exporter = open_export(crate::source::announced(&consumer).await) + let mut exporter = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() .with_mux_rate(20_000_000) - .with_delay(RECORDING_MAX_AGE) .with_replay(); drain_frames(&mut exporter).await } @@ -7438,7 +7470,9 @@ async fn a_frame_on_a_slot_boundary_finishes_before_it_decodes() { } video.finish().unwrap(); - let mut exporter = open_export(crate::source::announced(&consumer).await).await.unwrap(); + let mut exporter = open_export(crate::source::announced(&consumer).await, std::time::Duration::ZERO) + .await + .unwrap(); let frames = drain_frames(&mut exporter).await; let ts: Vec = frames.iter().flat_map(|f| f.payload.iter().copied()).collect(); let packets: Vec<&[u8]> = ts.chunks(188).collect(); @@ -7558,10 +7592,9 @@ async fn slices_go_out_on_the_clock() { let consumer = broadcast.consume(); let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); let mut audio = aac_rendition(&mut broadcast, &mut catalog, "a.aac"); - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, Duration::from_millis(1_500)) .await - .unwrap() - .with_delay(Duration::from_millis(1_500)); + .unwrap(); let writer = tokio::spawn(async move { for second in 0..4 { for ms in (0..1_000).step_by(20) { @@ -7603,11 +7636,10 @@ async fn pcr_stays_among_the_bytes_across_reordered_tracks() { import.decode(&BytesMut::from(&data[..])).unwrap(); import.finish().unwrap(); - let mut exporter = open_export(crate::source::announced(&consumer).await) + let mut exporter = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() - .with_mux_rate(20_000_000) - .with_delay(RECORDING_MAX_AGE); + .with_mux_rate(20_000_000); let frames = drain_frames(&mut exporter).await; let pcrs = media_pcrs(&frames); assert_eq!(exporter.dropped(), 0, "expected the full feed"); @@ -7729,11 +7761,10 @@ async fn a_heavy_passage_fills_the_slots_before_it() { } producer.finish().unwrap(); - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, Duration::from_millis(500)) .await .unwrap() .with_mux_rate(2_000_000) - .with_delay(Duration::from_millis(500)) .with_replay(); let frames = drain_frames(&mut export).await; let ts: Vec = frames.iter().flat_map(|f| f.payload.iter().copied()).collect(); @@ -7975,7 +8006,7 @@ async fn export_pcr_is_its_byte_position_at_the_rate() { } producer.finish().unwrap(); - let export = open_export(crate::source::announced(&consumer).await) + let export = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() .with_mux_rate(1_000_000); @@ -8087,7 +8118,7 @@ impl Reordered { /// An exporter with its subscriptions resolved, so later polls need no runtime. async fn export(&self, max_age: Duration) -> Export { - let mut export = open_export(self.source.clone()).await.unwrap().with_delay(max_age); + let mut export = open_export(self.source.clone(), max_age).await.unwrap(); assert!(drain_frames(&mut export).await.is_empty()); export } @@ -8177,7 +8208,7 @@ async fn export_reordered( out_b.extend(poll_frames(b)); } if tick + 1 == JOIN { - b = Some(padded(open_export(rig.source.clone()).await.unwrap().with_delay(delay))); + b = Some(padded(open_export(rig.source.clone(), delay).await.unwrap())); } } rig.finish(); @@ -8457,7 +8488,7 @@ async fn jitter_published_after_the_tables_keeps_the_encoders_clock() { assert_decodes_after_the_clock(&runner); } -/// A delivery skip flags the next PCR without changing the program or its release clock. +/// Only skipped schedule slots flag the PCR, keeping the program and release clock. /// Short holes keep the grid continuous; long ones exceed the scheduler's backfill window. #[tokio::test(start_paused = true)] async fn a_skipped_group_flags_the_clock() { @@ -8485,10 +8516,9 @@ async fn a_skipped_group_flags_the_clock() { .insert("a.aac".to_string(), cfg); } - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, delay) .await - .unwrap() - .with_delay(delay); + .unwrap(); if let Some(rate) = rate { export = export.with_mux_rate(rate); } @@ -8518,12 +8548,16 @@ async fn a_skipped_group_flags_the_clock() { out.extend(drain_frames(&mut export).await); assert_eq!(export.discontinuity(), 0, "a skipped group is not a new program clock"); - assert_skip_discontinuity(&out, 2 * usize::from(missing > 0)); + assert_skip_discontinuity(&out, 2 * usize::from(missing == 60 && delay.is_zero())); let ts: Vec = out.iter().flat_map(|f| f.payload.iter().copied()).collect(); + for pair in out.windows(2) { + let jumped = pair[1].timestamp.as_micros() > pair[0].timestamp.as_micros() + 25_000; + assert_eq!(count_discontinuity(&pair[1..]), usize::from(jumped)); + } // Each flag lands right before the audio that resumes after its hole. let resumes: Vec = [20 + missing, 40 + 2 * missing] .into_iter() - .filter(|_| missing > 0) + .filter(|_| missing == 60 && delay.is_zero()) .map(|sequence| (60_000 + sequence * 100) * 90) .collect(); let mut reader = TsPacketReader::new(Cursor::new(&ts)); @@ -8549,9 +8583,13 @@ async fn a_skipped_group_flags_the_clock() { } } -/// A same-program skip changes only the next PCR's flag, including across a backfilled gap. +/// A same-program clock jump flags only its first PCR. fn assert_skip_discontinuity(frames: &[Frame], count: usize) { - assert_eq!(count_discontinuity(frames), count, "only one PCR per skip is flagged"); + assert_eq!( + count_discontinuity(frames), + count, + "only one PCR per clock jump is flagged" + ); assert!( pats(frames).iter().all(|&(version, _)| version == 0), "the PAT keeps its version" @@ -8571,20 +8609,20 @@ fn assert_skip_discontinuity(frames: &[Frame], count: usize) { } } let start = kio::clock::Instant::now(); - for (count, packet) in ts.as_chunks::<188>().0.iter().enumerate() { + let mut count = 0; + for packet in ts.as_chunks::<188>().0 { if packet[3] & 0x20 != 0 && packet[4] > 0 && packet[5] & 0x80 != 0 { - assert_ne!(packet[5] & 0x10, 0, "a skip flags only a PCR packet"); + assert_ne!(packet[5] & 0x10, 0, "a clock jump flags only a PCR packet"); } let continuation = continuity.packet(packet); - health.packet( - packet, - continuation, - super::health::At { - count: count as u64 + 1, - arrival: start, - }, - ); + // Feed the PCR samples as a uniform synthetic stream to grade their values. + // VBR packet density cannot supply a constant byte rate for health extrapolation. + if super::import::pcr(packet).is_some() { + count += 1; + health.packet(packet, continuation, super::health::At { count, arrival: start }); + } } + let mut stats = stats::Snapshot::default(); health.errors().report(&mut stats); assert_eq!(stats.pcr_discontinuity_indicator_error, 0, "no unflagged PCR jump"); @@ -8615,10 +8653,9 @@ async fn a_timeline_restarting_at_zero_fails_the_export() { .insert("a.aac".to_string(), cfg); } - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, Duration::from_millis(500)) .await - .unwrap() - .with_delay(Duration::from_millis(500)); + .unwrap(); let start = kio::clock::now(); // One-second groups of 20 ms frames for 10 s, then a group back at zero. The track refuses // a group whose first frame rewinds, so that group opens at the previous start. @@ -8677,8 +8714,10 @@ async fn a_late_join_starts_at_the_live_edge() { }) .unwrap(); if k == 275 { - let joined = open_export(crate::source::announced(&consumer).await).await.unwrap(); - export = Some(joined.with_delay(Duration::from_secs(2))); + let joined = open_export(crate::source::announced(&consumer).await, Duration::from_secs(2)) + .await + .unwrap(); + export = Some(joined); } if let Some(export) = export.as_mut() { out.extend(poll_frames(export)); @@ -8792,10 +8831,9 @@ async fn moved_export( use crate::container::test_util::{Moving, marked_frame}; let mut moving = Moving::new(); - let mut export = open_export(moving.source().await) + let mut export = open_export(moving.source().await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); let write = |moving: &mut Moving, i: u64, mark: u8| { moving.a.write(marked_frame(i, mark)).unwrap(); @@ -8832,7 +8870,7 @@ async fn moved_export( /// A rendition the catalog moves to another track, in its broadcast or a sibling one, goes on /// out of the PID the PMT announced, with that track's media and none of the one it left. The -/// continuity counters run on, and only the PCR flags the media skipped by the move. Its +/// continuity counters and the PCR clock run on through the move. Its /// parameter sets ride in-band, so the new track may carry another SPS. #[tokio::test(start_paused = true)] async fn a_moved_rendition_continues_on_its_pid() { @@ -8850,7 +8888,7 @@ async fn a_moved_rendition_continues_on_its_pid() { "the new track's parameter sets go out in-band" ); assert_eq!(continuity_errors(&frames), 0); - assert_skip_discontinuity(&frames, 1); + assert_skip_discontinuity(&frames, 0); assert!( pmts(&frames).iter().all(|(version, _)| *version == 0), "the PMT stays as announced" @@ -8872,10 +8910,9 @@ async fn a_rendition_moved_to_another_aac_configuration_fails() { .unwrap(); let mut next = Producer::new(track, HangContainer::Legacy(Kind::Data)); - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut old, ms); @@ -8915,10 +8952,9 @@ async fn a_track_leaving_the_catalog_is_read_to_its_end() { let mut kept = aac_rendition(&mut broadcast, &mut catalog, "a.aac"); let mut leaving = aac_rendition(&mut broadcast, &mut catalog, "b.aac"); - let mut export = open_export(crate::source::announced(&consumer).await) + let mut export = open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut kept, ms); @@ -9040,7 +9076,7 @@ fn unflagged(frames: &[Frame]) -> Vec { } /// The same instance back after every route went carries on in the same export, under the -/// program already announced: the PSI keeps its version, and only the PCR flags the gap. +/// program already announced: the PSI keeps its version and the PCR clock continues. /// Both ways a broadcast ends, a clean finish and a drop, continue alike. async fn same_instance_returns(finish: bool) { let origin = crate::source::produce_origin(); @@ -9049,10 +9085,9 @@ async fn same_instance_returns(finish: bool) { let (broadcast, mut catalog) = publish_live::<()>(&origin, route()); let mut track = aac_rendition(&mut broadcast.broadcast().clone(), &mut catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); let start = kio::clock::now(); for ms in (0..200).step_by(20) { @@ -9097,7 +9132,7 @@ async fn same_instance_returns(finish: bool) { end.unwrap(); assert_eq!(pes_count(&frames), 20, "every frame of both spans went out"); - assert_skip_discontinuity(&frames, 1); + assert_skip_discontinuity(&frames, 0); assert_eq!(export.discontinuity(), 0); assert!( pats(&frames).iter().all(|&(version, _)| version == 0), @@ -9134,11 +9169,7 @@ async fn a_return_keeps_catalog_and_group_history() { let clock = catalog.snapshot().media.clock; let source = crate::source::resolved(&origin, "live").await; let old = source.broadcast(); - let mut export = open_export(source) - .await - .unwrap() - .with_delay(RECORDING_MAX_AGE) - .with_replay(); + let mut export = open_export(source, RECORDING_MAX_AGE).await.unwrap().with_replay(); for (sequence, ms) in (0..200).step_by(20).enumerate() { write_aac(&mut track, ms); assert_eq!(groups.recv_group().await.unwrap().unwrap().sequence, sequence as u64); @@ -9182,7 +9213,7 @@ async fn a_return_keeps_catalog_and_group_history() { frames.extend(rest); end.expect("the publisher's final finish is clean"); assert_eq!(pes_count(&frames), 20, "each span goes out exactly once"); - assert_skip_discontinuity(&frames, 1); + assert_skip_discontinuity(&frames, 0); assert_eq!(export.discontinuity(), 0); assert!(pats(&frames).iter().all(|&(version, _)| version == 0)); assert!(pmts(&frames).iter().all(|(version, _)| *version == 0)); @@ -9198,10 +9229,9 @@ async fn a_switch_carries_through_an_instance_that_never_built_its_tables() { let (old, mut old_catalog) = publish_live::<()>(&origin, route()); let mut old_audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut old_audio, ms); @@ -9256,10 +9286,9 @@ async fn another_instance_is_a_full_program_switch() { let (old, mut old_catalog) = publish_live::<()>(&origin, route()); let mut old_audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut old_audio, ms); @@ -9372,10 +9401,10 @@ async fn a_switch_moving_the_pmt_advances_the_pat() { let mut export = open_ts( crate::source::resolved(&origin, "live").await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut old_track, ms); @@ -9416,10 +9445,9 @@ async fn export_stats_count_only_returned_output() { let (broadcast, mut catalog) = publish_live::<()>(&origin, route()); let mut track = aac_rendition(&mut broadcast.broadcast().clone(), &mut catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, RECORDING_MAX_AGE) .await - .unwrap() - .with_delay(RECORDING_MAX_AGE); + .unwrap(); for ms in (0..200).step_by(20) { write_aac(&mut track, ms); } @@ -9498,7 +9526,9 @@ async fn export_liveness(sample: u64, stop: u64) -> (stats::Export, stats::Expor aac_rendition(&mut broadcast.clone(), &mut catalog, "secondary.aac"), ]; - let mut export = open_export(crate::source::announced(&consumer).await).await.unwrap(); + let mut export = open_export(crate::source::announced(&consumer).await, std::time::Duration::ZERO) + .await + .unwrap(); assert!(export.stats().streams.is_empty(), "no row before the program tables"); let (mut sampled, mut frames) = (None, Vec::new()); let mut audio_index = 0; @@ -9603,7 +9633,7 @@ async fn export_stats_catch_a_track_that_stops() { /// A stopped track's silence counts the whole of a media gap on the surviving track, not just /// the one second of clock the exporter backfills: the output's PCR jumps across the rest -/// unflagged, and that jump is time the stopped track was silent. +/// with a discontinuity flag, and that jump is time the stopped track was silent. #[tokio::test(start_paused = true)] async fn export_stats_count_a_gap_longer_than_the_backfill() { let mut broadcast = moq_net::broadcast::Info::new().produce(); @@ -9612,7 +9642,9 @@ async fn export_stats_count_a_gap_longer_than_the_backfill() { let mut stopped = aac_rendition(&mut broadcast, &mut catalog, "a.aac"); let mut running = aac_rendition(&mut broadcast, &mut catalog, "b.aac"); - let mut export = open_export(crate::source::announced(&consumer).await).await.unwrap(); + let mut export = open_export(crate::source::announced(&consumer).await, std::time::Duration::ZERO) + .await + .unwrap(); for ms in (0..2_000).step_by(20) { write_aac(&mut stopped, ms); write_aac(&mut running, ms); @@ -9696,9 +9728,13 @@ async fn large_pmt(streams: u16, mux_rate: Option) -> usize { producers.push(producer); } catalog.modify().unwrap().ext.mpegts = config; - let mut exporter = open_ts(crate::source::announced(&broadcast.consume()).await, Default::default()) - .await - .unwrap(); + let mut exporter = open_ts( + crate::source::announced(&broadcast.consume()).await, + Default::default(), + RECORDING_MAX_AGE, + ) + .await + .unwrap(); if let Some(mux_rate) = mux_rate { exporter = exporter.with_mux_rate(mux_rate); } @@ -9780,7 +9816,7 @@ fn publish_bare( /// Follow the broadcast at `live` with a fresh export of it. async fn follower(origin: &moq_net::origin::Producer) -> super::Follower { let source = crate::source::resolved(origin, "live").await; - super::Follower::new(open_export(source).await.unwrap()).unwrap() + super::Follower::new(open_export(source, std::time::Duration::ZERO).await.unwrap()).unwrap() } /// End a catalog-only broadcast cleanly. @@ -10165,11 +10201,13 @@ async fn a_return_after_an_abandoned_switch_restores_priority() { let (old, mut old_catalog) = publish_bare(&origin, Some(&instance)); let mut old_audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); write_aac(&mut old_audio, 0); - let export = open_export(crate::source::resolved(&origin, "live").await) - .await - .unwrap() - .with_delay(Duration::from_millis(500)) - .with_replay(); + let export = open_export( + crate::source::resolved(&origin, "live").await, + Duration::from_millis(500), + ) + .await + .unwrap() + .with_replay(); let mut follower = super::Follower::new(export) .unwrap() .with_follow(Duration::from_secs(10)) @@ -10282,9 +10320,12 @@ async fn a_follower_built_after_its_route_went_follows_the_return() { let origin = crate::source::produce_origin(); let instance = moq_net::Instance::random(); let first = publish_bare(&origin, Some(&instance)); - let export = open_export(crate::source::resolved(&origin, "live").await) - .await - .unwrap(); + let export = open_export( + crate::source::resolved(&origin, "live").await, + std::time::Duration::ZERO, + ) + .await + .unwrap(); let start = kio::clock::now(); finish(first); kio::clock::sleep(Duration::from_secs(1)).await; @@ -10328,7 +10369,7 @@ async fn a_follower_takes_its_own_instance_winning_back_for_no_replacement() { let mut catalog = crate::catalog::Producer::<()>::new(&mut served.broadcast().clone(), Default::default()).unwrap(); served.announce().unwrap(); let source = crate::source::resolved(&origin, "pool/job").await; - let mut follower = super::Follower::new(open_export(source).await.unwrap()).unwrap(); + let mut follower = super::Follower::new(open_export(source, std::time::Duration::ZERO).await.unwrap()).unwrap(); // A cheaper route takes the path for a while, then goes. let cheaper = origin.route("pool/job", route(1)).unwrap(); @@ -10431,7 +10472,7 @@ async fn drain_follower(follower: &mut super::Follower) -> (Vec, Option::new(&mut exact.broadcast().clone(), Default::default()).unwrap(); let mut track = aac_rendition(&mut exact.broadcast().clone(), &mut catalog, "a.aac"); let source = crate::source::resolved(&origin, "pool/job").await; - let export = open_export(source) - .await - .unwrap() - .with_delay(RECORDING_MAX_AGE) - .with_replay(); + let export = open_export(source, RECORDING_MAX_AGE).await.unwrap().with_replay(); let mut follower = super::Follower::new(export) .unwrap() .with_follow(Duration::from_secs(10)); @@ -10481,7 +10518,7 @@ async fn a_follower_continues_through_a_same_instance_handoff_after_a_gap() { ); assert_eq!(pes_count(&frames), 20, "every frame of both routes went out"); - assert_skip_discontinuity(&frames, 1); + assert_skip_discontinuity(&frames, 0); assert_eq!(follower.export().discontinuity(), 0); assert!( pats(&frames).iter().all(|&(version, _)| version == 0), @@ -10503,7 +10540,7 @@ async fn a_follower_ignores_a_covering_prefix_while_the_exact_route_serves() { let exact = origin.route("pool/job", route(moq_net::Instance::random())).unwrap(); let catalog = crate::catalog::Producer::<()>::new(&mut exact.broadcast().clone(), Default::default()).unwrap(); let source = crate::source::resolved(&origin, "pool/job").await; - let mut follower = super::Follower::new(open_export(source).await.unwrap()).unwrap(); + let mut follower = super::Follower::new(open_export(source, std::time::Duration::ZERO).await.unwrap()).unwrap(); let quiet = Duration::from_secs(1); let pool = origin @@ -10586,7 +10623,7 @@ async fn a_cancelled_replacement_after_the_follow_can_follow_again() { let source = crate::source::resolved(&origin, "live").await; let stream = source.catalog::<()>(format).await.unwrap(); let follow = Duration::from_secs(10); - let mut follower = super::Follower::new(Export::new(source, stream)) + let mut follower = super::Follower::new(Export::new(source, stream, std::time::Duration::ZERO)) .unwrap() .with_stitch(true) .with_follow(follow); @@ -10657,10 +10694,9 @@ async fn a_midstream_stitch_keeps_the_old_instance_playing_until_it_switches() { let origin = crate::source::produce_origin(); let (broadcast, mut catalog) = publish_bare(&origin, Some(&moq_net::Instance::random())); let mut track = aac_rendition(&mut broadcast.broadcast().clone(), &mut catalog, "a.aac"); - let export = open_export(crate::source::resolved(&origin, "live").await) + let export = open_export(crate::source::resolved(&origin, "live").await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); let mut follower = super::Follower::new(export).unwrap().with_stitch(true); @@ -10776,10 +10812,9 @@ fn audio_pts(frames: &[Frame]) -> Vec { async fn a_late_rendition_never_reads_a_replacement() { let origin = crate::source::produce_origin(); let (old, mut old_catalog) = publish_live::<()>(&origin, Default::default()); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); assert!(drain_frames(&mut export).await.is_empty(), "nothing listed yet"); @@ -10837,10 +10872,10 @@ async fn a_repointed_si_entry_never_reads_a_replacement() { let mut export = open_ts( crate::source::resolved(&origin, "live").await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..1_000).step_by(40) { write_h264(&mut video, ms); @@ -10884,10 +10919,9 @@ async fn a_repointed_si_entry_never_reads_a_replacement() { async fn a_switch_reads_the_instance_it_followed() { let origin = crate::source::produce_origin(); let (_old, _old_catalog) = publish_live::<()>(&origin, Default::default()); - let export = open_export(crate::source::resolved(&origin, "live").await) + let export = open_export(crate::source::resolved(&origin, "live").await, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); let (followed, mut followed_catalog) = publish_live::<()>(&origin, Default::default()); @@ -10926,7 +10960,7 @@ async fn a_backlog_stays_in_the_source_until_due() { let mut config = AudioConfig::new(AAC { profile: 2 }, 44100, 2); config.description = Some(Bytes::from_static(&[0x12, 0x10])); let mut live = Live::audio(config); - let mut export = open_export(live.source().await).await.unwrap().with_delay(delay); + let mut export = open_export(live.source().await, delay).await.unwrap(); let start = kio::clock::now(); live.track.write(raw_frame(0, b"first", true)).unwrap(); // A zero-delay slot needs a later frame to prove it is complete. @@ -10995,7 +11029,7 @@ async fn stalled_presentation_time_keeps_the_export_bounded() { for config in live.catalog.modify().unwrap().video.renditions.values_mut() { config.jitter = Some(jitter); } - let mut export = open_export(live.source().await).await.unwrap().with_delay(delay); + let mut export = open_export(live.source().await, delay).await.unwrap(); live.track.write(video_frame(pts(0), true)).unwrap(); let mut most = 0; for index in 1..500 { @@ -11162,9 +11196,9 @@ async fn export_placed(placed: Placed) -> crate::Result> { let mut exporter = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await? - .with_delay(RECORDING_MAX_AGE) .with_replay(); if let Some(rate) = placed.mux_rate { exporter = exporter.with_mux_rate(rate); @@ -11523,10 +11557,10 @@ async fn placed_sections_keep_their_source_positions() { let mut exporter = open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay() .with_mux_rate(RATE); let mut output = Vec::new(); @@ -11582,10 +11616,9 @@ async fn a_track_renamed_on_its_pid_across_returns_retires_the_one_before() { let (broadcast, mut catalog) = publish_live::(&origin, route()); let mut track = rendition(&mut broadcast.broadcast().clone(), &mut catalog, "0.aac"); let source = crate::source::resolved(&origin, "live").await; - let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang) + let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); let start = kio::clock::now(); for ms in (0..200).step_by(20) { @@ -11649,10 +11682,9 @@ async fn a_track_renamed_on_its_pid_across_a_return_is_quiet_from_its_listing() let (broadcast, mut catalog) = publish_live::(&origin, route()); let mut track = rendition(&mut broadcast.broadcast().clone(), &mut catalog, "0.aac"); let source = crate::source::resolved(&origin, "live").await; - let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang) + let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang, Duration::from_secs(1)) .await .unwrap() - .with_delay(Duration::from_secs(1)) .with_replay(); let start = kio::clock::now(); for ms in (0..200).step_by(20) { @@ -11708,10 +11740,9 @@ async fn a_switch_follows_the_last_pmt_out() { let (old, mut old_catalog) = publish_live::(&origin, route()); let mut audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); let source = crate::source::resolved(&origin, "live").await; - let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang) + let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut audio, ms); @@ -11756,10 +11787,9 @@ async fn a_switch_follows_the_last_pat_out() { let (old, mut old_catalog) = publish_live::(&origin, route()); let mut audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); let source = crate::source::resolved(&origin, "live").await; - let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang) + let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut audio, ms); @@ -11812,10 +11842,9 @@ async fn a_switch_past_an_instance_that_laid_no_tables_follows_the_last_pmt_out( let (first, mut first_catalog) = publish_live::(&origin, route()); let mut audio = aac_rendition(&mut first.broadcast().clone(), &mut first_catalog, "a.aac"); let source = crate::source::resolved(&origin, "live").await; - let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang) + let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang, RECORDING_MAX_AGE) .await .unwrap() - .with_delay(RECORDING_MAX_AGE) .with_replay(); for ms in (0..200).step_by(20) { write_aac(&mut audio, ms); @@ -11858,10 +11887,9 @@ async fn a_stitched_switch_keeps_the_healthy_legs_held_frames() { let route = || moq_net::origin::RouteOptions::default().with_instance(moq_net::Instance::random(), 0); let (old, mut old_catalog) = publish_live::<()>(&origin, route()); let mut old_audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, Duration::from_secs(3)) .await .unwrap() - .with_delay(Duration::from_secs(3)) .with_replay(); for ms in (0..1000).step_by(20) { write_aac(&mut old_audio, ms); @@ -11896,10 +11924,9 @@ async fn a_healthy_stitch_feeds_the_old_generation_until_the_new_clock_acquires( let route = || moq_net::origin::RouteOptions::default().with_instance(moq_net::Instance::random(), 0); let (old, mut old_catalog) = publish_live::<()>(&origin, route()); let mut old_audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); - let export = open_export(crate::source::resolved(&origin, "live").await) + let export = open_export(crate::source::resolved(&origin, "live").await, Duration::from_secs(3)) .await - .unwrap() - .with_delay(Duration::from_secs(3)); + .unwrap(); let mut follower = super::Follower::new(export) .unwrap() .with_stitch(true) @@ -11976,10 +12003,9 @@ async fn repeated_stitches_keep_the_delay() { let (first, mut catalog) = publish_live::<()>(&origin, route()); let audio = aac_rendition(&mut first.broadcast().clone(), &mut catalog, "a.aac"); publishers.push((first, catalog, audio)); - let export = open_export(crate::source::resolved(&origin, "live").await) + let export = open_export(crate::source::resolved(&origin, "live").await, Duration::from_secs(1)) .await - .unwrap() - .with_delay(Duration::from_secs(1)); + .unwrap(); let mut follower = super::Follower::new(export) .unwrap() .with_stitch(true) @@ -12059,10 +12085,9 @@ async fn an_expired_follow_stops_healthy_input_but_drains_its_buffered_tail() { let deadline = selected + follow; let media = Duration::from_millis(2000); let step = Duration::from_millis(20); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, delay) .await - .unwrap() - .with_delay(delay); + .unwrap(); let mut replacement = None; let mut frames = Vec::new(); let start = kio::clock::now(); @@ -12156,10 +12181,9 @@ async fn a_publisher_that_leaves_drains_its_buffered_tail_before_the_end() { let delay = Duration::from_millis(500); let left = Duration::from_millis(2000); let step = Duration::from_millis(20); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, delay) .await - .unwrap() - .with_delay(delay); + .unwrap(); let start = kio::clock::now(); let mut frames = Vec::new(); let end = loop { @@ -12219,10 +12243,12 @@ async fn a_replacement_leaving_before_it_commits_keeps_the_old_program() { let route = || moq_net::origin::RouteOptions::default().with_instance(moq_net::Instance::random(), 0); let (old, mut old_catalog) = publish_live::<()>(&origin, route()); let mut old_audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) - .await - .unwrap() - .with_delay(Duration::from_millis(500)); + let mut export = open_export( + crate::source::resolved(&origin, "live").await, + Duration::from_millis(500), + ) + .await + .unwrap(); let mut replacement = None; let mut frames = Vec::new(); for ms in (0..4000).step_by(20) { @@ -12282,10 +12308,9 @@ async fn a_newer_pending_instance_discards_the_unemitted_held_replacement() { let route = || moq_net::origin::RouteOptions::default().with_instance(moq_net::Instance::random(), 0); let (a, mut a_catalog) = publish_live::<()>(&origin, route()); let mut a_audio = aac_rendition(&mut a.broadcast().clone(), &mut a_catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, Duration::from_secs(3)) .await .unwrap() - .with_delay(Duration::from_secs(3)) .with_replay(); for ms in (0..1000).step_by(20) { write_aac(&mut a_audio, ms); @@ -12344,10 +12369,9 @@ async fn an_empty_finished_replacement_drains_the_old_program() { let route = || moq_net::origin::RouteOptions::default().with_instance(moq_net::Instance::random(), 0); let (old, mut catalog) = publish_live::<()>(&origin, route()); let mut audio = aac_rendition(&mut old.broadcast().clone(), &mut catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, Duration::from_secs(3)) .await .unwrap() - .with_delay(Duration::from_secs(3)) .with_replay(); for ms in (0..1000).step_by(20) { write_aac(&mut audio, ms); @@ -12396,10 +12420,9 @@ async fn cold_same_instance_return(pending: bool) { let (old, mut old_catalog) = publish_live::<()>(&origin, route()); let mut old_video = h264_rendition(&mut old.broadcast().clone(), &mut old_catalog, "v.avc1"); let mut old_audio = aac_rendition(&mut old.broadcast().clone(), &mut old_catalog, "a.aac"); - let mut export = open_export(crate::source::resolved(&origin, "live").await) + let mut export = open_export(crate::source::resolved(&origin, "live").await, Duration::from_secs(3)) .await .unwrap() - .with_delay(Duration::from_secs(3)) .with_replay(); let mut prefix = Vec::new(); let returned_instance = if pending { @@ -12496,3 +12519,30 @@ async fn a_same_instance_return_before_psi_rebuilds_cold_readers() { async fn a_pending_same_instance_return_rebuilds_cold_readers() { cold_same_instance_return(true).await; } + +/// Construction-time pacing keeps buffered PES ahead of the decoder clock. +#[tokio::test(start_paused = true)] +async fn fixed_delay_keeps_buffered_pes_ahead_of_the_clock() { + let origin = crate::source::produce_origin(); + let (live, mut catalog) = publish_live::<()>(&origin, Default::default()); + let mut audio = aac_rendition(&mut live.broadcast().clone(), &mut catalog, "a.aac"); + let mut export = open_export(crate::source::resolved(&origin, "live").await, Duration::from_secs(3)) + .await + .unwrap() + .with_replay(); + for ms in (0..5000).step_by(20) { + write_aac(&mut audio, ms); + } + let mut all = vec![export.next().await.unwrap().unwrap()]; + assert!(export.buffered() > 1, "several PES remain buffered"); + + audio.finish().unwrap(); + catalog.finish().unwrap(); + let (frames, end) = drain_to_end(&mut export).await; + end.unwrap(); + all.extend(frames); + assert_decodes_after_the_clock(&all); + // Joining retains half the delay plus its boundary frame. + assert_eq!(pes_count(&all), 76); + assert_skip_discontinuity(&all, 0); +} diff --git a/rs/moq-mux/src/container/ts/export_timing_test.rs b/rs/moq-mux/src/container/ts/export_timing_test.rs index d819e860a01..f4dc3dd3d1a 100644 --- a/rs/moq-mux/src/container/ts/export_timing_test.rs +++ b/rs/moq-mux/src/container/ts/export_timing_test.rs @@ -242,10 +242,9 @@ impl Live { /// A receiver joining now. async fn join(&self, every: Duration) -> Leg { - let export = super::export_test::open_export(self.source.clone()) + let export = super::export_test::open_export(self.source.clone(), DELAY) .await .unwrap() - .with_delay(DELAY) .with_mux_rate(self.rate); let woken = Arc::new(Woken(AtomicBool::new(true))); Leg { diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 93315b93318..0380f58f3ca 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -5837,10 +5837,11 @@ pub(super) mod test { let exporter = crate::container::ts::export_test::open_ts( crate::source::announced(&consumer).await, crate::catalog::CatalogFormat::Hang, + RECORDING_MAX_AGE, ) .await .unwrap(); - let mut exporter = exporter.with_delay(RECORDING_MAX_AGE).with_replay(); + let mut exporter = exporter.with_replay(); let mut ts = Vec::new(); while let Ok(Some(frame)) = kio::clock::timeout(RECORDING_MAX_AGE * 3, exporter.next()) .await @@ -9604,12 +9605,13 @@ pub(super) mod test { None => baseline = Some(observed.clone()), Some(expected) => assert_eq!(&observed, expected, "{kind:?} {layout}"), } - let mut export = - crate::container::ts::export_test::open_export(crate::source::announced(&consumer).await) - .await - .unwrap() - .with_delay(Duration::from_secs(30)) - .with_replay(); + let mut export = crate::container::ts::export_test::open_export( + crate::source::announced(&consumer).await, + Duration::from_secs(30), + ) + .await + .unwrap() + .with_replay(); let mut ts = Vec::new(); while let Ok(frame) = kio::clock::timeout(Duration::from_secs(100), export.next()).await { let Some(frame) = frame.unwrap() else { @@ -10239,11 +10241,13 @@ pub(super) mod test { "the first group ends at the break, one frame after its last picture" ); - let mut exporter = crate::container::ts::export_test::open_export(crate::source::announced(&consumer).await) - .await - .unwrap() - .with_delay(Duration::from_secs(30)) - .with_replay(); + let mut exporter = crate::container::ts::export_test::open_export( + crate::source::announced(&consumer).await, + Duration::from_secs(30), + ) + .await + .unwrap() + .with_replay(); let mut ts = Vec::new(); while let Ok(res) = kio::clock::timeout(Duration::from_secs(100), exporter.next()).await { let Some(frame) = res.expect("the exporter must take the open GOP after the break") else { @@ -11037,11 +11041,11 @@ pub(super) mod test { // groups and their boundary. Queued media is drained on its deadlines, without // generating output through an arbitrary silence before feeding the next phase. let delay = std::time::Duration::from_millis((RESET_LEAP_TICKS + RESET_PES * RESET_PES_TICKS) / 90); - let mut exporter = crate::container::ts::export_test::open_export(crate::source::announced(&consumer).await) - .await - .unwrap() - .with_delay(delay) - .with_replay(); + let mut exporter = + crate::container::ts::export_test::open_export(crate::source::announced(&consumer).await, delay) + .await + .unwrap() + .with_replay(); // A discontinuity before anything went out moves no clock, so the media before the // boundary goes out first, as it would live. let (frames, before_flags) = exported_discontinuities(&mut exporter).await; diff --git a/rs/moq-mux/src/container/ts/import_test.rs b/rs/moq-mux/src/container/ts/import_test.rs index 818e02dd488..5b0992bfee6 100644 --- a/rs/moq-mux/src/container/ts/import_test.rs +++ b/rs/moq-mux/src/container/ts/import_test.rs @@ -477,11 +477,11 @@ async fn import_export_import_roundtrip() { // Re-export to TS. `import` and `catalog` stay alive so the exporter can // subscribe to the finished, retained tracks. - let mut exporter = crate::container::ts::export_test::open_export(crate::source::announced(&consumer).await) - .await - .unwrap() - .with_delay(RECORDING_MAX_AGE) - .with_replay(); + let mut exporter = + crate::container::ts::export_test::open_export(crate::source::announced(&consumer).await, RECORDING_MAX_AGE) + .await + .unwrap() + .with_replay(); let mut out = BytesMut::new(); while let Ok(res) = kio::clock::timeout(DRAIN, exporter.next()).await { match res.expect("exporter error") { diff --git a/rs/moq-mux/src/container/ts/recovery_test.rs b/rs/moq-mux/src/container/ts/recovery_test.rs index 04d2f8721a7..e03c3f05d21 100644 --- a/rs/moq-mux/src/container/ts/recovery_test.rs +++ b/rs/moq-mux/src/container/ts/recovery_test.rs @@ -17,7 +17,7 @@ //! - Every delivered frame missing from the output is counted: `trimmed` before its track's //! first frame went out, `dropped` after, and each skip upstream is one of `gaps`. //! - Every unit arrives before it decodes, the continuity counters run unbroken, and the PCR -//! signals declared boundaries and delivery skips with PCR discontinuities. +//! signals declared boundaries and skipped clock slots with PCR discontinuities. //! - Each point a receiver acquires at carries its own PAT and PMT, and FFmpeg decodes the output //! without an error from there, presenting every picture that went out, once, as the //! continuous source decode's picture at the same presentation time. @@ -645,7 +645,7 @@ impl Bottleneck { struct Arm { profile: Profile, schedule: Schedule, - /// Expected PCR flags from declared boundaries and overlapping delivery skips. + /// Expected PCR flags from declared boundaries and skipped clock slots. breaks: usize, /// A bottleneck after the faults. path: Option, @@ -668,7 +668,7 @@ fn arm(profile: Profile, schedule: Schedule) -> Arm { } impl Arm { - /// Expect this many PCR flags from the arm's boundaries and losses. + /// Expect this many PCR flags from the arm's clock breaks. fn breaks(mut self, count: usize) -> Self { self.breaks = count; self @@ -960,7 +960,9 @@ async fn run(recording: &Recording, arm: &Arm) -> Run { kio::clock::sleep_until(start + arm.join).await; let source = crate::source::announced(&consumer).await; - let mut export = (open_ts(source, crate::catalog::CatalogFormat::Hang).await.unwrap()).with_delay(DELAY); + let mut export = open_ts(source, crate::catalog::CatalogFormat::Hang, DELAY) + .await + .unwrap(); let mut frames = Vec::new(); let mut samples = Vec::new(); while let Ok(next) = kio::clock::timeout(STALL, export.next()).await { @@ -1418,7 +1420,7 @@ async fn recover(arm: Arm) -> Run { assert_eq!( count_discontinuity(&run.frames), arm.breaks, - "one PCR flag per declared boundary or overlapping delivery gap" + "one PCR flag per declared boundary or skipped clock interval" ); let reference = Reference::new(profile, run.ledger(true)); @@ -1479,7 +1481,7 @@ async fn a_skipped_group_resumes_at_the_next() { true => Delivery::Omitted, false => clean(slot), }) - .breaks(1), + .breaks(0), ) .await; assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); @@ -1496,7 +1498,7 @@ async fn a_cut_group_keeps_what_it_delivered() { false => clean(slot), } }) - .breaks(1), + .breaks(0), ) .await; let video = run.ledger(true); @@ -1511,7 +1513,7 @@ async fn a_cut_group_keeps_what_it_delivered() { /// resumes on the first keyframe delivered in time, on the same clock. #[tokio::test(start_paused = true)] async fn a_stall_drops_what_it_made_late() { - let run = recover(arm(Profile::Closed, |slot| hold(slot.video, slot)).breaks(1)).await; + let run = recover(arm(Profile::Closed, |slot| hold(slot.video, slot)).breaks(0)).await; let (video, audio) = (run.ledger(true), run.ledger(false)); assert!( video.units.iter().any(|unit| unit.on_time == Some(false)), @@ -1530,32 +1532,13 @@ async fn a_stall_drops_what_it_made_late() { /// deadline is dropped rather than sent late. #[tokio::test(start_paused = true)] async fn a_path_stall_drops_what_it_made_late() { - recover(arm(Profile::Closed, |slot| hold(true, slot)).breaks(1)).await; -} - -/// Audio and reordered video lose the same last-frame interval before a keyframe. Their -/// decode reserves differ, but the missing media shares one PCR flag. -#[tokio::test(start_paused = true)] -async fn a_short_outage_across_both_tracks_flags_once() { - let run = recover( - arm(Profile::Closed, |slot| { - let missing = if slot.video { - slot.group == 5 && slot.frame == 11 - } else { - (SPLICE - FRAME..SPLICE).contains(&slot.nominal) - }; - if missing { Delivery::Omitted } else { clean(slot) } - }) - .breaks(1), - ) - .await; - assert_eq!(run.stats.release.gaps, 2, "both tracks skipped media"); + recover(arm(Profile::Closed, |slot| hold(true, slot)).breaks(0)).await; } /// Audio loses an interval that ends before video's, though video's send-ahead releases its -/// loss first. Disjoint losses each flag the PCR, whichever is muxed first. +/// loss first. Both holes keep the padded PCR clock. #[tokio::test(start_paused = true)] -async fn disjoint_outages_across_both_tracks_flag_twice() { +async fn disjoint_outages_across_both_tracks_keep_the_clock() { let run = recover( arm(Profile::Closed, |slot| { let missing = if slot.video { @@ -1565,7 +1548,7 @@ async fn disjoint_outages_across_both_tracks_flag_twice() { }; if missing { Delivery::Omitted } else { clean(slot) } }) - .breaks(2), + .breaks(0), ) .await; assert_eq!(run.stats.release.gaps, 2, "both tracks skipped media"); @@ -1584,7 +1567,7 @@ fn hold(held: bool, slot: Slot) -> Delivery { /// audio's row stops counting and its quiet grows on the output's clock, and the audio resumes. #[tokio::test(start_paused = true)] async fn a_stalled_track_leaves_the_others_running() { - let run = recover(arm(Profile::Closed, stall_audio).breaks(1)).await; + let run = recover(arm(Profile::Closed, stall_audio).breaks(0)).await; let video = run.ledger(true); assert_eq!(out(video), video.units.len(), "every picture goes out"); assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); @@ -1632,7 +1615,7 @@ async fn skewed_audio_survives_a_video_skip() { (true, _) => clean(slot), (false, _) => Delivery::At(slot.nominal + Duration::from_millis(200)), }) - .breaks(1), + .breaks(0), ) .await; assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); @@ -1641,7 +1624,7 @@ async fn skewed_audio_survives_a_video_skip() { /// A skipped group around the 33-bit wrap keeps the clock, and the pictures either side decode. #[tokio::test(start_paused = true)] async fn a_skip_across_the_wrap_keeps_the_clock() { - let run = recover(arm(Profile::Wrap, skip_group_5).breaks(1)).await; + let run = recover(arm(Profile::Wrap, skip_group_5).breaks(0)).await; let video = run.ledger(true); let pts: Vec = video .units @@ -1686,7 +1669,7 @@ fn leading_out(video: &Ledger, group: usize) -> usize { /// are trimmed, while every group after a whole one keeps its own. #[tokio::test(start_paused = true)] async fn open_gop_trims_only_the_orphans() { - let run = recover(arm(Profile::Open, skip_and_cut).breaks(2)).await; + let run = recover(arm(Profile::Open, skip_and_cut).breaks(0)).await; let video = run.ledger(true); assert_eq!( (leading_out(video, 6), leading_out(video, 9)), @@ -1707,7 +1690,7 @@ async fn open_gop_trims_only_the_orphans() { async fn mpeg2_open_gop_trims_only_the_orphans() { let run = recover(Arm { join: Duration::from_millis(1100), - ..arm(Profile::Mpeg2, skip_and_cut).breaks(2) + ..arm(Profile::Mpeg2, skip_and_cut).breaks(0) }) .await; let video = run.ledger(true); @@ -1729,7 +1712,7 @@ async fn mpeg2_open_gop_trims_only_the_orphans() { /// its own, and the output decodes straight through the CRA it resumes at. #[tokio::test(start_paused = true)] async fn hevc_trims_only_orphaned_rasl() { - let run = recover(arm(Profile::Rasl, skip_and_cut).breaks(2)).await; + let run = recover(arm(Profile::Rasl, skip_and_cut).breaks(0)).await; let video = run.ledger(true); assert_eq!( (leading_out(video, 6), leading_out(video, 9)), @@ -1746,7 +1729,7 @@ async fn hevc_trims_only_orphaned_rasl() { /// HEVC RADL pictures reference only their own IDR, so after a skip or a cut they go out with it. #[tokio::test(start_paused = true)] async fn hevc_keeps_radl_after_a_hole() { - let run = recover(arm(Profile::Radl, skip_and_cut).breaks(2)).await; + let run = recover(arm(Profile::Radl, skip_and_cut).breaks(0)).await; let video = run.ledger(true); assert_eq!( (leading_out(video, 6), leading_out(video, 9)), @@ -1783,7 +1766,7 @@ async fn a_late_catalog_starts_on_the_group_in_flight() { false => clean(slot), } }) - .breaks(1) + .breaks(0) }) .await; let video = run.ledger(true); @@ -1804,14 +1787,14 @@ async fn a_skip_across_a_resolution_change_takes_the_new_parameters() { false => clean(slot), } }) - .breaks(1), + .breaks(0), ) .await; assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); } /// The clock jumps 20 s at a declared discontinuity, and the path loses the group before the -/// break and the one after it: the output signals the declared break and the delivery skip +/// break and the one after it: the output signals only the declared break /// on the PCR, and resumes on the keyframe after. #[tokio::test(start_paused = true)] async fn a_declared_discontinuity_survives_loss_either_side() { @@ -1822,14 +1805,14 @@ async fn a_declared_discontinuity_survives_loss_either_side() { false => clean(slot), } }) - .breaks(2), + .breaks(1), ) .await; assert_eq!(run.stats.release.dropped, 0); } /// The path loses the marker that declares the break, and every frame on either side arrives: -/// the export marks the overlapping delivery skips and recovered clock jump once. +/// the export marks the recovered clock jump once. #[tokio::test(start_paused = true)] async fn a_lost_discontinuity_marker_still_breaks_the_clock() { let run = recover( @@ -1854,7 +1837,7 @@ async fn a_congested_path_sheds_what_it_made_late() { queue: 2 * DELAY, congested: |at| (Duration::from_millis(1500)..Duration::from_millis(3500)).contains(&at), }), - ..arm(Profile::Closed, clean).breaks(1) + ..arm(Profile::Closed, clean).breaks(0) }) .await; let video = run.ledger(true); @@ -1941,8 +1924,8 @@ async fn long_faults_hold_their_latency() { // Two seconds of every twenty. congested: |at| (17..19).contains(&(at.as_secs() % 20)) && at < Duration::from_secs(100), }), - // Five cycles, each with four scheduled losses and one shared congestion interval. - ..arm(Profile::Long, faults).breaks(5 * 5) + // Live pacing pads these delivery holes while media is absent, so the PCR clock never jumps. + ..arm(Profile::Long, faults).breaks(0) }) .await; assert!(run.stats.release.gaps >= 20, "every fault skipped: {:?}", run.stats); diff --git a/rs/moq-mux/src/container/ts/release.rs b/rs/moq-mux/src/container/ts/release.rs index 97b8f43d667..7b5dda0c650 100644 --- a/rs/moq-mux/src/container/ts/release.rs +++ b/rs/moq-mux/src/container/ts/release.rs @@ -43,9 +43,11 @@ pub struct Release { impl Release { /// Subscribe to the catalog of `source`, waiting for the `m2ts` section that names its track. /// + /// Each object is written `delay` after its PCR time; late objects are dropped. + /// /// Fails for [`CatalogFormat::Msf`], which has no `m2ts` section. #[cfg(feature = "async")] - pub async fn new(source: crate::Source, catalog_format: CatalogFormat) -> crate::Result { + pub async fn new(source: crate::Source, catalog_format: CatalogFormat, delay: Duration) -> crate::Result { if catalog_format == CatalogFormat::Msf { return Err( anyhow::anyhow!("the MSF catalog has no `m2ts` passthrough track; read the hang catalog").into(), @@ -53,29 +55,22 @@ impl Release { } let mut subscribing = crate::catalog::Subscribing::new(&source.broadcast(), catalog_format); let catalog = kio::wait(|waiter| subscribing.poll_ok(waiter)).await?; - let mut release = Self { + let mut jitter = jitter::Buffer::new(delay); + jitter.expect([()]); + let release = Self { source, catalog: Some(catalog), track: None, subscribed: kio::clock::now(), finished: false, released: false, - delay: Duration::ZERO, - jitter: jitter::Buffer::new(Duration::ZERO), + delay, + jitter, replay: false, }; - release.jitter = release.jitter(); Ok(release) } - /// Write each object this long after its PCR time. An object that arrives later is dropped. - /// Defaults to [`Duration::ZERO`], which holds nothing. - pub fn with_delay(mut self, delay: Duration) -> Self { - self.delay = delay; - self.jitter = self.jitter(); - self - } - /// Read the track from the oldest group the delay still reaches instead of the live edge, /// and anchor the clock on the first object, for a test that writes a broadcast whole first. #[cfg(test)] @@ -85,6 +80,7 @@ impl Release { self } + #[cfg(test)] fn jitter(&self) -> jitter::Buffer<(), Frame> { let mut jitter = jitter::Buffer::new(self.delay); #[cfg(test)] diff --git a/rs/moq-mux/src/container/ts/release_test.rs b/rs/moq-mux/src/container/ts/release_test.rs index ae7b665a661..3adab037e39 100644 --- a/rs/moq-mux/src/container/ts/release_test.rs +++ b/rs/moq-mux/src/container/ts/release_test.rs @@ -182,8 +182,10 @@ fn spawn_release(republished: &Republished) -> tokio::task::JoinHandle<(Vec<(Ins let consumer = republished.consumer.clone(); tokio::spawn(async move { let source = crate::source::announced(&consumer).await; - let release = Release::new(source, CatalogFormat::Hang).await.unwrap(); - drain(release.with_delay(Duration::from_millis(500))).await + let release = Release::new(source, CatalogFormat::Hang, Duration::from_millis(500)) + .await + .unwrap(); + drain(release).await }) } @@ -204,10 +206,9 @@ async fn a_capture_is_released_byte_identical_on_its_own_pcr() { for data in [KYRION, KYRION_H264] { let imported = import(data).await; let source = crate::source::announced(&imported.consumer).await; - let release = Release::new(source, CatalogFormat::Hang) + let release = Release::new(source, CatalogFormat::Hang, RECORDING) .await .unwrap() - .with_delay(RECORDING) .with_replay(); let (released, release) = drain(release).await; @@ -250,9 +251,11 @@ async fn a_flagged_jump_crosses_without_holding_the_output() { } let republished = republish(objects.clone(), move |i, _| arrivals[i]); let source = crate::source::announced(&republished.consumer).await; - let release = Release::new(source, CatalogFormat::Hang).await.unwrap(); + let release = Release::new(source, CatalogFormat::Hang, Duration::from_millis(500)) + .await + .unwrap(); let delay = Duration::from_millis(500); - let (released, release) = drain(release.with_delay(delay)).await; + let (released, release) = drain(release).await; let out = bytes(released.iter().map(|(_, frame)| &frame.payload)); assert!( @@ -467,7 +470,7 @@ fn pids(data: &[u8]) -> std::collections::BTreeSet { async fn an_msf_catalog_is_refused() { let imported = import(KYRION).await; let source = crate::source::announced(&imported.consumer).await; - let Err(err) = Release::new(source, CatalogFormat::Msf).await else { + let Err(err) = Release::new(source, CatalogFormat::Msf, std::time::Duration::ZERO).await else { panic!("a release reading the MSF catalog must fail"); }; assert!(err.to_string().contains("MSF"), "{err}"); @@ -488,10 +491,7 @@ async fn a_backlog_stays_in_the_source_until_due() { Duration::from_micros(object.micros - origin) }); let source = crate::source::announced(&republished.consumer).await; - let mut release = Release::new(source, CatalogFormat::Hang) - .await - .unwrap() - .with_delay(delay); + let mut release = Release::new(source, CatalogFormat::Hang, delay).await.unwrap(); let first = release.next().await.unwrap().unwrap(); let first = objects .iter() diff --git a/rs/moq-mux/src/container/ts/revision_test.rs b/rs/moq-mux/src/container/ts/revision_test.rs index 624b6eea09c..eeaf1a804e0 100644 --- a/rs/moq-mux/src/container/ts/revision_test.rs +++ b/rs/moq-mux/src/container/ts/revision_test.rs @@ -92,10 +92,10 @@ async fn reexport_frames(spans: &[(u16, &Span<'_>)], intervals: u64) -> Vec export, None => { let source = crate::source::announced(&consumer).await; - let opened = super::export_test::open_ts(source, crate::catalog::CatalogFormat::Hang) + let opened = super::export_test::open_ts(source, crate::catalog::CatalogFormat::Hang, DELAY) .await .unwrap(); - let export = export.insert(opened.with_delay(DELAY).with_replay()); + let export = export.insert(opened.with_replay()); // The first frame out carries the tables built from this span alone. let first = kio::clock::timeout(DELAY * 2, export.next()).await; frames.push(first.expect("a first frame").unwrap().unwrap()); @@ -697,10 +697,10 @@ fn marks(frames: &[Frame], pid: u16, marks: &[u8]) -> Vec { /// The export of `consumer`. async fn export_of(consumer: &moq_net::broadcast::Consumer) -> super::Export> { let source = crate::source::announced(consumer).await; - let opened = super::export_test::open_ts(source, crate::catalog::CatalogFormat::Hang) + let opened = super::export_test::open_ts(source, crate::catalog::CatalogFormat::Hang, DELAY) .await .unwrap(); - opened.with_delay(DELAY).with_replay() + opened.with_replay() } /// Open the export of `consumer` once its first frame is out. @@ -738,10 +738,10 @@ async fn a_pid_recorded_after_its_track_takes_the_held_frames_along() { write_aac(&mut [&mut first], millis); } let source = crate::source::announced(&consumer).await; - let opened = super::export_test::open_ts(source, crate::catalog::CatalogFormat::Hang) + let opened = super::export_test::open_ts(source, crate::catalog::CatalogFormat::Hang, DELAY) .await .unwrap(); - let mut export = opened.with_delay(DELAY).with_replay(); + let mut export = opened.with_replay(); let out = kio::clock::timeout(DELAY * 2, export.next()).await; let mut frames = vec![out.expect("the tables went out").unwrap().unwrap()]; settle(&mut export, &mut frames).await; diff --git a/rs/moq-mux/src/container/ts/routes_test.rs b/rs/moq-mux/src/container/ts/routes_test.rs index 9bb6e43f42f..10398ccb968 100644 --- a/rs/moq-mux/src/container/ts/routes_test.rs +++ b/rs/moq-mux/src/container/ts/routes_test.rs @@ -187,6 +187,7 @@ async fn mpeg2_video_rides_whole_and_survives_a_round_trip() { super::export_test::open_ts( crate::source::announced(&imported.consumer).await, crate::catalog::CatalogFormat::Hang, + Duration::from_secs(30), ) .await .unwrap(), diff --git a/rs/moq-mux/src/container/ts/schedule.rs b/rs/moq-mux/src/container/ts/schedule.rs index cb17a1554d2..6d28bf9086b 100644 --- a/rs/moq-mux/src/container/ts/schedule.rs +++ b/rs/moq-mux/src/container/ts/schedule.rs @@ -195,8 +195,6 @@ struct Unit { /// Packets already laid out. sent: usize, keyframe: bool, - /// Delivery loss precedes this unit; the next PCR at its release flags it. - gap: bool, /// It was pushed after its due slot went out, so it goes out as soon as it can. late: bool, /// The first byte's placement in nanoseconds and payload prefix (the section pointer field). @@ -422,7 +420,7 @@ fn with_pcr(packets: &[u8], ticks: u64) -> Option> { /// One grid slot's worth of output. pub(super) struct Slot { pub index: u128, - /// Delivery loss starts here, before any backfilled slots after a clock jump. + /// The first PCR after skipped schedule slots declares a clock jump. pub discontinuity: bool, /// The PCR that opens the slot, in system clock ticks. pub pcr: u64, @@ -627,8 +625,6 @@ pub(super) struct Schedule { /// The unit held packets last went out in, by `seq`: what is muxed after them waits until it /// has finished, as it rides a lane of its own. deferred: Option, - /// A delivery skip precedes the next unit queued. - gap: bool, } impl Schedule { @@ -656,7 +652,6 @@ impl Schedule { clocked: 0, unclocked: Vec::new(), deferred: None, - gap: false, } } @@ -837,11 +832,6 @@ impl Schedule { self.push_frames(pid, timing, packets, keyframe, Vec::new()) } - /// Mark delivery loss before the next unit, leaving earlier queued media on its clock. - pub fn skip(&mut self) { - self.gap = true; - } - /// Queue an access unit on `pid` (whole TS packets, the tables muxed ahead of it first) /// with its exact transport deadline and decoder removal time. A PES that packs several /// frames lists each one's decode time and elementary bytes, in decode order, which leave @@ -889,12 +879,6 @@ impl Schedule { keyframe: bool, mut frames: Vec<(u128, usize)>, ) -> anyhow::Result<()> { - let mut gap = std::mem::take(&mut self.gap); - if gap { - // The resumed unit keeps its send-ahead, rather than taking the slot after the - // old unit as though they ran continuously across the missing interval. - self.last = None; - } if let Some(tables) = self.lead.take() { let pmt = self.pmt.context("program tables without a PMT PID")?; let clock = self.psi.as_ref().context("program tables never set")?.clock; @@ -915,14 +899,11 @@ impl Schedule { }; // The unit they lead keeps its own start. let last = self.last; - // The tables release first, so a long outage's clock jump lands on them. - self.enqueue(pmt, lead, Vec::new(), Some(program)).gap = std::mem::take(&mut gap); + self.enqueue(pmt, lead, Vec::new(), Some(program)); self.last = last; self.retable(); } if packets.is_empty() { - // The loss waits for the next unit that goes out. - self.gap |= gap; return Ok(()); } if frames.len() < 2 { @@ -971,7 +952,6 @@ impl Schedule { self.horizon = self.horizon.max(Some(last.div_ceil(PCR_INTERVAL.as_nanos()))); let unit = self.enqueue(pid, timing, packets, None); unit.keyframe = keyframe; - unit.gap = gap; unit.place = place; unit.frames = frames; Ok(()) @@ -980,7 +960,11 @@ impl Schedule { /// Push a unit of `packets` on `pid` due by `timing`, after every unit pushed before it. fn enqueue(&mut self, pid: u16, timing: Timing, packets: Vec, tables: Option) -> &mut Unit { let due = slot(timing.by); - let start = self.last.map_or(due, |last| (last + 1).min(due)); + // A long silence cannot let resumed media precede its own send-ahead. + let start = self + .last + .filter(|last| due.saturating_sub(*last) <= BACKFILL) + .map_or(due, |last| (last + 1).min(due)); self.last = Some(due); let runs = self.runs(&packets); let seq = self.seq; @@ -994,7 +978,6 @@ impl Schedule { packets, sent: 0, keyframe: false, - gap: false, late: self.next.is_some_and(|next| due < next), place: None, held: 0, @@ -1070,7 +1053,6 @@ impl Schedule { self.clocked = 0; self.unclocked.clear(); self.deferred = None; - self.gap = false; } /// The first slot `unit` may go out in: [`ahead`] of its due slot, and never before the @@ -1157,15 +1139,7 @@ impl Schedule { } // A long outage jumps to the start of its backfill. Flag that first PCR, not a // still-queued slot on the old clock, nor the media slot after the backfill. - let jumped = self.next.is_some_and(|next| index > next); - let through = if jumped { index + BACKFILL } else { index }; - let mut discontinuity = false; - for i in 0..self.units.len() { - if self.units[i].gap && self.release(&self.units[i]) <= through { - self.units[i].gap = false; - discontinuity = true; - } - } + let discontinuity = self.next.is_some_and(|next| index > next); self.next = Some(index + 1); if let Some(rate) = self.rate { let mut out = self.padded(index, rate)?; @@ -1887,10 +1861,57 @@ mod tests { assert_eq!(tables, 2); } - /// A long loss ahead of a placed unit flags the PCR its clock jumps to, though the tables + /// After a long silence, media keeps its send-ahead instead of riding the old clock. + #[test] + fn resumed_media_keeps_its_send_ahead() { + for rate in [None, Some(RATE)] { + for window in [Duration::ZERO, Duration::from_millis(500)] { + let mut schedule = Schedule::new(window); + schedule.set_rate(rate); + schedule.set_buffer(1, VIDEO); + for time in [1_000, 7_000] { + schedule.push(1, timing(time), unit(1, 1), true).unwrap(); + } + let mut sent = Vec::new(); + while let Some(slot) = schedule.next(None).unwrap() { + if !slot.packets.is_empty() { + sent.push(slot.index); + } + } + assert_eq!(sent.len(), 2); + assert!(sent[1] >= slot(ms(7_000)).saturating_sub(ahead(Some(&VIDEO), slot(window.as_nanos())))); + } + } + } + + /// Advancing the known source horizon through silence skips slots without a delivery marker. + #[test] + fn a_source_stall_flags_only_skipped_slots() { + for rate in [None, Some(RATE)] { + let mut schedule = Schedule::new(Duration::ZERO); + schedule.set_rate(rate); + schedule.push(1, timing(1_000), unit(1, 1), true).unwrap(); + let mut slots = Vec::new(); + while let Some(slot) = schedule.next(None).unwrap() { + slots.push((slot.index, slot.discontinuity)); + } + for horizon in [1_500, 7_000, 13_000] { + while let Some(slot) = schedule.next(Some(slot(ms(horizon)))).unwrap() { + slots.push((slot.index, slot.discontinuity)); + } + } + assert!(!slots[0].1); + for pair in slots.windows(2) { + assert_eq!(pair[1].1, pair[1].0 > pair[0].0 + 1); + } + assert_eq!(slots.iter().filter(|slot| slot.1).count(), 2); + } + } + + /// A long silence ahead of a placed unit flags the PCR its clock jumps to, though the tables /// leading the unit release slots before it. #[test] - fn a_skip_flags_the_jump_to_the_tables_leading_a_placed_unit() { + fn a_stall_flags_the_jump_to_the_tables_leading_a_placed_unit() { for rate in [None, Some(RATE)] { let mut schedule = Schedule::new(Duration::ZERO); schedule.set_rate(rate); @@ -1902,10 +1923,7 @@ mod tests { added: Vec::new(), }; let mut slots = Vec::new(); - for (place, skip) in [(ms(1_000), false), (ms(10_000), true)] { - if skip { - schedule.skip(); - } + for place in [ms(1_000), ms(10_000)] { schedule.lead(tables()); schedule .push_placed(7, (place, place), 0, unit(7, 1), true, Vec::new()) @@ -1924,27 +1942,10 @@ mod tests { .filter(|pair| pair[1].0 > pair[0].0 + 1) .map(|pair| pair[1].0) .collect(); - assert_eq!( - flagged, - jumps.last().copied().into_iter().collect::>(), - "rate {rate:?}" - ); + assert_eq!(flagged, jumps, "rate {rate:?}"); } } - /// An empty unit carries no loss, so the next unit that goes out does. - #[test] - fn a_skip_waits_for_a_unit_with_packets() { - let mut schedule = Schedule::new(Duration::ZERO); - schedule.skip(); - schedule.push(1, timing(100), Vec::new(), true).unwrap(); - schedule.push(1, timing(120), unit(1, 1), true).unwrap(); - let flagged = std::iter::from_fn(|| schedule.next(None).unwrap()) - .filter(|slot| slot.discontinuity) - .count(); - assert_eq!(flagged, 1); - } - /// A timeline that opens holds a placed unit until a PCR interval has followed its first /// tables, while the unit they lead goes out where it would have without the opening, padded /// or not. diff --git a/rs/moq-srt/src/ts.rs b/rs/moq-srt/src/ts.rs index 827288935ba..2fc24f8b61d 100644 --- a/rs/moq-srt/src/ts.rs +++ b/rs/moq-srt/src/ts.rs @@ -296,7 +296,7 @@ impl Subscriber { }; let source = moq_mux::Source::new(origin.clone(), broadcast); let catalog = source.catalog::(moq_mux::catalog::CatalogFormat::Hang).await?; - let export = ts::Export::new(source, catalog).with_delay(options.latency.max(MIN_DELAY)); + let export = ts::Export::new(source, catalog, options.latency.max(MIN_DELAY)); let follower = ts::Follower::new(export)? .with_follow(options.follow) .with_stitch(options.stitch); diff --git a/test/ts/README.md b/test/ts/README.md index eef0467e317..55259e4555c 100644 --- a/test/ts/README.md +++ b/test/ts/README.md @@ -756,9 +756,9 @@ Semantic loss also has paused-clock wire checks in `export_test.rs` and | Case | Contract | Test | |---|---|---| -| Skipped groups, short or longer than PCR backfill | Only the next PCR flags the skip; PAT/PMT versions stay unchanged; importer Health reports no unflagged PCR jump | `a_skipped_group_flags_the_clock` | -| Shared A/V outage, including different decode reserves | One program-level PCR flag for the overlapping losses | `a_short_outage_across_both_tracks_flags_once` | -| Same-instance route handoff after a gap | The same PCR-only flag, with no program change | `a_follower_continues_through_a_same_instance_handoff_after_a_gap` | +| Skipped groups, short or longer than PCR backfill | Padded continuing-clock holes stay unflagged; only the first PCR after skipped slots flags the jump; PAT/PMT versions stay unchanged; importer Health reports no unflagged PCR jump | `a_skipped_group_flags_the_clock` | +| Disjoint A/V outages with different decode reserves | Padded continuing-clock holes carry no flag | `disjoint_outages_across_both_tracks_keep_the_clock` | +| Same-instance route handoff after a gap | Only a clock jump flags the PCR, with no program change | `a_follower_continues_through_a_same_instance_handoff_after_a_gap` | These run in `just rs test -p moq-mux --lib` and the selected Rust CI gate. The raw lane retains the unchanged-packet loss contract below.