diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 31ee5b908ee..05bf1de888e 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -231,7 +231,9 @@ to `--delay` ahead of its decode time and no further than its decoder buffer holds. That send-ahead comes out of the delay, so the output trails a TS source by `--delay`. A frame that cannot arrive in time at the rate fails the export, as does video whose HRD declares a longer buffer delay than `--delay`; the -error names the `--delay` it needs. +error names the `--delay` it needs. Skipped media within the same broadcast +flags the next PCR as discontinuous without changing the PMT version. Raw +`--passthrough` keeps surviving packets unchanged, including across loss. ## Play diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index 4c5f3754cde..3670c587f84 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -1,4 +1,4 @@ -# TS fidelity and recovery +# [M] TS fidelity and recovery ## Goal @@ -117,14 +117,20 @@ Decided 2026-11-04 (maintainer, design review of ends, and the initial reservation releases at the first PTS. This is independent of the handoff. +Decided 2026-11-05 (maintainer, /quest-complete, for +[moq-dev/moq#7785](https://github.com/moq-dev/moq/pull/7785)): ✅ cross-track +loss coalescing keeps only each track's last loss. A slower track can flag +outage A again after a faster track resumed A and then B (three flags for +two outages); state stays bounded by the track count, and the extra flag +lands where loss happened. Not chosen: keeping every loss until all tracks +pass it (unbounded while a track stalls); collapsing losses older than the +release point minus the delay (more code, and assumes muxing in release +order). + PMT fragmentation (#5220), initial mux-rate measurement (#5287), and decoder-time admission (#5306) have landed, as have PSI runs (#5606) and catalog linger (#5305). -## Required - -- [A skipped group flags the next PCR](/quest/m1/ts-reliability/skip-flag.md) - a within-instance skip reaches the wire as a flagged PCR discontinuity - ## Related - [MSFTS mapping](/quest/m3/msfts-convergence.md) - document the exact equivalence and differences between the carriage modes diff --git a/quest/m1/ts-reliability/skip-flag.md b/quest/m1/ts-reliability/skip-flag.md deleted file mode 100644 index bb1fad04010..00000000000 --- a/quest/m1/ts-reliability/skip-flag.md +++ /dev/null @@ -1,41 +0,0 @@ -# [S] A skipped group flags the next PCR - -## Goal - -When a semantic `moq export ts` skips media within one broadcast instance, -the next PCR carries `discontinuity_indicator`, so a decoder or groomer sees -a marked break instead of an unflagged forward jump in PCR and PTS. - -## Plan - -Decided 2026-11-03 (user, /quest-import of -[moq-dev/moq#7741](https://github.com/moq-dev/moq/issues/7741)). - -- ✅ Flag the PCR only, with no PMT version bump: the program is unchanged, - only its clock jumped. Not chosen: mark it like a `--stitch` switch (every - PID flagged and a new PMT version), or won't fix. -- Milestone: ✅ m1, in this epic. Not chosen: m2. - -Today a skip (`Item::Gap`, `container/source.rs:601` at 976fece173) reaches -`ts::Export` only as a count that drives decodable-start recovery -(`export.rs:447`). PCR keeps following media time, so it jumps forward with -nothing marked. Our own importer counts that jump as a -`PCR_discontinuity_indicator_error` (`health.rs:261`). - -- Set `pcr_discontinuity` for the first PCR after a skip, as `rewind()` - (`export.rs:2852`) already does, without touching `version` or - `pat_version`. Decide where the flag lands relative to the backfilled slots - (`schedule.rs` `BACKFILL`), so no unflagged jump over 100 ms remains. -- Rewrite `a_skipped_group_keeps_the_clock` (`export_test.rs:8256`) and the - same-instance handoff test (`:10151`) to expect one flagged PCR and no - table version change. Feed the output back through the importer's `Health` - and assert no `PCR_discontinuity_indicator_error`. -- The raw `--passthrough` lane stays unflagged: under this epic's rule, loss - in the fallback never rewrites packets. -- Update the TS guide's coverage row for loss in the semantic lane. - -Public API: none. Wire: none (output TS only). - -## Closes - -- [#7741](https://github.com/moq-dev/moq/issues/7741) - a skip within one instance reaches the wire unflagged diff --git a/rs/moq-mux/benches/ts_schedule.rs b/rs/moq-mux/benches/ts_schedule.rs index 89be607de95..c0115bc8dbe 100644 --- a/rs/moq-mux/benches/ts_schedule.rs +++ b/rs/moq-mux/benches/ts_schedule.rs @@ -37,8 +37,14 @@ fn hold_heap() { } fn admission(c: &mut Criterion) { + for skipped in [false, true] { + admission_case(c, skipped); + } +} + +fn admission_case(c: &mut Criterion, skipped: bool) { hold_heap(); - let mut group = c.benchmark_group("ts_schedule"); + let mut group = c.benchmark_group(if skipped { "ts_schedule_skip" } else { "ts_schedule" }); group.measurement_time(Duration::from_secs(1)); group.warm_up_time(Duration::from_millis(200)); group.sample_size(10); @@ -63,7 +69,11 @@ fn admission(c: &mut Criterion) { } for frame in 0..frames { for pid in 1..=pids { - let at = (999 + frame * 40) * 1_000_000; + let gap = if skipped && frame >= frames / 2 { 4_000 } else { 0 }; + if skipped && frame == frames / 2 { + schedule.skip(); + } + let at = (999 + gap + frame * 40) * 1_000_000; schedule .push_frames( pid, diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index ec4b9102bf4..75806816de2 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -272,6 +272,8 @@ struct Pending { /// A frame waiting in the jitter buffer, with what it needs from the moment it was read. struct Queued { frame: Frame, + /// Delivery skips before this frame, excluding declared timeline boundaries. + gap: u64, /// When the frame decodes: its DTS, else its PTS. decode: Timestamp, /// Where the jitter buffer orders it ([`Track::release_key`]). @@ -317,6 +319,14 @@ struct Track { /// The marker, skip and cut counts of the sources the track read before this one, so the /// counts keep rising across a resubscription. base: (u64, u64, u64), + /// Delivery skips before the last frame muxed, so read-ahead cannot flag earlier output. + gap: u64, + /// Furthest presentation time muxed, so overlapping track losses share one break. + last_timestamp: Option, + /// The last delivery loss the track signalled or joined ([`coalesce`]). Only the last, so + /// state stays bounded by the track count: a slower track can flag an outage again after a + /// faster one moved past it, an extra flag where loss did happen. + lost: Option, /// How many unmarked steps back the track's timeline took ([`Self::steps_back`]), each /// counted as a marker of its own. steps: u64, @@ -399,6 +409,9 @@ impl Track { latest: kio::clock::Instant::ZERO, restart: 0, base: (0, 0, 0), + gap: 0, + last_timestamp: None, + lost: None, steps: 0, high: None, finished: false, @@ -840,6 +853,7 @@ impl Track { let Pending { frame, restart, + skip, arrived, read, .. @@ -888,6 +902,7 @@ impl Track { bytes: frame.payload.len() + self.key.name.len(), item: Queued { frame, + gap: skip.saturating_sub(restart), decode, key, dts, @@ -1224,6 +1239,8 @@ struct PesUnit { pid: u16, is_video: bool, keyframe: bool, + /// Delivery loss precedes this frame, so its scheduled PCR flags the gap. + skipped: bool, timestamp: Timestamp, /// Authored decode timestamp for a reordered (B-frame) video frame, in continuous /// (unwrapped) 90 kHz ticks (wrapped to the wire field in `write_pes`). `Some` only when @@ -3016,6 +3033,8 @@ where psi.led = psi.out.as_ref().map(|laid| laid.pmt.clone()); } for track in self.program.tracks.values_mut() { + track.last_timestamp = None; + track.lost = None; track.pack = None; track.parked.clear(); } @@ -3494,13 +3513,29 @@ where fn mux(&mut self, name: &str, queued: Queued) -> anyhow::Result<()> { let Queued { frame, + gap, decode, dts, description, pes, .. } = queued; - let kind = &self.program.tracks.get(name).context("missing track")?.kind; + let track = self.program.tracks.get_mut(name).context("missing track")?; + let skipped = gap > track.gap; + track.gap = gap; + // The track's missing interval runs from its furthest media muxed, or from the last + // reset when it has muxed none since, a start every such track shares (`None`). + let start = track.last_timestamp; + let end = frame.timestamp; + track.last_timestamp = Some(start.map_or(end, |start| start.max(end))); + let skipped = skipped && { + let others = self.program.tracks.values().filter_map(|track| track.lost); + let (lost, separate) = coalesce((start, end), others); + self.program.tracks.get_mut(name).context("missing track")?.lost = Some(lost); + separate + }; + let track = self.program.tracks.get(name).context("missing track")?; + let kind = &track.kind; let is_video = matches!(kind, Kind::Video(_)); // Refresh PSI at keyframes, MPEG-1/2 video's carried whole among them, or after the // interval lapses, and lead with a revision. @@ -3561,7 +3596,7 @@ where // The tables lead the unit that carries this frame, behind whatever went before it. if kind.placed() { self.program.psi_due |= psi; - return self.place(name, &frame, out); + return self.place(name, &frame, out, skipped); } match es_payload { // Section-framed verbatim (SCTE-35, ...) rides in private sections, not PES; @@ -3575,6 +3610,7 @@ where pid, is_video, keyframe, + skipped, timestamp: frame.timestamp, dts, stream_id: stream_id(&kind), @@ -3589,7 +3625,7 @@ where self.write_pes(&mut out, &unit, &es_payload)?; } } - self.push_unit(name, pid, decode, out, keyframe, Vec::new()) + self.push_unit(name, pid, decode, out, (keyframe, skipped), Vec::new()) } /// Queue a whole PES or section unchanged on `name`'s PID behind `tables`, its @@ -3600,7 +3636,7 @@ where /// unit that begins in it, so each one after counts on from there and leaves at its own time. /// The bytes ahead of the first end a frame an earlier PES began; they leave with it, /// which only overstates what the buffer holds. A PES without a PTS holds nothing there. - fn place(&mut self, name: &str, frame: &Frame, tables: Vec) -> anyhow::Result<()> { + fn place(&mut self, name: &str, frame: &Frame, tables: Vec, skipped: bool) -> anyhow::Result<()> { let track = self.program.tracks.get(name).context("missing track")?; let pes = match track.kind { Kind::PesPacket { stream_type } => Some( @@ -3640,6 +3676,9 @@ where self.program.schedule.set_buffer(pid, buffer); } self.lead_tables()?; + if skipped { + self.program.schedule.skip(); + } self.program.schedule.push_placed( pid, (place, decode.unwrap_or(place)), @@ -3677,7 +3716,8 @@ where let (size, fit) = track.fit(payload.len()); let place = place(unit.timestamp, samples, rate); // A PTS is a whole 90 kHz tick, so the frames run on within one of where they end. - let joins = start.is_some() + let joins = !unit.skipped + && start.is_some() && track.pack.as_ref().is_some_and(|pack| { pack.start == start && pack.rate == rate @@ -3724,7 +3764,7 @@ where } = pack; let mut out = Vec::new(); self.write_pes(&mut out, &unit, &payload)?; - self.push_unit(name, unit.pid, decode, out, unit.keyframe, frames) + self.push_unit(name, unit.pid, decode, out, (unit.keyframe, unit.skipped), frames) } /// Send on, in decode order, the packed frames of each track for which `done` holds, given @@ -3756,7 +3796,7 @@ where pid: u16, decode: Timestamp, out: Vec, - keyframe: bool, + (keyframe, skipped): (bool, bool), frames: Vec<(u128, usize)>, ) -> anyhow::Result<()> { let mut tables = std::mem::take(&mut self.program.tables); @@ -3786,6 +3826,9 @@ where } let by = decode.as_nanos().saturating_sub(track.drain().as_nanos()); self.lead_tables()?; + if skipped { + self.program.schedule.skip(); + } self.program.schedule.push_frames( pid, super::schedule::Timing { @@ -3920,7 +3963,7 @@ where let mut payload = slot.layout(&clock, pmt_pid, &NULL_PACKET); // The slot opens with its PCR, in the clock packet or the clock PID's packet that // carries it, whose adaptation field flags the break. - let discontinuity = std::mem::take(&mut self.program.pcr_discontinuity); + let discontinuity = std::mem::take(&mut self.program.pcr_discontinuity) || slot.discontinuity; if discontinuity { payload[5] |= 0x80; } @@ -4292,6 +4335,30 @@ fn dolby_removals(codec: Dolby, payload: &[u8], decode: u128) -> Vec<(u128, usiz frames } +/// A delivery loss's presentation interval: from the furthest media its track muxed before, +/// `None` when it had muxed none since the last reset, to the frame it resumed at. +type Lost = (Option, Timestamp); + +/// Merge `lost` with every interval in `others` it overlaps, and whether it overlapped none, +/// so it needs a PCR flag of its own. +/// +/// Tracks resume the same outage in different release slots, and in any order. One flag covers +/// their overlapping intervals, including a later read on another track that extends one +/// already signalled. Presentation time shares the content's timeline across their different +/// decode reserves, and every track that muxed nothing since the last reset starts its loss +/// there. +fn coalesce((start, end): Lost, others: impl IntoIterator) -> (Lost, bool) { + let shared: Vec = others + .into_iter() + .filter(|&(first, last)| start.is_none_or(|start| start < last) && first.is_none_or(|first| first < end)) + .collect(); + // `None` sorts first, so a merge keeps an unbounded start. + let lost = shared.iter().fold((start, end), |(start, end), &(first, last)| { + (start.min(first), end.max(last)) + }); + (lost, shared.is_empty()) +} + /// Whether `timestamp` has crossed into a later repetition slot than `last`. /// /// Slots are absolute on the media timeline (`floor(timestamp / interval)`) rather than @@ -5047,6 +5114,29 @@ mod tests { due, is_complete_section, si_due, slot, unwrap_pts, verbatim_buffer, }; + /// Losses resumed out of presentation order flag each outage once, however they interleave. + #[test] + fn coalesce_flags_each_outage_once() { + let ms = |ms: u64| Timestamp::from_millis(ms).unwrap(); + let a = (Some(ms(10_200)), ms(10_400)); + let b = (Some(ms(9_800)), ms(10_000)); + assert_eq!(super::coalesce(a, []), (a, true)); + assert_eq!(super::coalesce(b, [a]), (b, true), "a disjoint earlier outage"); + assert_eq!(super::coalesce(a, [a, b]), (a, false), "the first outage again"); + let wider = (Some(ms(9_900)), ms(10_300)); + assert_eq!( + super::coalesce(wider, [a, b]), + ((Some(ms(9_800)), ms(10_400)), false), + "spanning both" + ); + let reset = (None, ms(24_280)); + assert_eq!( + super::coalesce((None, ms(24_216)), [reset]), + ((None, ms(24_280)), false), + "both since a reset" + ); + } + /// The tables as they stand take the version after the last laid, whatever was set on the /// schedule meanwhile, and the last laid's own when they come back to it. The PAT keeps its /// own version until its network entry or PMT PID changes. diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index f0c1ea39498..b607696acee 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -6086,20 +6086,28 @@ pub(super) fn assert_on_time(frames: &[Frame]) -> usize { const WRAP: i128 = 1 << 33; let ts: Vec = frames.iter().flat_map(|f| f.payload.iter().copied()).collect(); // PCRs unwrapped from the first, which a start a window before the timeline may wrap. - let mut pcrs: Vec<(usize, f64)> = Vec::new(); + let mut pcrs: Vec<(usize, f64, bool)> = Vec::new(); for (at, base, _) in collect_pcrs(frames) { + let discontinuity = ts[at * 188 + 5] & 0x80 != 0; let base = base as i128; let value = match pcrs.last() { - Some(&(_, last)) => last + ((base - last as i128).rem_euclid(WRAP)) as f64, + Some(&(_, last, _)) => last + ((base - last as i128).rem_euclid(WRAP)) as f64, None => base as f64, }; - pcrs.push((at, value)); + pcrs.push((at, value, discontinuity)); } - let first = pcrs.first().map_or(0.0, |&(_, value)| value); + let first = pcrs.first().map_or(0.0, |&(_, value, _)| value); let time = |at: usize| { - let k = pcrs.partition_point(|&(index, _)| index <= at).checked_sub(1)?; - let (&(a, ta), &(b, tb)) = (pcrs.get(k)?, pcrs.get(k + 1)?); - Some(ta + (tb - ta) * (at - a) as f64 / (b - a) as f64) + let k = pcrs.partition_point(|&(index, _, _)| index < at).checked_sub(1)?; + let (&(a, ta, _), &(b, tb, discontinuity)) = (pcrs.get(k)?, pcrs.get(k + 1)?); + // A flagged jump starts after the preceding slot's bytes finished. Spreading it + // over those bytes would make media delivered on the old clock look late. + let span = if discontinuity { + PCR_INTERVAL.as_secs_f64() * 90_000.0 + } else { + tb - ta + }; + Some(ta + span * (at - a) as f64 / (b - a) as f64) }; // Each PES: its PID, DTS (else PTS) on the PCRs' unwrapped scale, and its last packet. let mut units: Vec<(u16, f64, usize)> = Vec::new(); @@ -8451,70 +8459,138 @@ async fn jitter_published_after_the_tables_keeps_the_encoders_clock() { assert_decodes_after_the_clock(&runner); } -/// A group the consumer skips (here one that never arrives) leaves the timeline where it was, -/// so the export keeps its clock: the program clock runs on with no break flagged, and a -/// frame the wait made late would be dropped like any other. +/// A delivery skip flags the next PCR without changing the program or its release clock. +/// Short holes keep the grid continuous; long ones exceed the scheduler's backfill window. #[tokio::test(start_paused = true)] -async fn a_skipped_group_keeps_the_clock() { - use crate::container::Container as _; - - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let consumer = broadcast.consume(); - let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); - let track = broadcast - .create_track("a.aac", hang::container::track_info(hang::catalog::PRIORITY.audio)) - .unwrap(); - { - let mut cfg = AudioConfig::new(AAC { profile: 2 }, 48_000, 2); - cfg.description = Some(Bytes::from_static(&[0x11, 0x90])); - cfg.container = Container::Legacy; - catalog - .modify() - .unwrap() - .audio - .renditions - .insert("a.aac".to_string(), cfg); - } +async fn a_skipped_group_flags_the_clock() { + let cases = [Duration::ZERO, Duration::from_millis(500)] + .into_iter() + .flat_map(|delay| [None, Some(1_000_000)].map(|rate| (delay, rate))); + for (delay, rate) in cases { + for missing in [0, 1, 60u64] { + println!("delay {delay:?}, mux rate {rate:?}, missing {missing} groups per hole"); + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let track = broadcast + .create_track("a.aac", hang::container::track_info(hang::catalog::PRIORITY.audio)) + .unwrap(); + { + let mut cfg = AudioConfig::new(AAC { profile: 2 }, 48_000, 2); + cfg.description = Some(Bytes::from_static(&[0x11, 0x90])); + cfg.container = Container::Legacy; + catalog + .modify() + .unwrap() + .audio + .renditions + .insert("a.aac".to_string(), cfg); + } - let delay = Duration::from_millis(500); - let mut export = open_export(crate::source::announced(&consumer).await) - .await - .unwrap() - .with_delay(delay); - let start = kio::clock::now(); - let mut out = Vec::new(); - // 100 ms groups of five 20 ms frames, a minute in; group 20 never arrives. - for sequence in 0..60u64 { - kio::clock::sleep_until(start + Duration::from_millis(sequence * 100)).await; - if sequence != 20 { - let mut group = track.create_group(moq_net::group::Info { sequence }).unwrap(); - for k in 0..5u64 { - let frame = Frame { - timestamp: Timestamp::from_millis(60_000 + sequence * 100 + k * 20).unwrap(), - duration: None, - payload: Bytes::from_static(&[0x21, 0x10, 0x04, 0x60]), - keyframe: k == 0, - }; - HangContainer::Legacy(crate::container::Kind::Audio) - .write(&mut group, &[frame]) - .unwrap(); + let mut export = open_export(crate::source::announced(&consumer).await) + .await + .unwrap() + .with_delay(delay); + if let Some(rate) = rate { + export = export.with_mux_rate(rate); } - group.finish().unwrap(); + let start = kio::clock::now(); + let mut out = Vec::new(); + // 100 ms groups of five 20 ms frames, a minute in, with two short or long holes. + for sequence in 0..80 + 2 * missing { + kio::clock::sleep_until(start + Duration::from_millis(sequence * 100)).await; + if !(20..20 + missing).contains(&sequence) && !(40 + missing..40 + 2 * missing).contains(&sequence) { + let mut group = track.create_group(moq_net::group::Info { sequence }).unwrap(); + for k in 0..5u64 { + let frame = Frame { + timestamp: Timestamp::from_millis(60_000 + sequence * 100 + k * 20).unwrap(), + duration: None, + payload: Bytes::from_static(&[0x21, 0x10, 0x04, 0x60]), + keyframe: k == 0, + }; + HangContainer::Legacy(crate::container::Kind::Audio) + .write(&mut group, &[frame]) + .unwrap(); + } + group.finish().unwrap(); + } + out.extend(poll_frames(&mut export)); + } + track.finish().unwrap(); + out.extend(drain_frames(&mut export).await); + + assert_eq!(export.discontinuity(), 0, "a skipped group is not a new program clock"); + assert_skip_discontinuity(&out, 2 * usize::from(missing > 0)); + let ts: Vec = out.iter().flat_map(|f| f.payload.iter().copied()).collect(); + // Each flag lands right before the audio that resumes after its hole. + let resumes: Vec = [20 + missing, 40 + 2 * missing] + .into_iter() + .filter(|_| missing > 0) + .map(|sequence| (60_000 + sequence * 100) * 90) + .collect(); + let mut reader = TsPacketReader::new(Cursor::new(&ts)); + let (mut flagged, mut after) = (false, Vec::new()); + while let Some(packet) = reader.read_ts_packet().unwrap() { + flagged |= packet + .adaptation_field + .is_some_and(|field| field.discontinuity_indicator); + if let Some(TsPayload::PesStart(pes)) = packet.payload + && std::mem::take(&mut flagged) + { + after.extend(pes.header.pts.map(|pts| pts.as_u64())); + } + } + assert_eq!(after, resumes, "the first PES after each flag resumes its hole"); + let (_, audio) = collect_pes_pts(&ts); + assert!( + audio.iter().any(|&pts| pts >= (60_000 + (70 + 2 * missing) * 100) * 90), + "the audio after the skip went out" + ); + assert_on_time(&out); } - out.extend(poll_frames(&mut export)); } - track.finish().unwrap(); - out.extend(drain_frames(&mut export).await); +} - assert_eq!(export.discontinuity(), 0, "a skipped group is not a new program clock"); - assert_eq!(count_discontinuity(&out), 0, "no PCR break is flagged"); - let ts: Vec = out.iter().flat_map(|f| f.payload.iter().copied()).collect(); - let (_, audio) = collect_pes_pts(&ts); +/// A same-program skip changes only the next PCR's flag, including across a backfilled gap. +fn assert_skip_discontinuity(frames: &[Frame], count: usize) { + assert_eq!(count_discontinuity(frames), count, "only one PCR per skip is flagged"); assert!( - audio.iter().any(|&pts| pts >= (60_000 + 5_000) * 90), - "the audio after the skip went out" + pats(frames).iter().all(|&(version, _)| version == 0), + "the PAT keeps its version" ); - assert_on_time(&out); + assert!( + pmts(frames).iter().all(|(version, _)| *version == 0), + "the PMT keeps its version" + ); + let mut health = super::health::Health::default(); + let mut continuity = super::health::Multiplex::default(); + let ts: Vec = frames.iter().flat_map(|f| f.payload.iter().copied()).collect(); + let mut reader = TsPacketReader::new(Cursor::new(&ts)); + while let Some(packet) = reader.read_ts_packet().unwrap() { + if let Some(TsPayload::Pmt(pmt)) = packet.payload { + health.pcr_pid(pmt.pcr_pid.map(|pid| pid.as_u16())); + break; + } + } + let start = kio::clock::Instant::now(); + for (count, packet) in ts.as_chunks::<188>().0.iter().enumerate() { + if packet[3] & 0x20 != 0 && packet[4] > 0 && packet[5] & 0x80 != 0 { + assert_ne!(packet[5] & 0x10, 0, "a skip flags only a PCR packet"); + } + let continuation = continuity.packet(packet); + health.packet( + packet, + continuation, + super::health::At { + count: count as u64 + 1, + arrival: start, + }, + ); + } + let mut stats = stats::Snapshot::default(); + health.errors().report(&mut stats); + assert_eq!(stats.pcr_discontinuity_indicator_error, 0, "no unflagged PCR jump"); + assert_eq!(stats.pcr_repetition_error, 0, "no unflagged gap after the flagged PCR"); } /// Timestamps restarting within one broadcast are a publisher bug, since a name always @@ -8758,8 +8834,8 @@ async fn moved_export( /// A rendition the catalog moves to another track, in its broadcast or a sibling one, goes on /// out of the PID the PMT announced, with that track's media and none of the one it left. The -/// continuity counters run on, and the move onto the same clock is no break. Its parameter sets -/// ride in-band, so the new track may carry another SPS. +/// continuity counters run on, and only the PCR flags the media skipped by the move. Its +/// parameter sets ride in-band, so the new track may carry another SPS. #[tokio::test(start_paused = true)] async fn a_moved_rendition_continues_on_its_pid() { use crate::container::test_util::{Moving, marks}; @@ -8776,11 +8852,7 @@ async fn a_moved_rendition_continues_on_its_pid() { "the new track's parameter sets go out in-band" ); assert_eq!(continuity_errors(&frames), 0); - assert_eq!( - count_discontinuity(&frames), - 0, - "a move onto the same clock is no break" - ); + assert_skip_discontinuity(&frames, 1); assert!( pmts(&frames).iter().all(|(version, _)| *version == 0), "the PMT stays as announced" @@ -8970,8 +9042,8 @@ fn unflagged(frames: &[Frame]) -> Vec { } /// The same instance back after every route went carries on in the same export, under the -/// program already announced: the PSI keeps its version, and nothing flags a break across a -/// gap the clock spans. Both ways a broadcast ends, a clean finish and a drop, continue alike. +/// program already announced: the PSI keeps its version, and only the PCR flags the gap. +/// Both ways a broadcast ends, a clean finish and a drop, continue alike. async fn same_instance_returns(finish: bool) { let origin = crate::source::produce_origin(); let instance = moq_net::Instance::random(); @@ -9026,7 +9098,7 @@ async fn same_instance_returns(finish: bool) { end.unwrap(); assert_eq!(pes_count(&frames), 20, "every frame of both spans went out"); - assert_eq!(count_discontinuity(&frames), 0, "a gap the clock spans is no break"); + assert_skip_discontinuity(&frames, 1); assert_eq!(export.discontinuity(), 0); assert!( pats(&frames).iter().all(|&(version, _)| version == 0), @@ -9110,7 +9182,7 @@ async fn a_return_keeps_catalog_and_group_history() { frames.extend(rest); end.expect("the publisher's final finish is clean"); assert_eq!(pes_count(&frames), 20, "each span goes out exactly once"); - assert_eq!(count_discontinuity(&frames), 0); + assert_skip_discontinuity(&frames, 1); assert_eq!(export.discontinuity(), 0); assert!(pats(&frames).iter().all(|&(version, _)| version == 0)); assert!(pmts(&frames).iter().all(|(version, _)| *version == 0)); @@ -10350,8 +10422,8 @@ async fn drain_follower(follower: &mut super::Follower) -> (Vec, Option, /// When the export starts. @@ -658,12 +660,21 @@ fn arm(profile: Profile, schedule: Schedule) -> Arm { Arm { profile, schedule, + breaks: profile.breaks(), path: None, join: Duration::ZERO, catalog: Duration::ZERO, } } +impl Arm { + /// Expect this many PCR flags from the arm's boundaries and losses. + fn breaks(mut self, count: usize) -> Self { + self.breaks = count; + self + } +} + /// What the schedule decided for one recorded frame, and what the export made of it. #[derive(Clone, Debug)] struct Unit { @@ -1406,8 +1417,8 @@ async fn recover(arm: Arm) -> Run { assert_continuous(&run.ts); assert_eq!( count_discontinuity(&run.frames), - profile.breaks(), - "a PCR discontinuity where the source declared one, and no other" + arm.breaks, + "one PCR flag per declared boundary or overlapping delivery gap" ); let reference = Reference::new(profile, run.ledger(true)); @@ -1463,10 +1474,13 @@ async fn a_clean_path_loses_nothing() { /// on the next keyframe. #[tokio::test(start_paused = true)] async fn a_skipped_group_resumes_at_the_next() { - let run = recover(arm(Profile::Closed, |slot| match slot.video && slot.group == 5 { - true => Delivery::Omitted, - false => clean(slot), - })) + let run = recover( + arm(Profile::Closed, |slot| match slot.video && slot.group == 5 { + true => Delivery::Omitted, + false => clean(slot), + }) + .breaks(1), + ) .await; assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); } @@ -1475,12 +1489,15 @@ async fn a_skipped_group_resumes_at_the_next() { /// settle their decode times, and loses only its tail. #[tokio::test(start_paused = true)] async fn a_cut_group_keeps_what_it_delivered() { - let run = recover(arm(Profile::Closed, |slot| { - match slot.video && slot.group == 5 && slot.frame >= 4 { - true => Delivery::Omitted, - false => clean(slot), - } - })) + let run = recover( + arm(Profile::Closed, |slot| { + match slot.video && slot.group == 5 && slot.frame >= 4 { + true => Delivery::Omitted, + false => clean(slot), + } + }) + .breaks(1), + ) .await; let video = run.ledger(true); let kept = video.units.iter().filter(|unit| unit.group == 5 && unit.output).count(); @@ -1494,7 +1511,7 @@ async fn a_cut_group_keeps_what_it_delivered() { /// resumes on the first keyframe delivered in time, on the same clock. #[tokio::test(start_paused = true)] async fn a_stall_drops_what_it_made_late() { - let run = recover(arm(Profile::Closed, |slot| hold(slot.video, slot))).await; + let run = recover(arm(Profile::Closed, |slot| hold(slot.video, slot)).breaks(1)).await; let (video, audio) = (run.ledger(true), run.ledger(false)); assert!( video.units.iter().any(|unit| unit.on_time == Some(false)), @@ -1513,7 +1530,45 @@ async fn a_stall_drops_what_it_made_late() { /// deadline is dropped rather than sent late. #[tokio::test(start_paused = true)] async fn a_path_stall_drops_what_it_made_late() { - recover(arm(Profile::Closed, |slot| hold(true, slot))).await; + recover(arm(Profile::Closed, |slot| hold(true, slot)).breaks(1)).await; +} + +/// Audio and reordered video lose the same last-frame interval before a keyframe. Their +/// decode reserves differ, but the missing media shares one PCR flag. +#[tokio::test(start_paused = true)] +async fn a_short_outage_across_both_tracks_flags_once() { + let run = recover( + arm(Profile::Closed, |slot| { + let missing = if slot.video { + slot.group == 5 && slot.frame == 11 + } else { + (SPLICE - FRAME..SPLICE).contains(&slot.nominal) + }; + if missing { Delivery::Omitted } else { clean(slot) } + }) + .breaks(1), + ) + .await; + assert_eq!(run.stats.release.gaps, 2, "both tracks skipped media"); +} + +/// Audio loses an interval that ends before video's, though video's send-ahead releases its +/// loss first. Disjoint losses each flag the PCR, whichever is muxed first. +#[tokio::test(start_paused = true)] +async fn disjoint_outages_across_both_tracks_flag_twice() { + let run = recover( + arm(Profile::Closed, |slot| { + let missing = if slot.video { + slot.group == 5 && slot.frame == 11 + } else { + (SPLICE - 5 * FRAME..SPLICE - 4 * FRAME).contains(&slot.nominal) + }; + if missing { Delivery::Omitted } else { clean(slot) } + }) + .breaks(2), + ) + .await; + assert_eq!(run.stats.release.gaps, 2, "both tracks skipped media"); } /// Hold what is `held` and due in [1.92 s, 4.4 s) until the end, delivering the rest cleanly. @@ -1529,7 +1584,7 @@ fn hold(held: bool, slot: Slot) -> Delivery { /// audio's row stops counting and its quiet grows on the output's clock, and the audio resumes. #[tokio::test(start_paused = true)] async fn a_stalled_track_leaves_the_others_running() { - let run = recover(arm(Profile::Closed, stall_audio)).await; + let run = recover(arm(Profile::Closed, stall_audio).breaks(1)).await; let video = run.ledger(true); assert_eq!(out(video), video.units.len(), "every picture goes out"); assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); @@ -1571,11 +1626,14 @@ fn stall_audio(slot: Slot) -> Delivery { /// Audio sent 200 ms behind the video, within the delay, survives a skipped video group whole. #[tokio::test(start_paused = true)] async fn skewed_audio_survives_a_video_skip() { - let run = recover(arm(Profile::Closed, |slot| match (slot.video, slot.group) { - (true, 5) => Delivery::Omitted, - (true, _) => clean(slot), - (false, _) => Delivery::At(slot.nominal + Duration::from_millis(200)), - })) + let run = recover( + arm(Profile::Closed, |slot| match (slot.video, slot.group) { + (true, 5) => Delivery::Omitted, + (true, _) => clean(slot), + (false, _) => Delivery::At(slot.nominal + Duration::from_millis(200)), + }) + .breaks(1), + ) .await; assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); } @@ -1583,7 +1641,7 @@ async fn skewed_audio_survives_a_video_skip() { /// A skipped group around the 33-bit wrap keeps the clock, and the pictures either side decode. #[tokio::test(start_paused = true)] async fn a_skip_across_the_wrap_keeps_the_clock() { - let run = recover(arm(Profile::Wrap, skip_group_5)).await; + let run = recover(arm(Profile::Wrap, skip_group_5).breaks(1)).await; let video = run.ledger(true); let pts: Vec = video .units @@ -1628,7 +1686,7 @@ fn leading_out(video: &Ledger, group: usize) -> usize { /// are trimmed, while every group after a whole one keeps its own. #[tokio::test(start_paused = true)] async fn open_gop_trims_only_the_orphans() { - let run = recover(arm(Profile::Open, skip_and_cut)).await; + let run = recover(arm(Profile::Open, skip_and_cut).breaks(2)).await; let video = run.ledger(true); assert_eq!( (leading_out(video, 6), leading_out(video, 9)), @@ -1649,7 +1707,7 @@ async fn open_gop_trims_only_the_orphans() { async fn mpeg2_open_gop_trims_only_the_orphans() { let run = recover(Arm { join: Duration::from_millis(1100), - ..arm(Profile::Mpeg2, skip_and_cut) + ..arm(Profile::Mpeg2, skip_and_cut).breaks(2) }) .await; let video = run.ledger(true); @@ -1671,7 +1729,7 @@ async fn mpeg2_open_gop_trims_only_the_orphans() { /// its own, and the output decodes straight through the CRA it resumes at. #[tokio::test(start_paused = true)] async fn hevc_trims_only_orphaned_rasl() { - let run = recover(arm(Profile::Rasl, skip_and_cut)).await; + let run = recover(arm(Profile::Rasl, skip_and_cut).breaks(2)).await; let video = run.ledger(true); assert_eq!( (leading_out(video, 6), leading_out(video, 9)), @@ -1688,7 +1746,7 @@ async fn hevc_trims_only_orphaned_rasl() { /// HEVC RADL pictures reference only their own IDR, so after a skip or a cut they go out with it. #[tokio::test(start_paused = true)] async fn hevc_keeps_radl_after_a_hole() { - let run = recover(arm(Profile::Radl, skip_and_cut)).await; + let run = recover(arm(Profile::Radl, skip_and_cut).breaks(2)).await; let video = run.ledger(true); assert_eq!( (leading_out(video, 6), leading_out(video, 9)), @@ -1725,6 +1783,7 @@ async fn a_late_catalog_starts_on_the_group_in_flight() { false => clean(slot), } }) + .breaks(1) }) .await; let video = run.ledger(true); @@ -1738,39 +1797,48 @@ async fn a_late_catalog_starts_on_the_group_in_flight() { /// hole both decode at the new size. #[tokio::test(start_paused = true)] async fn a_skip_across_a_resolution_change_takes_the_new_parameters() { - let run = recover(arm(Profile::Resized, |slot| { - match slot.video && (slot.group == 6 || (slot.group == 5 && slot.frame >= 4)) { - true => Delivery::Omitted, - false => clean(slot), - } - })) + let run = recover( + arm(Profile::Resized, |slot| { + match slot.video && (slot.group == 6 || (slot.group == 5 && slot.frame >= 4)) { + true => Delivery::Omitted, + false => clean(slot), + } + }) + .breaks(1), + ) .await; assert_eq!((run.stats.release.dropped, run.stats.release.gaps), (0, 1)); } /// The clock jumps 20 s at a declared discontinuity, and the path loses the group before the -/// break and the one after it: the output signals the break once, on the PCR, and resumes on the -/// keyframe after. +/// break and the one after it: the output signals the declared break and the delivery skip +/// on the PCR, and resumes on the keyframe after. #[tokio::test(start_paused = true)] async fn a_declared_discontinuity_survives_loss_either_side() { - let run = recover(arm(Profile::Spliced, |slot| { - match slot.video && (slot.group == 7 || (slot.group == 5 && slot.frame >= 4)) { - true => Delivery::Omitted, - false => clean(slot), - } - })) + let run = recover( + arm(Profile::Spliced, |slot| { + match slot.video && (slot.group == 7 || (slot.group == 5 && slot.frame >= 4)) { + true => Delivery::Omitted, + false => clean(slot), + } + }) + .breaks(2), + ) .await; assert_eq!(run.stats.release.dropped, 0); } /// The path loses the marker that declares the break, and every frame on either side arrives: -/// the export still finds the break where the clock jumps and signals it once. +/// the export marks the overlapping delivery skips and recovered clock jump once. #[tokio::test(start_paused = true)] async fn a_lost_discontinuity_marker_still_breaks_the_clock() { - let run = recover(arm(Profile::Spliced, |slot| match slot.marker { - true => Delivery::Omitted, - false => clean(slot), - })) + let run = recover( + arm(Profile::Spliced, |slot| match slot.marker { + true => Delivery::Omitted, + false => clean(slot), + }) + .breaks(1), + ) .await; assert_eq!(run.stats.release.dropped, 0); } @@ -1786,7 +1854,7 @@ async fn a_congested_path_sheds_what_it_made_late() { queue: 2 * DELAY, congested: |at| (Duration::from_millis(1500)..Duration::from_millis(3500)).contains(&at), }), - ..arm(Profile::Closed, clean) + ..arm(Profile::Closed, clean).breaks(1) }) .await; let video = run.ledger(true); @@ -1873,7 +1941,8 @@ async fn long_faults_hold_their_latency() { // Two seconds of every twenty. congested: |at| (17..19).contains(&(at.as_secs() % 20)) && at < Duration::from_secs(100), }), - ..arm(Profile::Long, faults) + // Five cycles, each with four scheduled losses and one shared congestion interval. + ..arm(Profile::Long, faults).breaks(5 * 5) }) .await; assert!(run.stats.release.gaps >= 20, "every fault skipped: {:?}", run.stats); diff --git a/rs/moq-mux/src/container/ts/schedule.rs b/rs/moq-mux/src/container/ts/schedule.rs index 117b4b3541d..930f72d8b35 100644 --- a/rs/moq-mux/src/container/ts/schedule.rs +++ b/rs/moq-mux/src/container/ts/schedule.rs @@ -195,6 +195,8 @@ struct Unit { /// Packets already laid out. sent: usize, keyframe: bool, + /// Delivery loss precedes this unit; the next PCR at its release flags it. + gap: bool, /// It was pushed after its due slot went out, so it goes out as soon as it can. late: bool, /// The first byte's placement in nanoseconds and payload prefix (the section pointer field). @@ -420,6 +422,8 @@ fn with_pcr(packets: &[u8], ticks: u64) -> Option> { /// One grid slot's worth of output. pub(super) struct Slot { pub index: u128, + /// Delivery loss starts here, before any backfilled slots after a clock jump. + pub discontinuity: bool, /// The PCR that opens the slot, in system clock ticks. pub pcr: u64, /// The media packets, in the order they were muxed. @@ -620,6 +624,8 @@ pub(super) struct Schedule { /// The unit held packets last went out in, by `seq`: what is muxed after them waits until it /// has finished, as it rides a lane of its own. deferred: Option, + /// A delivery skip precedes the next unit queued. + gap: bool, } impl Schedule { @@ -646,6 +652,7 @@ impl Schedule { clocked: 0, unclocked: Vec::new(), deferred: None, + gap: false, } } @@ -823,6 +830,11 @@ impl Schedule { self.push_frames(pid, timing, packets, keyframe, Vec::new()) } + /// Mark delivery loss before the next unit, leaving earlier queued media on its clock. + pub fn skip(&mut self) { + self.gap = true; + } + /// Queue an access unit on `pid` (whole TS packets, the tables muxed ahead of it first) /// with its exact transport deadline and decoder removal time. A PES that packs several /// frames lists each one's decode time and elementary bytes, in decode order, which leave @@ -870,6 +882,12 @@ impl Schedule { keyframe: bool, mut frames: Vec<(u128, usize)>, ) -> anyhow::Result<()> { + let mut gap = std::mem::take(&mut self.gap); + if gap { + // The resumed unit keeps its send-ahead, rather than taking the slot after the + // old unit as though they ran continuously across the missing interval. + self.last = None; + } if let Some(tables) = self.lead.take() { let pmt = self.pmt.context("program tables without a PMT PID")?; let clock = self.psi.as_ref().context("program tables never set")?.clock; @@ -890,11 +908,14 @@ impl Schedule { }; // The unit they lead keeps its own start. let last = self.last; - self.enqueue(pmt, lead, Vec::new(), Some(program)); + // The tables release first, so a long outage's clock jump lands on them. + self.enqueue(pmt, lead, Vec::new(), Some(program)).gap = std::mem::take(&mut gap); self.last = last; self.retable(); } if packets.is_empty() { + // The loss waits for the next unit that goes out. + self.gap |= gap; return Ok(()); } if frames.len() < 2 { @@ -943,6 +964,7 @@ impl Schedule { self.horizon = self.horizon.max(Some(last.div_ceil(PCR_INTERVAL.as_nanos()))); let unit = self.enqueue(pid, timing, packets, None); unit.keyframe = keyframe; + unit.gap = gap; unit.place = place; unit.frames = frames; Ok(()) @@ -965,6 +987,7 @@ impl Schedule { packets, sent: 0, keyframe: false, + gap: false, late: self.next.is_some_and(|next| due < next), place: None, held: 0, @@ -1038,6 +1061,7 @@ impl Schedule { self.clocked = 0; self.unclocked.clear(); self.deferred = None; + self.gap = false; } /// The first slot `unit` may go out in: [`ahead`] of its due slot, and never before the @@ -1122,9 +1146,22 @@ impl Schedule { self.deferred = Some(unit.seq); self.last = last; } + // A long outage jumps to the start of its backfill. Flag that first PCR, not a + // still-queued slot on the old clock, nor the media slot after the backfill. + let jumped = self.next.is_some_and(|next| index > next); + let through = if jumped { index + BACKFILL } else { index }; + let mut discontinuity = false; + for i in 0..self.units.len() { + if self.units[i].gap && self.release(&self.units[i]) <= through { + self.units[i].gap = false; + discontinuity = true; + } + } self.next = Some(index + 1); if let Some(rate) = self.rate { - return self.padded(index, rate).map(Some); + let mut out = self.padded(index, rate)?; + out.discontinuity = discontinuity; + return Ok(Some(out)); } let clock = self.clock; @@ -1185,6 +1222,7 @@ impl Schedule { self.units.retain(|unit| unit.remaining() > 0); Ok(Some(Slot { index, + discontinuity, pcr: grid_pcr(index) as u64, packets, nulls: 0, @@ -1209,6 +1247,7 @@ impl Schedule { }; let mut out = Slot { index, + discontinuity: false, pcr: rate_pcr(index, rate) as u64, packets: Vec::new(), nulls: count - 1, @@ -1827,6 +1866,64 @@ mod tests { assert_eq!(tables, 2); } + /// A long loss ahead of a placed unit flags the PCR its clock jumps to, though the tables + /// leading the unit release slots before it. + #[test] + fn a_skip_flags_the_jump_to_the_tables_leading_a_placed_unit() { + for rate in [None, Some(RATE)] { + let mut schedule = Schedule::new(Duration::ZERO); + schedule.set_rate(rate); + schedule.set_tables(100); + set_psi(&mut schedule); + schedule.set_clock(1); + let tables = || Tables { + rate, + added: Vec::new(), + }; + let mut slots = Vec::new(); + for (place, skip) in [(ms(1_000), false), (ms(10_000), true)] { + if skip { + schedule.skip(); + } + schedule.lead(tables()); + schedule + .push_placed(7, (place, place), 0, unit(7, 1), true, Vec::new()) + .unwrap(); + while let Some(slot) = schedule.next(None).unwrap() { + slots.push((slot.index, slot.discontinuity)); + } + } + let flagged: Vec = slots + .iter() + .filter(|(_, flag)| *flag) + .map(|&(index, _)| index) + .collect(); + let jumps: Vec = slots + .windows(2) + .filter(|pair| pair[1].0 > pair[0].0 + 1) + .map(|pair| pair[1].0) + .collect(); + assert_eq!( + flagged, + jumps.last().copied().into_iter().collect::>(), + "rate {rate:?}" + ); + } + } + + /// An empty unit carries no loss, so the next unit that goes out does. + #[test] + fn a_skip_waits_for_a_unit_with_packets() { + let mut schedule = Schedule::new(Duration::ZERO); + schedule.skip(); + schedule.push(1, timing(100), Vec::new(), true).unwrap(); + schedule.push(1, timing(120), unit(1, 1), true).unwrap(); + let flagged = std::iter::from_fn(|| schedule.next(None).unwrap()) + .filter(|slot| slot.discontinuity) + .count(); + assert_eq!(flagged, 1); + } + /// A timeline that opens holds a placed unit until a PCR interval has followed its first /// tables, while the unit they lead goes out where it would have without the opening, padded /// or not. @@ -3208,6 +3305,7 @@ mod tests { packets.extend(unit(131, 1)); let media = packets.len() / TsPacket::SIZE; let slot = Slot { + discontinuity: false, index: 0, pcr: 0, packets, @@ -3249,6 +3347,7 @@ mod tests { packets.extend(unit(1, 1)); packets.extend(unit(3, 4)); let slot = Slot { + discontinuity: false, index: 0, pcr: 0, packets, diff --git a/test/ts/README.md b/test/ts/README.md index 93c2a7f7ffa..eef0467e317 100644 --- a/test/ts/README.md +++ b/test/ts/README.md @@ -751,6 +751,18 @@ paths run beside it, and `just rs ts-recovery` runs two minutes of the same faul nightly, failing if the latency drifts and keeping a failing arm's export and ledgers under `$HARNESS_RUNS/ts-recovery/`. +Semantic loss also has paused-clock wire checks in `export_test.rs` and +`recovery_test.rs`: + +| Case | Contract | Test | +|---|---|---| +| Skipped groups, short or longer than PCR backfill | Only the next PCR flags the skip; PAT/PMT versions stay unchanged; importer Health reports no unflagged PCR jump | `a_skipped_group_flags_the_clock` | +| Shared A/V outage, including different decode reserves | One program-level PCR flag for the overlapping losses | `a_short_outage_across_both_tracks_flags_once` | +| Same-instance route handoff after a gap | The same PCR-only flag, with no program change | `a_follower_continues_through_a_same_instance_handoff_after_a_gap` | + +These run in `just rs test -p moq-mux --lib` and the selected Rust CI gate. +The raw lane retains the unchanged-packet loss contract below. + ### Lossy path (`--loss`) `--loss ` drops that share of the relay's datagrams at random.