diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 7dd336193af..e94a8e798b5 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -47,7 +47,8 @@ A TS import or export adds sections of its own, which cumulative and omitted at zero, and a gauge is omitted until it is measured. `mpegts` holds `streams`, one row per elementary stream keyed by its PID in -decimal, and the stream-wide counters. A row has `track` (the track suffix), +decimal (plus a `.si` row for an SI PID that dropped sections), and the +stream-wide counters. A row has `track` (the track suffix), `class` (`audio`, `video`, or `data`; an unknown class reads as `data`), `units` (access units delivered), and `quiet`, a gauge: milliseconds of program clock since its last access unit. Its losses are `resyncs`, bytes diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index 0952ea7bad8..827648225b0 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -98,7 +98,6 @@ catalog linger (#5305). ## Required -- [State bound](/quest/m1/ts-reliability/state-bound.md) - SI and revision state retire with what the source stops repeating - [PAT changes](/quest/m1/ts-reliability/pat-lifecycle.md) - a revised PAT never leaves a broadcast meaning another program - [Stream routes](/quest/m1/ts-reliability/stream-routes.md) - every PID gets a declared route, and what is not carried is counted diff --git a/quest/m1/ts-reliability/state-bound.md b/quest/m1/ts-reliability/state-bound.md deleted file mode 100644 index fd0d907938a..00000000000 --- a/quest/m1/ts-reliability/state-bound.md +++ /dev/null @@ -1,54 +0,0 @@ -# [M] TS import holds only what its source keeps repeating - -## Goal - -A TS import's retained state is bounded by what its source repeats, not by -how long it runs or by a constant that refuses. SI arriving without program -clock progress is bounded too, and a stream revision costs O(1) state once -its route has ended, so a feed revising its PMT indefinitely runs -indefinitely. - -## Plan - -SI already retires when not repeated within its table's maximum interval on -the accepted program clock (`si::Entry::expire`, 30 s for unknown tables), so -an input stall retires nothing. Two owners remain unbounded. - -Decided 2026-10-22 by the maintainer: - -- Replace `Import::MAX_REVISIONS` by making an ended route cost O(1): mint - fresh names from a counter so reserving them is a high-water mark, and - drop the retired route's entries. Keep the guard until every owner is - bounded. -- Rejected: one byte budget shared by SI, PES and staged media, wall-clock - expiry, or any constant that only refuses the source. - -Open, for the maintainer: - -- **SI without clock progress.** Standalone SI is accepted before any PCR and - while PCR stalls, and its sections are not charged to the PES staging - owner, so a source sending fresh EIT identities with a frozen clock grows - `Entry::pending` forever. Choose an admission or ownership contract for it, - or narrow the Goal to sources with clock progress. -- **Retired route names and archives.** `BroadcastState::unique` - (`rs/moq-net/src/model/broadcast.rs`) already is a high-water counter and its track lookup prunes dead entries, but - `BroadcastState::sequences` keeps every written name's sequence and - timestamp bounds to enforce explicit same-name continuation. MKV, fMP4, - C/FFI and GStreamer callers keep `unique_name` strings for a later - create/reserve, so a missing name below the counter may be a delayed first - creation rather than a retired one. `moq-mux`'s `timeline::Registry::tracks` likewise - keeps finished recorders' outputs, numbering and `archive.timelines` - discovery after `Recorder` drops. Decide when each may retire while held - readers and same-name re-enrollment stay valid; inferring ownership from a - name prefix is not enough. - -Already bounded, so not per-revision growth: catalog rendition reservations -(released on drop), `retired_stats`, damage and liveness (keyed by PID), and -`si::Capture::entries` (four SI PIDs by 256 table IDs, kept for track -reappearance). - -Tests: a fixture revising its PMT past 65,536 times keeps running with flat -sequence, timeline, catalog and route state, with name continuation and -delayed creation still covered; the chosen no-clock SI contract. - -Public API: none expected. Wire: none. diff --git a/rs/moq-cli/tests/integration/report.rs b/rs/moq-cli/tests/integration/report.rs index 9f7c3f93123..3d0927acd2f 100644 --- a/rs/moq-cli/tests/integration/report.rs +++ b/rs/moq-cli/tests/integration/report.rs @@ -196,10 +196,14 @@ fn suppress(clip: &[u8], pid: u16, from: f64, to: f64) -> Vec { } /// Each PID's access units in `first` fell short of `last`'s: the rows advance while it runs. +/// An SI row counts only the sections the program clock could not place, so it has none. fn assert_advancing(first: &ts::stats::Snapshot, last: &ts::stats::Snapshot) { assert!(!last.streams.is_empty(), "no elementary stream reported"); let first = units(first); for (pid, units) in units(last) { + if last.streams[&pid].track == ".si" { + continue; + } let before = first.get(&pid).copied().unwrap_or(0); assert!(before < units, "PID {pid}: {before} access units, then {units}"); } diff --git a/rs/moq-mux/src/catalog/producer.rs b/rs/moq-mux/src/catalog/producer.rs index 6aad1714cfc..2cbab9b54b7 100644 --- a/rs/moq-mux/src/catalog/producer.rs +++ b/rs/moq-mux/src/catalog/producer.rs @@ -956,6 +956,21 @@ impl Producer { Ok(recorder) } + /// Retire `track`'s timeline once its producer dropped, for an owner that never enrolls the + /// name again: the timeline track finishes and leaves the root `archive` entry. + pub(crate) fn retire(&mut self, track: &str) -> crate::Result<()> { + self.timeline.retire(track)?; + let advertised = take(&self.current) + .catalog + .archive + .as_ref() + .is_some_and(|archive| archive.timelines.contains_key(track)); + if advertised && let Some(archive) = self.modify()?.archive.as_mut() { + archive.timelines.remove(track); + } + Ok(()) + } + /// The broadcast's [`Timelines`](crate::timeline::Timelines): each enrolled track's index. /// /// A track's timeline is created (and advertised in the catalog's root `archive` entry) when @@ -1938,6 +1953,35 @@ mod test { assert!(catalog.enroll("video0", None).is_err()); } + /// A retired track's timeline ends for the readers already on it and leaves the archive, while + /// one still recording is refused. + #[test] + fn retiring_a_track_ends_its_timeline() { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let mut catalog = Producer::new(&mut broadcast, Config::default()).unwrap(); + let recorder = catalog + .enroll("video0", Some(std::time::Duration::from_secs(4))) + .unwrap(); + let waiter = kio::Waiter::noop(); + let mut subscribing = broadcast + .consume() + .track("video0.timeline.z") + .unwrap() + .subscribe(None) + .into_inner(); + let std::task::Poll::Ready(Ok(mut reader)) = kio::Task::poll(&mut subscribing, &waiter) else { + panic!("the timeline track is live"); + }; + assert!(catalog.retire("video0").is_err(), "its recorder is still alive"); + + drop(recorder); + catalog.retire("video0").unwrap(); + let archive = catalog.snapshot().archive.unwrap(); + assert_eq!(archive.timelines.keys().collect::>(), vec!["catalog.json"]); + assert_eq!(catalog.timeline().len(), 1); + assert!(matches!(reader.poll_finished(&waiter), std::task::Poll::Ready(Ok(_)))); + } + #[test] fn enrolling_advertises_the_catalog_section() { let mut broadcast = moq_net::broadcast::Info::new().produce(); diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 9d95fe3e0cc..e85cd7097a3 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -3237,14 +3237,18 @@ async fn service_layer_survives_roundtrip() { let data = include_bytes!("test_data/bbb.ts"); let nit = make_long_section(0x40, 0x1234, 1, 0, 0, &[0xff, 0x01]); - // Prepend a synthetic NIT Actual packet (0x0010); prepend keeps bbb's alignment. - // Twice, because real SI repeats every few seconds: the repetition must collapse - // into the same single committed section rather than cutting a second group. The - // repeat varies the continuity counter so it reaches the store (an identical - // packet would be dropped as a TS duplicate before reassembly). - let mut input = si_packet(0x0010, &nit); + // Append a synthetic NIT Actual packet (0x0010), past bbb's PCRs: SI before the + // program clock can place it is dropped, which is where bbb's own SDT sits, so it is + // repeated there too. The NIT goes twice, because real SI repeats every few seconds: + // the repetition must collapse into the same single committed section rather than + // cutting a second group. Each repeat advances the continuity counter so it reaches + // the store (an identical packet would be dropped as a TS duplicate before reassembly). + let mut input = data.to_vec(); + input.extend_from_slice(&si_packet(0x0010, &nit)); input.extend_from_slice(&si_packet_cc(0x0010, &nit, 1)); - input.extend_from_slice(&data[..]); + let mut sdt = data[..188].to_vec(); + sdt[3] = (sdt[3] & 0xf0) | ((sdt[3] + 1) & 0x0f); + input.extend_from_slice(&sdt); let mut broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); @@ -3330,9 +3334,16 @@ async fn service_layer_survives_roundtrip() { } assert!(checked_pat, "missing PAT"); - // Re-import: the SDT and NIT must come back byte-for-byte. + // The SDT and NIT go out byte-for-byte. + assert_eq!(ts_sections(&ts, 0x0011), vec![sdt], "the SDT survived byte-for-byte"); + assert_eq!( + ts_sections(&ts, 0x0010), + vec![Bytes::from(nit)], + "the NIT survived byte-for-byte" + ); + + // Re-import: the identity comes back. let mut broadcast2 = moq_net::broadcast::Info::new().produce(); - let consumer2 = broadcast2.consume(); let catalog2 = crate::catalog::Producer::new( &mut broadcast2, crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::::default()), @@ -3358,17 +3369,6 @@ async fn service_layer_survives_roundtrip() { "program number survived" ); assert_eq!(program2.pmt_pid, program.pmt_pid, "PMT PID survived"); - assert_eq!(snapshot2.ext.mpegts.si, si, "every SI entry survived the round-trip"); - assert_eq!( - read_si_sections(&consumer2, &snapshot2.ext.mpegts.si[&0x0011][&0x42].track).await, - vec![sdt], - "the SDT survived byte-for-byte" - ); - assert_eq!( - read_si_sections(&consumer2, &snapshot2.ext.mpegts.si[&0x0010][&0x40].track).await, - vec![Bytes::from(nit)], - "the NIT survived byte-for-byte" - ); } /// Each SI PID must be re-emitted on its own interval, independently of the PSI cadence @@ -4134,24 +4134,17 @@ async fn multi_packet_si_section_is_captured() { let input = si_packets_multi(0x0011, &sdt); assert!(input.len() > 188, "the SDT must span more than one TS packet"); - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let consumer = broadcast.consume(); - let catalog = crate::catalog::Producer::new( - &mut broadcast, - crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::::default()), - ) - .unwrap(); - let mut import = crate::container::ts::Import::new(broadcast, catalog.reserve()); - import.decode(&BytesMut::from(&input[..])).unwrap(); - import.finish().unwrap(); + let mut rig = si_rig(); + rig.import.decode(&BytesMut::from(&input[..])).unwrap(); + rig.import.finish().unwrap(); - let si = catalog.snapshot().ext.mpegts.si.clone(); + let si = rig.catalog.snapshot().ext.mpegts.si.clone(); let entry = si .get(&0x0011) .and_then(|tables| tables.get(&0x42)) .expect("an SDT entry"); assert_eq!( - read_si_sections(&consumer, &entry.track).await, + read_si_sections(&rig.consumer, &entry.track).await, vec![Bytes::from(sdt)], "the multi-packet SDT was reassembled and captured byte-for-byte" ); @@ -4166,6 +4159,13 @@ struct SiRig { } fn si_rig() -> SiRig { + let mut rig = si_import(); + rig.import.decode(&BytesMut::from(&si_clock(1)[..])).unwrap(); + rig +} + +/// A rig with no input decoded yet. +fn si_import() -> SiRig { let mut broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); let catalog = crate::catalog::Producer::new( @@ -4181,6 +4181,16 @@ fn si_rig() -> SiRig { } } +/// A program clock for SI-only input to be placed by: a PAT, a PMT for `program` with no +/// streams, and one PCR interval. +fn si_clock(program: u16) -> Vec { + use super::fixture::{Flags, MS, Mux}; + let mut mux = Mux::default(); + mux.pat(&[(program, 0x20)]).pmt(0x20, program, 0x1ff, &[], 0); + mux.packet(0x1ff, Flags::pcr(0)).packet(0x1ff, Flags::pcr(40 * MS)); + mux.out +} + /// #2881: a multi-section table revised one section at a time must never publish a /// torn set. The store buffers the incoming generation and commits it atomically, /// so every snapshot group holds a single version. @@ -4271,9 +4281,11 @@ fn si_rig_selecting(program: u16) -> SiRig { import, catalog, consumer, - } = si_rig(); + } = si_import(); + let mut import = import.with_program(program); + import.decode(&BytesMut::from(&si_clock(program)[..])).unwrap(); SiRig { - import: import.with_program(program), + import, catalog, consumer, } @@ -4432,10 +4444,11 @@ async fn an_sdt_revision_dropping_the_service_retires_it() { .unwrap(); let mut capture = crate::container::ts::si::Capture::new(broadcast, catalog.clone()); capture.select(1); + // One PCR interval, so the program clock places SI. + capture.pcr(0, 0); + capture.pcr(40 * 27_000, 10); let mut revise = |version: u8, services: &[(u16, &[u8])]| { - capture - .section(0x0011, sdt_actual(version, 0, 0, services).into()) - .unwrap(); + assert!(capture.section(0x0011, sdt_actual(version, 0, 0, services).into(), 10)); capture.flush(Timestamp::ZERO, true).unwrap(); catalog .snapshot() @@ -4477,8 +4490,11 @@ async fn a_corrupt_sdt_keeps_the_last_good_snapshot() { .unwrap(); let mut capture = crate::container::ts::si::Capture::new(broadcast, catalog.clone()); capture.select(1); + // One PCR interval, so the program clock places SI. + capture.pcr(0, 0); + capture.pcr(40 * 27_000, 10); let mut feed = |section: Vec| { - capture.section(0x0011, section.into()).unwrap(); + assert!(capture.section(0x0011, section.into(), 10)); capture.flush(Timestamp::ZERO, true).unwrap(); catalog .snapshot() @@ -4518,8 +4534,11 @@ async fn a_partial_sdt_revision_keeps_the_last_good_snapshot() { .unwrap(); let mut capture = crate::container::ts::si::Capture::new(broadcast, catalog.clone()); capture.select(1); + // One PCR interval, so the program clock places SI. + capture.pcr(0, 0); + capture.pcr(40 * 27_000, 10); let mut feed = |section: Vec| { - capture.section(0x0011, section.into()).unwrap(); + assert!(capture.section(0x0011, section.into(), 10)); capture.flush(Timestamp::ZERO, true).unwrap(); catalog .snapshot() @@ -4564,12 +4583,15 @@ async fn a_selection_filters_eit_schedule() { .unwrap(); let mut capture = crate::container::ts::si::Capture::new(broadcast, catalog.clone()); capture.select(1); + // One PCR interval, so the program clock places SI. + capture.pcr(0, 0); + capture.pcr(40 * 27_000, 10); let schedule = |service: u16| { let body = [0x00, 0x01, 0x00, 0x02, 0x00, 0x50, service as u8]; make_long_section(0x50, service, 0, 0, 0, &body) }; for service in [1, 2] { - capture.section(0x0012, schedule(service).into()).unwrap(); + assert!(capture.section(0x0012, schedule(service).into(), 10)); } capture.finish(Timestamp::ZERO).unwrap(); @@ -4604,20 +4626,31 @@ async fn a_selected_programs_si_survives_export() { let mut reader = TsPacketReader::new(Cursor::new(ts.as_ref())); while reader.read_ts_packet().unwrap().is_some() {} - let mut again = si_rig(); - again.import.decode(&BytesMut::from(ts.as_ref())).unwrap(); - again.import.finish().unwrap(); - let si = again.catalog.snapshot().ext.mpegts.si.clone(); - let sdt = read_si_sections(&again.consumer, &si[&0x0011][&0x42].track).await; + let sdt = ts_sections(&ts, 0x0011); assert_eq!(sdt.len(), 1, "one SDT actual section"); assert_sdt_lists_only(&sdt[0], 0, 2, b"Two"); assert_eq!( - read_si_sections(&again.consumer, &si[&0x0012][&0x4E].track).await, + ts_sections(&ts, 0x0012), vec![eit_pf(2, 0), eit_pf(2, 1)], "only program 2's EIT" ); } +/// The sections `ts` carries on `pid`, each whole in one packet. Read off the wire rather +/// than by a re-import: SI an export leads with precedes its PCRs, so an import drops it +/// until it repeats. +fn ts_sections(ts: &[u8], pid: u16) -> Vec { + ts.chunks(188) + .filter(|p| (u16::from(p[1] & 0x1f) << 8 | u16::from(p[2])) == pid && p[1] & 0x40 != 0) + .map(|p| { + let pointer = if p[3] & 0x20 != 0 { 5 + usize::from(p[4]) } else { 4 }; + let start = pointer + 1 + usize::from(p[pointer]); + let len = 3 + (usize::from(p[start + 1] & 0x0f) << 8 | usize::from(p[start + 2])); + Bytes::copy_from_slice(&p[start..start + len]) + }) + .collect() +} + /// #2842: SDT other sections from two networks that reuse a transport_stream_id /// must not collide. The identity reads original_network_id (bytes 8..10) for /// table_id 0x46, so both survive as separate sub-tables; a revision within one @@ -6596,7 +6629,8 @@ async fn si_rapid_revisions_are_rate_bounded() { #[tokio::test(start_paused = true)] async fn overlapping_capture_teardown_keeps_the_survivors_mapping() { let sdt = make_long_section(0x42, 1, 0, 0, 0, &[0xaa; 4]); - let mut input = si_packet(0x0011, &sdt); + let mut input = si_clock(1); + input.extend_from_slice(&si_packet(0x0011, &sdt)); input.extend_from_slice(&si_packet_cc(0x0011, &sdt, 1)); let mut broadcast = moq_net::broadcast::Info::new().produce(); diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 1a6a85dce64..493e3ebeebe 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -139,8 +139,6 @@ pub struct Import { streams: HashMap>, /// Counters from routes a later PMT replaced, keyed by PID. retired_stats: BTreeMap, - /// Stream revisions applied so far; see [`Self::MAX_REVISIONS`]. - revisions: usize, /// Damaged packets, PES, or access units refused, cumulative across PID remaps. damaged: BTreeMap, damage_log: super::damage::Log, @@ -255,7 +253,6 @@ impl Import { crc_errors: BTreeMap::new(), streams: HashMap::new(), retired_stats: BTreeMap::new(), - revisions: 0, damaged: BTreeMap::new(), damage_log: Default::default(), liveness: Liveness::default(), @@ -447,16 +444,14 @@ impl Import { // The new timebase starts with the PCR the flag rides, if it rides one. if let Some(pcr) = pcr(pkt) { self.place_pcr(pcr, offset)?; + self.si.pcr(pcr, packet.count); } } else if let Some(pcr) = pcr(pkt) { self.place_pcr(pcr, offset)?; self.liveness.pcr(pcr); + self.si.pcr(pcr, packet.count); if let Some(now) = self.liveness.now() { self.health.tick(now); - self.si.advance( - std::time::Duration::from_secs(now / PCR_HZ) - + std::time::Duration::from_nanos((now % PCR_HZ) * 1_000 / 27), - ); } if self.supports_mpegts { let ready = self.mux_rate.ready(); @@ -488,7 +483,9 @@ impl Import { // `table_id`. if self.supports_mpegts && catalog::SI_PIDS.contains(&pid) { for section in packet.sections { - self.si.section(pid, section.clone())?; + if !self.si.section(pid, section.clone(), packet.count) { + *self.unplaced.entry(pid).or_default() += 1; + } } return Ok(()); } @@ -607,6 +604,7 @@ impl Import { // A new clock: the intervals straddling the switch measure nothing, and the whole PES // the old one began are placed on its timebase, never between its PCRs and the new. self.liveness.discontinuity(); + self.si.discontinuity(); self.reset_mux_rate()?; self.settle_placement()?; } @@ -623,7 +621,6 @@ impl Import { // Record the program-level descriptors as the PMT revises them; export re-emits them // verbatim, including the original CUEI. if self.supports_mpegts && self.program_info != pmt.program_info { - self.count_revision()?; if let Some(mpegts) = self.catalog.modify()?.ext.mpegts_mut() { mpegts.program_descriptors = pmt.program_info.clone(); } @@ -792,6 +789,9 @@ impl Import { section.finish()?; self.pending.remove(&pid); self.headers.remove(&pid); + let name = section.entry.name.clone(); + drop(section); + self.retire_track(Some(name))?; } // A private PES stream follows the PMT's PCR_PID: refused while there is none to place // its PES by, and routed again once there is, or once the PID carries another type. @@ -1028,23 +1028,12 @@ impl Import { } } - /// Stream revisions one import may apply: routes ended or refused, and program descriptor - /// changes. A route that reopens takes a fresh track name, which the broadcast reserves for - /// its whole life, and each revision republishes the catalog to every viewer, so a source that - /// keeps revising its PMT would otherwise grow the broadcast and churn its viewers without - /// bound. Under 10 MB of reserved names; a feed revising its PMT 50 times a day would take - /// years to reach it. - const MAX_REVISIONS: usize = 65_536; - - /// Count a revision, refusing the import once its source has churned through - /// [`Self::MAX_REVISIONS`]. - fn count_revision(&mut self) -> anyhow::Result<()> { - self.revisions += 1; - anyhow::ensure!( - self.revisions <= Self::MAX_REVISIONS, - "the TS source revised its streams {} times, more than an import accepts", - Self::MAX_REVISIONS - ); + /// Retire the timeline of a track whose route has ended. Every route mints its own name, so + /// nothing enrolls it again, and a source revising its PMT holds only its live routes. + fn retire_track(&mut self, name: Option) -> anyhow::Result<()> { + if let Some(name) = name { + self.catalog.retire(&name)?; + } Ok(()) } @@ -1053,7 +1042,6 @@ impl Import { /// PCRs so far or counted unplaced (refusing the import, as end of input does, if no media /// timestamp anchored it), and its track finishes and leaves the catalog. fn end_route(&mut self, pid: Pid) -> anyhow::Result<()> { - self.count_revision()?; if self.headers.remove(&pid).is_some() { self.damage(pid, &anyhow::anyhow!("PES header truncated by a PMT revision"))?; } @@ -1073,8 +1061,11 @@ impl Import { ); section.finish()?; } - if self.sections.remove(&pid.as_u16()).is_some() { + if let Some(section) = self.sections.remove(&pid.as_u16()) { self.keep_stats(pid, Some(stats::Stream::new(".ts", stats::Class::Data))); + let name = section.entry.name.clone(); + drop(section); + self.retire_track(Some(name))?; } self.unclocked.remove(&pid); if let Some(stream) = self.streams.get_mut(&pid) @@ -1102,6 +1093,9 @@ impl Import { if let Some(stream) = self.streams.remove(&pid) { // Finishing can publish a held frame, which its counters then include. self.keep_stats(pid, stream.stats()); + let name = stream.track_name(); + drop(stream); + self.retire_track(name)?; } if unplaced > 0 { *self.unplaced.entry(pid.as_u16()).or_default() += unplaced; @@ -1431,15 +1425,17 @@ impl Import { /// Refuse the undecoded stream on `pid` rather than carry it altered: its track ends with an /// error and leaves the catalog, and the PID is dropped from here on. fn refuse(&mut self, pid: Pid, why: &str) -> anyhow::Result<()> { - // A PMT that keeps losing and regaining its PCR reopens the stream each time. - self.count_revision()?; self.pending.remove(&pid); self.headers.remove(&pid); if let Some(section) = self.sections.remove(&pid.as_u16()) { + let name = section.entry.name.clone(); section.abort(moq_net::Error::Unsupported); + self.retire_track(Some(name))?; } if let Some(stream) = self.retire_stream(pid) { + let name = stream.track_name(); stream.abort(moq_net::Error::Unsupported); + self.retire_track(name)?; } self.streams.insert(pid, Stream::Ignored); *self.refused.entry(pid.as_u16()).or_default() += 1; @@ -1516,6 +1512,7 @@ impl Import { self.held_edge = None; self.published = false; self.liveness.discontinuity(); + self.si.discontinuity(); if self.mux_rate.discontinuity() { self.record_mux_rate()?; } @@ -1754,11 +1751,13 @@ impl Import { .or_insert_with(|| stats::Stream::new("", stats::Class::Data)) .damaged = damaged; } - // A stream refused before it had a track has no row of its own either. + // A stream refused before it had a track has no row of its own either, nor does an + // SI PID. for (&pid, &unplaced) in &self.unplaced { + let track = if catalog::SI_PIDS.contains(&pid) { ".si" } else { ".ts" }; streams .entry(pid) - .or_insert_with(|| stats::Stream::new(".ts", stats::Class::Data)) + .or_insert_with(|| stats::Stream::new(track, stats::Class::Data)) .unplaced = unplaced; } for (&pid, &refused) in &self.refused { @@ -2032,8 +2031,8 @@ impl Drop for VerbatimEntry { /// track and catalog entry and places each section on the PCR clock. struct SectionStream { track: crate::container::Producer, - /// Held for its `Drop`, which clears this track's catalog entry. - _entry: VerbatimEntry, + /// Clears this track's catalog entry on drop; its name retires the timeline when the route ends. + entry: VerbatimEntry, reassembler: SectionReassembler, /// Sections waiting for the next PCR or the timebase's offset, in input order. held: VecDeque, @@ -2061,7 +2060,7 @@ impl SectionStream { }; Ok(Self { track, - _entry: entry, + entry, reassembler: SectionReassembler::default(), held: VecDeque::new(), staging: Arc::default(), @@ -2211,8 +2210,8 @@ fn adjust_splice(section: &mut [u8], offset: Offset) { /// its stream. struct PesPacketStream { track: crate::container::Producer, - /// Held for its `Drop`, which clears this track's catalog entry. - _entry: VerbatimEntry, + /// Clears this track's catalog entry on drop; its name retires the timeline when the route ends. + entry: VerbatimEntry, /// The PMT `stream_type`, which says whether an unbounded PES is video. stream_type: u8, /// Whole PES waiting for the PCR after them or the timebase's offset, in input order. @@ -2259,7 +2258,7 @@ impl PesPacketStream { }; Ok(Self { track, - _entry: entry, + entry, stream_type, held: VecDeque::new(), }) @@ -3162,6 +3161,14 @@ impl Stream { }) } + /// The MoQ track name this route publishes, decoded or verbatim, once its track exists. + fn track_name(&self) -> Option { + match self { + Stream::PesPacket(stream) => Some(stream.entry.name.clone()), + stream => stream.media_track_name(), + } + } + /// The MoQ track name of a decoded media stream, once its (lazily created) track /// exists. `None` for verbatim/clock/ignored streams (verbatim self-registers). fn media_track_name(&self) -> Option { @@ -10730,40 +10737,14 @@ pub(super) mod test { assert!(err.to_string().contains("twice"), "{err:#}"); } - /// Every ended route leaves its track name reserved, so a source that keeps revising its PMT - /// is refused once it has applied as many revisions as an import accepts. - #[test] - fn pmt_churn_past_the_revision_bound_refuses_the_import() { - use crate::container::ts::psi::{Es, Pmt}; - let (_catalog, mut import) = psi_import(); - let pmt = |streams: &[u8]| Pmt { - program_number: 1, - pcr_pid: Some(Pid::new(PCR_PID).unwrap()), - program_info: Vec::new(), - streams: streams - .iter() - .map(|&stream_type| Es { - stream_type, - pid: Pid::new(0x0062).unwrap(), - descriptors: Vec::new(), - }) - .collect(), - }; - import.handle_pmt(pmt(&[0x06])).unwrap(); - import.revisions = super::Import::<()>::MAX_REVISIONS - 1; - - import - .handle_pmt(pmt(&[0x15])) - .expect("the last revision within the bound"); - let err = import.handle_pmt(pmt(&[])).expect_err("one revision past the bound"); - assert!(err.to_string().contains("revised"), "{err:#}"); - } - - /// A PMT that keeps dropping and restoring its PCR reopens the stream each time, so its - /// refusals count toward the same bound. + /// A source revising its PMT without end holds only its live routes: each revision ends a + /// decoded or verbatim route, refuses one for a lost PCR, or rewrites the program descriptors, + /// and none leaves its track's timeline or catalog entry behind. #[test] - fn pcr_churn_past_the_revision_bound_refuses_the_import() { + fn pmt_churn_keeps_flat_state() { use crate::container::ts::psi::{Es, Pmt}; + // Each round of three revisions ends where it started. + const ROUNDS: u32 = 333; let mut broadcast = moq_net::broadcast::Info::new().produce(); let catalog = crate::catalog::Producer::new( &mut broadcast, @@ -10771,58 +10752,45 @@ pub(super) mod test { ) .unwrap(); let mut import = super::Import::new(broadcast, catalog.reserve()); - let pmt = |pcr_pid: Option| Pmt { - program_number: 1, - pcr_pid: pcr_pid.map(|pid| Pid::new(pid).unwrap()), - program_info: Vec::new(), - streams: vec![Es { - stream_type: 0x06, - pid: Pid::new(0x0062).unwrap(), - descriptors: Vec::new(), - }], + let pmt = |revision: u32| { + // A decoded H.264 route, a private PES route, then the same one refused for want of a PCR. + let (stream_type, pcr) = match revision % 3 { + 0 => (StreamType::H264 as u8, true), + 1 => (0x06, true), + _ => (0x06, false), + }; + Pmt { + program_number: 1, + pcr_pid: pcr.then(|| Pid::new(PCR_PID).unwrap()), + program_info: vec![super::catalog::Descriptor { + tag: 0x80, + data: bytes::Bytes::copy_from_slice(&revision.to_be_bytes()), + }], + streams: vec![Es { + stream_type, + pid: Pid::new(0x0062).unwrap(), + descriptors: Vec::new(), + }], + } }; - import.handle_pmt(pmt(Some(PCR_PID))).unwrap(); - import.revisions = super::Import::::MAX_REVISIONS; - - let err = import.handle_pmt(pmt(None)).expect_err("a refusal past the bound"); - assert!(err.to_string().contains("revised"), "{err:#}"); - } - - /// Each program descriptor change republishes the catalog without ending a route, so it - /// counts toward the same bound. - #[test] - fn descriptor_churn_past_the_revision_bound_refuses_the_import() { - use crate::container::ts::psi::{Es, Pmt}; - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let catalog = crate::catalog::Producer::new( - &mut broadcast, - crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::::default()), - ) - .unwrap(); - let mut import = super::Import::new(broadcast, catalog.reserve()); - let pmt = |data: &'static [u8]| Pmt { - program_number: 1, - pcr_pid: Some(Pid::new(PCR_PID).unwrap()), - program_info: vec![super::catalog::Descriptor { - tag: 0x80, - data: bytes::Bytes::from_static(data), - }], - streams: vec![Es { - stream_type: StreamType::Mpeg2Video as u8, - pid: Pid::new(PCR_PID).unwrap(), - descriptors: Vec::new(), - }], + let state = |import: &super::Import| { + let snapshot = catalog.snapshot(); + ( + catalog.timeline().len(), + snapshot.archive.map_or(0, |archive| archive.timelines.len()), + snapshot.ext.mpegts.tracks.len(), + import.streams.len() + import.sections.len() + import.listed.len() + import.retired_stats.len(), + ) }; - import.handle_pmt(pmt(&[0xaa])).unwrap(); - import.revisions = super::Import::::MAX_REVISIONS - 1; - import - .handle_pmt(pmt(&[0xbb])) - .expect("the last revision within the bound"); - let err = import - .handle_pmt(pmt(&[0xaa])) - .expect_err("one revision past the bound"); - assert!(err.to_string().contains("revised"), "{err:#}"); + for revision in 0..6 { + import.handle_pmt(pmt(revision)).unwrap(); + } + let settled = state(&import); + for revision in 6..3 * ROUNDS { + import.handle_pmt(pmt(revision)).unwrap(); + } + assert_eq!(state(&import), settled); } /// The first PMT records its program descriptors over whatever the catalog held, even when it diff --git a/rs/moq-mux/src/container/ts/import_test.rs b/rs/moq-mux/src/container/ts/import_test.rs index 6499c891c09..301adc2bad3 100644 --- a/rs/moq-mux/src/container/ts/import_test.rs +++ b/rs/moq-mux/src/container/ts/import_test.rs @@ -960,7 +960,8 @@ fn import_reports_a_silent_scte35_pid() { } /// Every elementary stream the importer carries reports a row, decoded, clock-only, and -/// verbatim alike, and every row has delivered. +/// verbatim alike, and every row has delivered. An SI PID has a row only for the sections +/// it sent before the program clock could place them, as ffmpeg leads with its SDT. #[test] fn import_reports_every_elementary_stream() { for (data, rows) in [ @@ -970,12 +971,15 @@ fn import_reports_every_elementary_stream() { ), ( &include_bytes!("test_data/scte35/bbb5s.ts")[..], - &[(0x21, ".ts"), (0x100, ".hev1"), (0x101, ".opus")][..], + &[(0x11, ".si"), (0x21, ".ts"), (0x100, ".hev1"), (0x101, ".opus")][..], ), ] { - let stats = sample(data, &[]).pop().unwrap(); + let mut stats = sample(data, &[]).pop().unwrap(); let tracks: Vec<(u16, &str)> = stats.streams.iter().map(|(&pid, s)| (pid, s.track.as_str())).collect(); assert_eq!(tracks, rows); + if let Some(sdt) = stats.streams.remove(&0x11) { + assert_eq!((sdt.units, sdt.unplaced), (0, 1), "{sdt:?}"); + } for (pid, stream) in &stats.streams { assert!(stream.units > 0, "{pid:#x} delivered nothing: {stream:?}"); assert!(stream.quiet.is_some(), "{pid:#x} has no program clock: {stream:?}"); diff --git a/rs/moq-mux/src/container/ts/si.rs b/rs/moq-mux/src/container/ts/si.rs index ac60dc02f96..09bd8790304 100644 --- a/rs/moq-mux/src/container/ts/si.rs +++ b/rs/moq-mux/src/container/ts/si.rs @@ -22,7 +22,7 @@ //! service's entry, with a fresh CRC. An SDT actual section whose own CRC fails is //! dropped, and a revision commits only once every section has arrived, so the last //! good snapshot stays in force until then or until the source stops repeating it -//! for its table's maximum interval on the accepted program clock. An SDT that does not list the +//! for its table's maximum interval on the program clock ([`Clock`]). An SDT that does not list the //! service carries no SDT actual, so a revision that drops it retires the earlier //! one. Network-wide tables (NIT, BAT, SDT other, EIT other, TDT/TOT) pass through //! verbatim, as does everything when no program is selected. @@ -36,6 +36,8 @@ use bytes::Bytes; use moq_net::Timestamp; use super::catalog; +use super::mux_rate::{MAX_INTERVAL, PCR_WRAP}; +use super::passthrough::PCR_HZ; use super::psi::{self, CRC}; const SDT_PID: u16 = 0x0011; @@ -161,7 +163,7 @@ fn sdt_service<'a>(sections: impl IntoIterator, service: u16) /// unused section numbers, so 0..=last is not a contiguous range there. #[derive(Default, PartialEq)] struct Generation { - /// The accepted program-clock deadline for this generation's next repetition. + /// The program-clock deadline for this generation's next repetition. expires: Duration, version: Option, sections: BTreeMap, @@ -438,17 +440,91 @@ impl Entry { } } +/// The program clock SI ages on, admitted as whole PES are placed: SI before the first +/// PCR, or past it with no interval to extrapolate from, is dropped, so a source cannot +/// hold SI by withholding its clock. +/// +/// A valid PCR interval (forward, at most [`MAX_INTERVAL`]) advances it by its own ticks and +/// joins the measured multiplex rate. Any other, a stalled or rewound PCR, a corrupt jump, or +/// one across a clock break, advances it by the input packets it spans at the measured rate. +/// So does the interval still open past the last PCR, once it outruns any valid one: until +/// then the next PCR may yet close it, and a burst above the average rate must not retire SI +/// early. A valid interval that closes a longer open one keeps what the open one charged, so +/// PCR silence ages SI whatever step resumes it. +/// The rate pools every valid interval rather than reading the catalog's `mux_rate`, which +/// a variable-rate source leaves unset. +#[derive(Default)] +struct Clock { + /// Ticks elapsed through the last PCR. + elapsed: u64, + /// The last PCR (`None` past a clock break) and the input packet count at it. + last: Option<(Option, u64)>, + /// Ticks and packets over every valid interval. + measured: (u64, u64), +} + +impl Clock { + /// A PCR on the program's clock PID, carried by input packet `count`. + fn pcr(&mut self, pcr: u64, count: u64) { + if let Some((last, at)) = self.last { + let packets = count - at; + let extrapolated = self.extrapolate(packets).unwrap_or(0); + let step = last + .map(|last| (pcr + PCR_WRAP - last) % PCR_WRAP) + .filter(|step| (1..=MAX_INTERVAL).contains(step)); + let ticks = match step { + // What [`Self::at`] already charged the open interval stays charged, or a + // small step after a long silence would freeze SI for as long as it lasted. + Some(step) => { + let open = extrapolated.saturating_sub(MAX_INTERVAL); + self.measured.0 += step; + self.measured.1 += packets; + step.max(open) + } + None => extrapolated, + }; + self.elapsed = self.elapsed.saturating_add(ticks); + } + self.last = Some((Some(pcr), count)); + } + + /// The clock broke or changed PID: the interval across it is extrapolated. + fn discontinuity(&mut self) { + if let Some((pcr, _)) = &mut self.last { + *pcr = None; + } + } + + /// Ticks `packets` span at the measured rate, or `None` before a valid interval. + fn extrapolate(&self, packets: u64) -> Option { + let (ticks, total) = self.measured; + (total > 0) + .then(|| u64::try_from(u128::from(packets) * u128::from(ticks) / u128::from(total)).unwrap_or(u64::MAX)) + } + + /// The clock at input packet `count`, or `None` when it cannot place SI there. + fn at(&self, count: u64) -> Option { + let (_, at) = self.last?; + let open = self.extrapolate(count - at)?.saturating_sub(MAX_INTERVAL); + let ticks = self.elapsed.saturating_add(open); + Some(Duration::from_secs(ticks / PCR_HZ) + Duration::from_nanos((ticks % PCR_HZ) * 1_000 / 27)) + } +} + /// Import-side SI capture: reassembled sections in, snapshot tracks and catalog /// entries out. /// /// Owned by the TS importer, which feeds it every completed section on a captured -/// SI PID and calls [`flush`](Self::flush) once per decode batch. Dropping it -/// removes the advertised entries from the catalog, mirroring the verbatim tracks. +/// SI PID and every PCR, and calls [`flush`](Self::flush) once per decode batch. +/// Dropping it removes the advertised entries from the catalog, mirroring the verbatim +/// tracks. pub(super) struct Capture { broadcast: moq_net::broadcast::Producer, catalog: crate::catalog::Producer, entries: BTreeMap<(u16, u8), Entry>, - /// Accepted PCR time elapsed, independent of host time and media PTS. + clock: Clock, + /// The latest [`Clock`] reading, independent of host time and media PTS. Monotonic: + /// an open interval's extrapolation may run ahead of the PCR that closes it. now: Duration, /// The selected program_number, which is the service_id of the only service /// carried; `None` captures every service. @@ -461,14 +537,29 @@ impl Capture { broadcast, catalog, entries: BTreeMap::new(), + clock: Clock::default(), now: Duration::ZERO, program: None, } } - /// Advance from the importer's accepted, accumulated PCR intervals. - pub fn advance(&mut self, now: Duration) { + /// A PCR on the program's clock PID, carried by input packet `count`. + pub fn pcr(&mut self, pcr: u64, count: u64) { + self.clock.pcr(pcr, count); + self.advance(count); + } + + /// The program clock broke or changed PID. + pub fn discontinuity(&mut self) { + self.clock.discontinuity(); + } + + fn advance(&mut self, count: u64) -> bool { + let Some(now) = self.clock.at(count) else { + return false; + }; self.now = self.now.max(now); + true } /// Carry only the service numbered `program`: its EIT actual, and an SDT actual @@ -478,18 +569,21 @@ impl Capture { self.program = Some(program); } - /// Fold one completed section from `pid` into its `(pid, table_id)` entry, - /// creating the entry on first sight of the pair. - pub fn section(&mut self, pid: u16, section: Bytes) -> anyhow::Result<()> { + /// Fold one completed section from `pid`, completed by input packet `count`, into its + /// `(pid, table_id)` entry, creating the entry on first sight of the pair. + /// + /// Returns `false` when the program clock cannot place it ([`Clock`]): it is dropped, + /// for the caller to count unplaced. + pub fn section(&mut self, pid: u16, section: Bytes, count: u64) -> bool { let Some(&table_id) = section.first() else { - return Ok(()); + return true; }; // A next-version section (current_next_indicator clear) describes a future // state, not the current one; carrying it would need a second generation slot // per sub-table for a table nothing re-emits, so it is dropped until the // version becomes current. if long_form(§ion) && section[5] & 0x01 == 0 { - return Ok(()); + return true; } // EIT actual (now/next and schedule) is keyed by service_id: another // service's sub-table describes a program this import does not carry. @@ -498,14 +592,17 @@ impl Capture { && matches!(table_id, 0x4E | 0x50..=0x5F) && !matches!(identity(§ion), Identity::Sub { ext, .. } if ext == program) { - return Ok(()); + return true; } let service = self.program.filter(|_| (pid, table_id) == (SDT_PID, SDT_ACTUAL)); // The rebuilt SDT gets a fresh CRC, which would certify corruption the // source's CRC flags; dropping the section keeps the last good generation. if service.is_some() && !psi::crc_ok(§ion) { - return Ok(()); + return true; + } + if !self.advance(count) { + return false; } let entry = self.entries.entry((pid, table_id)).or_insert_with(|| Entry { track: None, @@ -520,7 +617,7 @@ impl Capture { last_cut: None, }); entry.section(section, self.now); - Ok(()) + true } /// Cut snapshot groups for the dirty entries, then advertise each entry whose @@ -678,7 +775,7 @@ mod tests { use crate::container::ts::fixture::{Flags, Mux}; #[moq_clock::test] - async fn program_clock_retires_si_without_media_pts() { + async fn program_clock_places_and_retires_si_without_media_pts() { let mut broadcast = moq_net::broadcast::Info::new().produce(); let catalog = crate::catalog::Producer::new( &mut broadcast, @@ -688,12 +785,23 @@ mod tests { let mut import = super::super::Import::new(broadcast, catalog.reserve()); let mut mux = Mux::default(); mux.pat(&[(1, 0x20)]).pmt(0x20, 1, 0x1ff, &[], 0); + let sdt = psi::section(0x42, 1, 0, 0, 0, &[0, 1, 0xff]); + // Before the first PCR, and past it with no interval to extrapolate from, the program + // clock cannot age SI, so it is dropped as an unplaced whole PES is. + mux.section(0x11, &sdt); mux.packet(0x1ff, Flags::pcr(0)); - mux.section(0x11, &psi::section(0x42, 1, 0, 0, 0, &[0, 1, 0xff])); + mux.section(0x11, &sdt); + import.decode(&std::mem::take(&mut mux.out)).unwrap(); + assert!(catalog.snapshot().ext.mpegts.si.is_empty()); + let row = &import.stats().streams[&0x11]; + assert_eq!((row.track.as_str(), row.unplaced), (".si", 2)); + mux.packet(0x1ff, Flags::pcr(40 * 27_000)); + mux.section(0x11, &sdt); import.decode(&std::mem::take(&mut mux.out)).unwrap(); assert!(catalog.snapshot().ext.mpegts.si.contains_key(&0x11)); + assert_eq!(import.stats().streams[&0x11].unplaced, 2); // Accepted PCR intervals move the program clock; no video PTS or host time passes. - for n in 1..=51 { + for n in 2..=52 { mux.packet(0x1ff, Flags::pcr(n * 40 * 27_000)); import.decode(&std::mem::take(&mut mux.out)).unwrap(); } @@ -713,24 +821,63 @@ mod tests { psi::section(table, id, version, 0, last, &[0, 0, 0, 0, 0, 0]).into() } + const MS: u64 = 27_000; + + /// Drives a capture's program clock from test time zero: a PCR every 40 ms, ten input + /// packets apart, each section completed by the packet carrying the latest PCR. + struct Input { + /// Ticks past the first PCR, which every interval so far accepted. + pcr: u64, + count: u64, + } + + impl Input { + /// One valid interval measures the rate, so SI is placed from test time zero. + fn new(capture: &mut Capture) -> Self { + capture.pcr(0, 0); + let mut input = Self { pcr: 0, count: 0 }; + input.step(capture, 40 * MS); + input + } + + fn step(&mut self, capture: &mut Capture, ticks: u64) { + self.pcr += ticks; + self.count += 10; + capture.pcr(self.pcr % PCR_WRAP, self.count); + } + + /// Advance the program clock to `t` past test time zero, rounding up to a whole tick. + fn to(&mut self, capture: &mut Capture, t: Duration) { + let target = 40 * MS + u64::try_from((t.as_nanos() * 27).div_ceil(1_000)).unwrap(); + while self.pcr < target { + self.step(capture, (target - self.pcr).min(40 * MS)); + } + } + + fn section(&self, capture: &mut Capture, pid: u16, section: Bytes) { + assert!(capture.section(pid, section, self.count), "placed"); + } + } + #[moq_clock::test] async fn incomplete_generations_expire_on_their_table_intervals() { let (mut capture, _) = capture(); + let mut input = Input::new(&mut capture); for id in 0..4096 { for table in [0x4e, 0x80] { - capture.section(0x12, section(table, id, 0, 1)).unwrap(); + input.section(&mut capture, 0x12, section(table, id, 0, 1)); } } let payload = capture.entries[&(0x12, 0x4e)].pending.values().next().unwrap().sections[&0].clone(); assert!(!payload.is_unique()); - capture.advance(Duration::from_secs(2)); + input.to(&mut capture, Duration::from_secs(2)); capture.flush(Timestamp::ZERO, false).unwrap(); assert_eq!( capture.entries[&(0x12, 0x4e)].pending.len(), 4096, "the deadline is inclusive" ); - capture.advance(Duration::from_secs(2) + Duration::from_nanos(1)); + input.to(&mut capture, Duration::from_secs(2) + Duration::from_nanos(1)); capture.flush(Timestamp::ZERO, false).unwrap(); let eit = &capture.entries[&(0x12, 0x4e)]; assert!(eit.pending.is_empty()); @@ -740,7 +887,7 @@ mod tests { "expiry releases the generation's payload ownership" ); assert_eq!(capture.entries[&(0x12, 0x80)].pending.len(), 4096); - capture.advance(Duration::from_secs(30) + Duration::from_nanos(1)); + input.to(&mut capture, Duration::from_secs(30) + Duration::from_nanos(1)); capture.flush(Timestamp::ZERO, false).unwrap(); assert!( capture @@ -753,22 +900,23 @@ mod tests { #[moq_clock::test] async fn pending_renewal_replaces_its_deadline_without_renewing_other_versions() { let (mut capture, _) = capture(); + let mut input = Input::new(&mut capture); capture.select(1); for version in [0, 1] { - capture.section(0x11, section(0x42, 1, version, 2)).unwrap(); + input.section(&mut capture, 0x11, section(0x42, 1, version, 2)); } - capture.advance(Duration::from_millis(1500)); + input.to(&mut capture, Duration::from_millis(1500)); for _ in 0..1000 { - capture.section(0x11, section(0x42, 1, 0, 2)).unwrap(); + input.section(&mut capture, 0x11, section(0x42, 1, 0, 2)); } assert_eq!(capture.entries[&(0x11, 0x42)].expiry.len(), 2); - capture.advance(Duration::from_millis(2100)); + input.to(&mut capture, Duration::from_millis(2100)); capture.flush(Timestamp::ZERO, false).unwrap(); let entry = &capture.entries[&(0x11, 0x42)]; assert_eq!(entry.pending.len(), 1); assert!(entry.pending.contains_key(&(Identity::Sub { ext: 1, scope: 0 }, 0))); assert_eq!(entry.expiry.len(), 1); - capture.advance(Duration::from_millis(3501)); + input.to(&mut capture, Duration::from_millis(3501)); capture.flush(Timestamp::ZERO, false).unwrap(); assert!(capture.entries[&(0x11, 0x42)].pending.is_empty()); } @@ -776,8 +924,9 @@ mod tests { #[moq_clock::test] async fn retirement_publishes_reduced_and_empty_snapshots_without_host_debounce() { let (mut capture, catalog) = capture(); + let mut input = Input::new(&mut capture); for id in [1, 2] { - capture.section(0x10, section(0x40, id, 0, 0)).unwrap(); + input.section(&mut capture, 0x10, section(0x40, id, 0, 0)); } capture.flush(Timestamp::ZERO, false).unwrap(); let name = catalog.snapshot().ext.mpegts.si[&0x10][&0x40].track.clone(); @@ -790,19 +939,19 @@ mod tests { .await .unwrap(); let mut original = reader.recv_group().await.unwrap().unwrap(); - capture.advance(Duration::from_secs(9)); + input.to(&mut capture, Duration::from_secs(9)); for _ in 0..1000 { - capture.section(0x10, section(0x40, 1, 0, 0)).unwrap(); + input.section(&mut capture, 0x10, section(0x40, 1, 0, 0)); } assert_eq!(capture.entries[&(0x10, 0x40)].expiry.len(), 2); - capture.advance(Duration::from_millis(10_001)); + input.to(&mut capture, Duration::from_millis(10_001)); capture.flush(Timestamp::ZERO, false).unwrap(); let mut reduced = reader.recv_group().await.unwrap().unwrap(); let frame = reduced.read_frame().await.unwrap().unwrap(); assert_eq!(identity(&frame.payload), Identity::Sub { ext: 1, scope: 0 }); assert!(reduced.read_frame().await.unwrap().is_none()); assert_eq!(catalog.snapshot().ext.mpegts.si[&0x10][&0x40].track, name); - capture.advance(Duration::from_millis(19_001)); + input.to(&mut capture, Duration::from_millis(19_001)); capture.flush(Timestamp::ZERO, false).unwrap(); let mut empty = reader.recv_group().await.unwrap().unwrap(); assert!(empty.read_frame().await.unwrap().is_none()); @@ -811,7 +960,7 @@ mod tests { assert!(capture.entries[&(0x10, 0x40)].expiry.is_empty()); // Retraction cannot invalidate a reader already holding a prior snapshot. assert!(original.read_frame().await.unwrap().is_some()); - capture.section(0x10, section(0x40, 1, 0, 0)).unwrap(); + input.section(&mut capture, 0x10, section(0x40, 1, 0, 0)); capture.flush(Timestamp::ZERO, true).unwrap(); let mut resumed = reader.recv_group().await.unwrap().unwrap(); assert!(resumed.read_frame().await.unwrap().is_some()); @@ -819,13 +968,14 @@ mod tests { } #[moq_clock::test] - async fn host_time_and_new_si_do_not_advance_a_stalled_program_clock() { + async fn host_time_and_new_si_at_one_input_position_do_not_age_si() { let (mut capture, _) = capture(); - capture.section(0x12, section(0x4e, 0, 0, 0)).unwrap(); + let input = Input::new(&mut capture); + input.section(&mut capture, 0x12, section(0x4e, 0, 0, 0)); capture.flush(Timestamp::ZERO, false).unwrap(); moq_clock::sleep(Duration::from_secs(60)).await; for id in 1..1024 { - capture.section(0x12, section(0x4e, id, 0, 1)).unwrap(); + input.section(&mut capture, 0x12, section(0x4e, id, 0, 1)); } capture.flush(Timestamp::ZERO, false).unwrap(); let entry = &capture.entries[&(0x12, 0x4e)]; @@ -834,55 +984,97 @@ mod tests { assert_eq!(entry.expiry.len(), 1024); } - #[moq_clock::test] - async fn pcr_wrap_reset_and_invalid_jumps_preserve_repetition_age() { - let (mut capture, _) = capture(); - capture.section(0x11, section(0x42, 1, 0, 0)).unwrap(); - let mut clock = super::super::import::Liveness::default(); - let wrap = super::super::mux_rate::PCR_WRAP; - let ms = 27_000; - clock.pcr(wrap - 40 * ms); - clock.pcr(0); - assert_eq!(clock.now(), Some(40 * ms)); + #[test] + fn the_clock_counts_valid_intervals_and_extrapolates_the_rest() { + let mut clock = Clock::default(); + let at = |clock: &Clock, count| clock.at(count).map(|t| t.as_millis()); + clock.pcr(PCR_WRAP - 40 * MS, 0); + assert_eq!(at(&clock, 0), None, "no interval measures the rate"); + // Across the wrap: 40 ms over 10 packets. + clock.pcr(0, 10); + assert_eq!(at(&clock, 10), Some(40)); + // A stall, a rewind, a corrupt jump and a flagged break each charge the input they + // span at the measured rate, not their PCR values. + clock.pcr(0, 20); + clock.pcr(PCR_WRAP - 1_000 * MS, 30); + clock.pcr(100_000 * MS, 40); clock.discontinuity(); - clock.pcr(500 * ms); - clock.pcr(540 * ms); - assert_eq!(clock.now(), Some(80 * ms)); - // A corrupt forward jump and a rewind charge no made-up silence. - clock.pcr(100_000 * ms); - clock.pcr(0); - assert_eq!(clock.now(), Some(80 * ms)); - for n in 1..=48 { - clock.pcr(n * 40 * ms); - capture.advance(Duration::from_nanos(clock.now().unwrap() * 1_000 / 27)); - capture.flush(Timestamp::ZERO, false).unwrap(); + clock.pcr(40 * MS, 50); + assert_eq!(at(&clock, 50), Some(200)); + // The interval still open is charged only once it outruns any valid interval. + assert_eq!(at(&clock, 50 + 250), Some(200)); + assert_eq!(at(&clock, 50 + 260), Some(240)); + clock.pcr(80 * MS, 60); + assert_eq!(at(&clock, 60), Some(240)); + // Ten seconds of PCR silence, then a valid step: the silence stays charged and the + // clock runs on from there rather than freezing until the PCR catches up. + assert_eq!(at(&clock, 60 + 2_500), Some(9_240)); + clock.pcr(120 * MS, 2_560); + assert_eq!(at(&clock, 2_560), Some(9_240)); + clock.pcr(160 * MS, 2_570); + assert_eq!(at(&clock, 2_570), Some(9_280)); + } + + #[test] + fn the_clock_follows_a_rate_increase() { + let mut clock = Clock::default(); + let at = |clock: &Clock, count| clock.at(count).map(|t| t.as_millis()); + for n in 0..=10 { + clock.pcr(n * 40 * MS, n * 10); } - assert_eq!( - capture.entries[&(0x11, 0x42)].active.len(), - 1, - "exactly two accepted seconds" - ); - clock.pcr(49 * 40 * ms); - capture.advance(Duration::from_nanos(clock.now().unwrap() * 1_000 / 27)); - capture.flush(Timestamp::ZERO, false).unwrap(); - assert!(capture.entries[&(0x11, 0x42)].active.is_empty()); + let mut count = 100; + // 26 times the packets per interval: each valid step still counts as its own 40 ms, + // rather than the second its packets span at the old rate. + for n in 11..=100 { + count += 260; + clock.pcr(n * 40 * MS, count); + } + assert_eq!(at(&clock, count), Some(4_000)); } + + #[moq_clock::test] + async fn fresh_si_ages_out_while_the_pcr_stalls() { + for silent in [false, true] { + let (mut capture, _) = capture(); + let mut input = Input::new(&mut capture); + let stalled = input.pcr; + let mut peak = 0; + for id in 0..2000 { + input.count += 10; + if !silent { + capture.pcr(stalled, input.count); + } + input.section(&mut capture, 0x12, section(0x4e, id, 0, 1)); + capture.flush(Timestamp::ZERO, false).unwrap(); + let entry = &capture.entries[&(0x12, 0x4e)]; + assert_eq!(entry.expiry.len(), entry.pending.len()); + peak = peak.max(entry.pending.len()); + } + // Each identity arrives 40 ms of input after the last and lives the table's 2 s. A + // silent clock charges nothing for the open interval's first second, so the 25 + // identities in it outlive their 2 s once. + assert_eq!(peak, if silent { 75 } else { 51 }, "silent: {silent}"); + assert_eq!(capture.entries[&(0x12, 0x4e)].pending.len(), 51, "silent: {silent}"); + } + } + #[moq_clock::test] async fn short_form_repetitions_replace_their_deadline_and_latest_value() { let (mut capture, _) = capture(); + let mut input = Input::new(&mut capture); let mut time = vec![0x70, 0x70, 5, 0, 0, 0, 0, 0]; - capture.section(0x14, Bytes::from(time.clone())).unwrap(); - capture.advance(Duration::from_secs(29)); - capture.section(0x14, Bytes::from(time.clone())).unwrap(); - capture.advance(Duration::from_secs(31)); + input.section(&mut capture, 0x14, Bytes::from(time.clone())); + input.to(&mut capture, Duration::from_secs(29)); + input.section(&mut capture, 0x14, Bytes::from(time.clone())); + input.to(&mut capture, Duration::from_secs(31)); time[7] = 1; - capture.section(0x14, Bytes::from(time.clone())).unwrap(); - capture.advance(Duration::from_secs(60)); + input.section(&mut capture, 0x14, Bytes::from(time.clone())); + input.to(&mut capture, Duration::from_secs(60)); capture.flush(Timestamp::ZERO, false).unwrap(); let entry = &capture.entries[&(0x14, 0x70)]; assert_eq!(entry.expiry.len(), 1); assert_eq!(entry.active[&Identity::Table].sections[&0], Bytes::from(time)); - capture.advance(Duration::from_millis(61_001)); + input.to(&mut capture, Duration::from_millis(61_001)); capture.flush(Timestamp::ZERO, false).unwrap(); assert!(capture.entries[&(0x14, 0x70)].active.is_empty()); assert!(capture.entries[&(0x14, 0x70)].expiry.is_empty()); diff --git a/rs/moq-mux/src/container/ts/stats.rs b/rs/moq-mux/src/container/ts/stats.rs index f86c790ee02..82e575e284e 100644 --- a/rs/moq-mux/src/container/ts/stats.rs +++ b/rs/moq-mux/src/container/ts/stats.rs @@ -52,6 +52,7 @@ pub struct Report { pub struct Snapshot { /// One row per elementary stream PID the importer carries, ordered by PID. A PID the /// importer drops (an undecoded stream without the `mpegts` catalog section) has none. + /// An SI PID has a `.si` row only once it counts [`Stream::unplaced`] sections. // Keys read from strings: a flattened section loses JSON's integer-key parsing. #[serde_as(as = "BTreeMap")] #[serde(skip_serializing_if = "BTreeMap::is_empty")] @@ -160,7 +161,7 @@ pub struct Stream { /// Damaged packets, PES, or access units refused whole on this PID. Export stays zero. #[serde(skip_serializing_if = "is_zero")] pub damaged: u64, - /// Whole PES or sections dropped because the program clock could not place + /// Whole PES or sections, SI included, dropped because the program clock could not place /// them: before the first PCR, after the last with no interval to extrapolate from, or /// before the broadcast clock's zero. Any PES lands before that zero while its stream is /// still behind a wrap the import's first PTS already crossed. Export stays zero. diff --git a/rs/moq-mux/src/timeline.rs b/rs/moq-mux/src/timeline.rs index df03b0bdac4..423aceb09b2 100644 --- a/rs/moq-mux/src/timeline.rs +++ b/rs/moq-mux/src/timeline.rs @@ -554,7 +554,8 @@ type OnDeclare = Box; struct Registry { /// Each enrolled track's timeline, by track name. Kept after its recorder drops, so a finished - /// span stays discoverable and a re-enrolled track continues its numbering. + /// span stays discoverable and a re-enrolled track continues its numbering, until + /// [`Timelines::retire`] says the name is done. tracks: BTreeMap>)>, finished: bool, } @@ -644,6 +645,34 @@ impl Timelines { section } + /// Finish the timeline of `name`, whose recorder dropped and whose owner never enrolls the + /// name again, and forget it, so a producer minting a name per track holds only its live ones. + /// + /// Readers already on the timeline track read it to its end. Nothing happens for a name never + /// enrolled; a name whose recorder is still alive errors. + pub(crate) fn retire(&self, name: &str) -> crate::Result<()> { + let mut registry = self.registry.lock().unwrap(); + let Some((_, live)) = registry.tracks.get(name) else { + return Ok(()); + }; + let mut live = live.lock().unwrap(); + if live.recording { + return Err(moq_net::Error::Duplicate.into()); + } + // The catalog stops advertising the timeline, so a last declaration must not add it back. + live.on_declare = None; + live.finish(); + drop(live); + registry.tracks.remove(name); + Ok(()) + } + + /// How many timelines are held, live or kept for a re-enrollment. + #[cfg(test)] + pub(crate) fn len(&self) -> usize { + self.registry.lock().unwrap().tracks.len() + } + /// Close every open record now, publishing any declaration it fixes. pub(crate) fn flush(&self) { let registry = self.registry.lock().unwrap();