Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions doc/bin/ts.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,11 +151,14 @@ track, so an `.msf` broadcast needs `--catalog-format hang`. The encoding is in

An importer joining a live feed waits for what it needs before publishing.

- **Semantic:** the import waits for the PAT, PMT, and a PCR, and holds the
catalog until two seconds of PCR have measured a constant mux rate or shown
the input to be variable-rate. A subscriber joins at a keyframe, and the
export puts a PAT and PMT ahead of it. From there a late exporter writes the
same packets as a running one, but for continuity counters.
- **Semantic:** the import waits for the PAT, PMT, and a PCR, and publishes the
catalog with the first media. A constant mux rate follows as a catalog update
once two seconds of PCR have measured it; a variable-rate source never records
one. An export without `--mux-rate` waits up to those two seconds for the rate
before it starts, so a variable-rate source costs it one wait. A subscriber
joins at a keyframe, and the export puts a PAT and PMT ahead of it. From
there a late exporter writes the same packets as a running one, but for
continuity counters.
- **Passthrough:** nothing is published until the PAT, the clock program's
PMT, a PCR, and a group start. Bytes before that, and bytes off the 188-byte
grid, are dropped: the output equals the input over the *admitted interval*,
Expand Down
31 changes: 0 additions & 31 deletions quest/m1/ts-mux-rate-hold.md

This file was deleted.

6 changes: 3 additions & 3 deletions rs/moq-cli/src/publish.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2279,13 +2279,13 @@ pub(crate) mod tests {

/// A source whose clock steps back republishes the path under a new instance, announced once
/// its catalog goes out, with the sections the first instance's catalog carried. The new
/// catalog reuses the first instance's mux rate, so it goes out with the first media rather
/// than after a fresh two-second measurement window.
/// catalog reuses the first instance's mux rate, so an export of it needn't wait on a fresh
/// measurement window.
#[tokio::test(start_paused = true)]
async fn a_clock_step_republishes_under_a_new_instance() {
// 5 s of H.264 and AAC at a constant 400 kb/s, its clock on PID 0x100.
const CLIP: &[u8] = include_bytes!("../../moq-mux/src/container/ts/test_data/bbb5s_cbr.ts");
// Its first 1.5 s: short of a measurement window, so only a reused rate releases a catalog.
// Its first 1.5 s: short of a measurement window, so only a reused rate is in the catalog.
const HEAD: &[u8] = CLIP.split_at(400 * 188).0;
let mut flag = vec![0x47, 0x01, 0x00, 0x20, 183, 0x80];
flag.resize(188, 0xff);
Expand Down
6 changes: 2 additions & 4 deletions rs/moq-cli/tests/integration/import.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ async fn import_delivers_the_catalog_finish_at_eof() {

let mut child = import(fixture.url.as_str(), &fixture.fingerprint);
let mut stdin = child.stdin.take().expect("stdin");
// A live TS catalog waits for its initial two-second PCR rate measurement.
stdin.write_all(BBB).await.expect("write stdin");

let origin = moq_tokio::origin::spawn();
Expand Down Expand Up @@ -128,14 +127,13 @@ async fn import_delivers_the_catalog_finish_at_eof() {
}

/// About one second of CBR HEVC and Opus: shorter than the two-second mux-rate window, so
/// the catalog is only released at EOF.
/// the import ends before it records a rate.
const SHORT: &[u8] = include_bytes!("../../../moq-mux/src/container/ts/test_data/bbb_cbr.ts")
.split_at(256 * 188)
.0;

/// A subscriber already waiting when a short import ends still receives its media: the
/// import announces at the finish, once its catalog lists the tracks, and keeps them servable
/// for the linger after EOF.
/// import keeps its tracks servable for the linger after EOF.
#[tokio::test]
async fn a_short_import_drains_its_media_to_a_waiting_subscriber() {
moq_tokio::crypto::install_default().unwrap();
Expand Down
1 change: 1 addition & 0 deletions rs/moq-mux/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- A TS import's catalog goes out with the first media instead of waiting two seconds to measure `mpegts.muxRate`, which arrives as a catalog update once a PCR window measures it. An export with no `with_mux_rate` waits up to one window for it before it subscribes the tracks, so a padded export still starts padded; a variable-rate source costs it that one wait.
- [**breaking**] Declare per-track archive durations from GOP hints or the first completed record.
- [**breaking**] Update the re-exported `mp4_atom` to 0.17, whose `Trun.entries` is now `Trun.samples`: a run without per-sample fields decodes as a count, so a moof of field-less runs no longer materializes an entry per sample before import refuses it. Read entries with `trun.samples.iter()` and build them with `samples: entries.into()`. `TrunEntry::cts` widens to `i64` (a version 0 offset is unsigned), `EsDescriptor` gains its optional fields, and `Dref.urls` becomes typed `entries`. `Dops` gains `channel_mapping: Option<OpusChannelMapping>` (`None` for family 0).

Expand Down
68 changes: 67 additions & 1 deletion rs/moq-mux/src/container/ts/export.rs
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,14 @@ where
/// none and the output is unpadded ([`Schedule`]).
mux_rate: Option<u64>,
mux_rate_override: Option<u64>,
/// The newest catalog snapshot while the export waits for the rate a TS import has yet to
/// measure: the tracks are subscribed once it ends, so the export joins the media as it is
/// then, not on frames that went stale during the wait.
held_catalog: Option<Catalog<S::Ext>>,
/// Until when that wait lasts, set by the first snapshot that records no rate.
rate_until: Option<kio::clock::Instant>,
rate_wait: Duration,
rate_timer: Option<std::pin::Pin<Box<kio::clock::Sleep>>>,
/// Tune-in point: the first video keyframe's timestamp, captured when the program
/// tables are built. Non-video frames before it are dropped so the keyframe leads
/// the stream.
Expand Down Expand Up @@ -1639,6 +1647,10 @@ where
trimmed: 0,
mux_rate: None,
mux_rate_override: None,
held_catalog: None,
rate_until: None,
rate_wait: super::mux_rate::WAIT,
rate_timer: None,
}
}
}
Expand Down Expand Up @@ -1713,6 +1725,13 @@ where
self
}

/// Start at once on a catalog that records no rate, for a test whose clock a wait would move.
#[cfg(test)]
pub(super) fn without_rate_wait(mut self) -> Self {
self.program.rate_wait = Duration::ZERO;
self
}

/// Read each track from the oldest group the delay still reaches instead of the live
/// edge, and anchor the clock on the first frame instead of acquiring it, for a test that
/// writes a broadcast whole before exporting it.
Expand Down Expand Up @@ -2132,7 +2151,7 @@ where
match catalog.poll_next(waiter)? {
Poll::Ready(Some(snapshot)) => {
self.program.cataloged = true;
self.update_catalog(snapshot)?
self.program.held_catalog = Some(snapshot);
}
Poll::Ready(None) => {
self.program.feed = Feed::Ended;
Expand All @@ -2141,6 +2160,13 @@ where
Poll::Pending => break,
}
}
if let Some(mut snapshot) = self.program.held_catalog.take() {
if self.awaiting_rate(&mut snapshot, waiter) {
self.program.held_catalog = Some(snapshot);
} else {
self.update_catalog(snapshot)?;
}
}

// 1b. Drive the SI subscriptions. Ordinary SI updates its active set; clock
// revisions wait for their stamped media position in `mux`.
Expand Down Expand Up @@ -2232,6 +2258,45 @@ where
Ok(true)
}

/// Whether the first catalog waits for the rate a TS import is still measuring, which only a
/// padded output reads. The wait ends with the rate or at its deadline, whichever comes first.
fn awaiting_rate(&mut self, catalog: &mut Catalog<S::Ext>, waiter: &kio::Waiter) -> bool {
let program = &mut self.program;
// A catalog that ended gains no rate.
if program.mux_rate_override.is_some()
|| program.psi.is_some()
|| !program.tracks.is_empty()
|| !matches!(program.feed, Feed::Live)
{
return false;
}
// A catalog without the section came from no TS import.
let measuring = catalog
.ext
.mpegts_mut()
.is_some_and(|mpegts| mpegts.mux_rate.is_none() && !mpegts.is_empty());
if !measuring {
program.rate_timer = None;
return false;
}
let until = *program
.rate_until
.get_or_insert_with(|| kio::clock::now().saturating_add(program.rate_wait));
if kio::clock::now() >= until {
program.rate_timer = None;
return false;
}
let timer = program
.rate_timer
.get_or_insert_with(|| Box::pin(kio::clock::sleep_until(until)));
// An elapsed deadline is ready on this first poll, so a wait never ends without a waker.
if waiter.poll_future(timer.as_mut()).is_ready() {
program.rate_timer = None;
return false;
}
true
Comment thread
pullfrog[bot] marked this conversation as resolved.
Resolved
}

/// The program playing failed with `err`. Only a publisher that left is survivable: its
/// input stops at once, as a follow that runs out stops it, but what it already buffered
/// goes out on schedule. Without a replacement, the export then reports `err`, so a
Expand Down Expand Up @@ -3003,6 +3068,7 @@ where
self.next_program += 1;
next.replay = self.program.replay;
next.mux_rate_override = self.program.mux_rate_override;
next.rate_wait = self.program.rate_wait;
next
}

Expand Down
119 changes: 117 additions & 2 deletions rs/moq-mux/src/container/ts/export_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,15 @@ pub(super) async fn open_export(
Ok(Export::new(source, catalog, delay))
}

/// Open an export with the MPEG-TS catalog extension and explicit encoding.
/// Open an export with the MPEG-TS catalog extension and explicit encoding, which starts at once
/// whether or not the catalog records a rate.
pub(super) async fn open_ts(
source: crate::Source,
format: crate::catalog::CatalogFormat,
delay: Duration,
) -> crate::Result<Export<crate::catalog::Consumer<tscat::Ext>>> {
let catalog = source.catalog::<tscat::Ext>(format).await?;
Ok(Export::new(source, catalog, delay))
Ok(Export::new(source, catalog, delay).without_rate_wait())
}

/// A millisecond track: the frames written here carry timestamps.
Expand Down Expand Up @@ -7969,6 +7970,120 @@ async fn export_without_a_mux_rate_is_unpadded() {
assert_eq!(Clocked::of(&ts).nulls, 0, "null packets in an unpadded export");
}

/// A TS source arriving in real time, written to an import 100 ms of data a step, and an export
/// that has not been told to start at once. It joins before the import's first PCR window
/// completes: `head` bytes are all the import has read.
struct Joined {
data: &'static [u8],
read: usize,
import: crate::container::ts::Import<tscat::Ext>,
export: Export<crate::catalog::Consumer<tscat::Ext>>,
_broadcast: moq_net::broadcast::Consumer,
started: kio::clock::Instant,
/// What the export wrote, and when its first output came.
out: BytesMut,
first: Option<Duration>,
}

/// How much the source and the clock advance in a step.
const STEP: Duration = Duration::from_millis(100);

impl Joined {
async fn new(data: &'static [u8], head: usize) -> Self {
let mut broadcast = moq_net::broadcast::Info::new().produce();
let consumer = broadcast.consume();
let catalog = crate::catalog::Producer::new(
&mut broadcast,
crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::<tscat::Ext>::default()),
)
.unwrap();
let mut import = crate::container::ts::Import::new(broadcast, catalog.reserve());
import.decode(&data[..head]).unwrap();
let source = crate::source::announced(&consumer).await;
let stream = source
.catalog::<tscat::Ext>(crate::catalog::CatalogFormat::Hang)
.await
.unwrap();
Self {
data,
read: head,
import,
export: Export::new(source, stream, Duration::from_millis(500)),
_broadcast: consumer,
started: kio::clock::now(),
out: BytesMut::new(),
first: None,
}
}

/// Run for `steps`, the source writing a step's worth of its data each.
async fn run(&mut self, steps: usize) {
// The bytes a step of a five-second source carries, in whole packets.
let chunk = self.data.len() / 50 / 188 * 188;
for _ in 0..steps {
let end = (self.read + chunk).min(self.data.len());
self.import.decode(&self.data[self.read..end]).unwrap();
self.read = end;
let until = kio::clock::now().saturating_add(STEP);
while let Ok(frame) = kio::clock::timeout_at(until, self.export.next()).await {
let Some(frame) = frame.expect("exporter error") else {
break;
};
self.first.get_or_insert(self.started.elapsed());
self.out.extend_from_slice(&frame.payload);
}
}
}
}

/// A padded export joining inside the first PCR window waits for the rate the import is still
/// measuring, then pads to it.
#[tokio::test(start_paused = true)]
async fn export_waits_for_the_rate_a_window_is_measuring() {
// The first 0.4 s of a 400 kb/s constant-rate source: its catalog is out, its rate not yet.
let mut joined = Joined::new(include_bytes!("test_data/bbb5s_cbr.ts"), 188 * 100).await;
joined.run(15).await;
assert_eq!(
joined.first, None,
"the export started before the import measured a rate"
);

joined.run(25).await;
joined.first.expect("the rate arrived and the export started");
assert_packet_aligned(&joined.out);
assert!(
Clocked::of(&joined.out).nulls > 0,
"the export joined a window early without padding"
);
}

/// A source that records no rate costs an export one wait, and its output is unpadded.
#[tokio::test(start_paused = true)]
async fn export_stops_waiting_for_a_rate_that_never_comes() {
let mut joined = Joined::new(include_bytes!("test_data/scte35/bbb5s.ts"), 188 * 100).await;
joined.run(40).await;
let first = joined.first.expect("the export gave up on the rate");
assert!(
first >= super::mux_rate::WAIT,
"the export started {first:?} in, before its wait ran out"
);
assert_eq!(
Clocked::of(&joined.out).nulls,
0,
"null packets in an export with no rate"
);
}

/// A rate given on the command line stands in for the one being measured.
#[tokio::test(start_paused = true)]
async fn export_with_a_rate_override_does_not_wait() {
let mut joined = Joined::new(include_bytes!("test_data/bbb5s_cbr.ts"), 188 * 100).await;
joined.export = joined.export.with_mux_rate(2_000_000);
joined.run(30).await;
let first = joined.first.expect("an export with a rate override started");
assert!(first < STEP * 5, "the export started {first:?} in");
}

/// 1 Mb/s is 16.62 packets per 25 ms slot. The fractional remainder carries across slots
/// instead of rounding each one, and each PCR is the time of its own byte at the rate, so
/// every PCR is within a system clock tick of its byte position: far inside the ±500 ns
Expand Down
Loading