diff --git a/doc/bin/cli.md b/doc/bin/cli.md index b42a7222f3b..930c2bba63b 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -452,8 +452,7 @@ continuity error, and the rest carry on. It takes neither `--linger`, `export ts --sync` renders every packet from the broadcast alone, so two exports of it, joined at different moments and fed with different skew, emit identical TS for the content both receive, continuity counters included, as the legs of a -1+1 pair must; an SMPTE ST 2022-7 pair also needs identical RTP framing around -it, which this command does not write. Each PID gets a fixed lane of the constant rate, sized +1+1 pair must. Each PID gets a fixed lane of the constant rate, sized for its codec's most, with video taking the rest; content one leg lacks leaves its packets as stuffing there. The PCR rides a PID of its own, and each PID's continuity counter restarts at every group with the break flagged. It needs a @@ -465,7 +464,21 @@ narrow to absorb that one, is logged and counted in `overruns`: the legs can differ there. So can reordered video whose SPS declares no reorder depth (`max_num_reorder_frames`), which is logged, and SI while one leg holds an older revision than the other or around an SI table the catalog adds or drops, which -each leg takes up when its catalog update arrives. +each leg takes up when its catalog update arrives. With `--rtp IP:PORT` (such +as `--rtp 239.1.1.1:5004`) it sends the multiplex as RTP (RFC 2250) over UDP +instead of to stdout, for an SMPTE ST 2022-7 pair: two legs send the same +datagram, headers included, wherever both hold its packets, and each withholds +one it is not sure of, so legs whose losses fall in different datagrams send +between them every datagram a clean leg sends; a datagram both lost part of +goes out from neither (the framing is +`Export::with_rtp`'s, in [`moq-mux`](https://docs.rs/moq-mux)). SI +one leg missed, and a unit of a sparse track (such as SCTE-35) lost with no +later jump in it, still go out as stuffing where the other leg has them. The +SSRC follows the publisher [epoch](/concept/moq-lite#publisher-epochs), so +every hop must carry it: moq-transport draft 17 or later, or the opt-in +moq-lite 07 (see [Publisher runs](#publisher-runs)). The datagrams go out at +the multiplex rate on the `--delay` clock, so `--rtp` refuses `--delay 0s`. A +multicast destination gets the system's default TTL. A stdout export ends with the broadcast. `export ts --linger 10s` waits that long for the same publisher instance to come back, and carries on with the same diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index c30fabb339e..24a9e692a46 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -109,6 +109,5 @@ remain with their current quests. ## Related -- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - TS identity alone is not ST 2022-7, which also needs matching RTP headers - [MSFTS mapping](/quest/m3/msfts-convergence.md) - document the exact equivalence and differences between the carriage modes diff --git a/quest/m2/README.md b/quest/m2/README.md index c3d434b002d..fbadd51a47e 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -133,7 +133,6 @@ ranked just below deterministic TS legs, which it requires. - [Watch decode gate](/quest/m2/watch-decode-gate.md) - video lookahead stays encoded until it is near presentation, like audio - [Drop announce="never"](/quest/m2/publish-announce-never.md) - `@moq/publish` offers only announce modes a watcher can reach - [Deterministic H.265 TS legs](/quest/m2/ts-hitless-h265.md) - H.265's buffering period gets the same group anchor as H.264 -- [TS over RTP legs](/quest/m2/ts-rtp-egress.md) - two legs send identical RTP packets, headers included, so an ST 2022-7 receiver can merge them - [moqsrc A/V sync](/quest/m2/moqsrc-av-sync.md) - a nightly test measures A/V skew from `moqsrc` through clock-synced sinks - [MP4 export](/quest/m2/mp4-export.md) - `moq export mp4 --output` records crash-safe fragments, then finishes a regular MP4 with moov at the end - [fMP4 edit lists](/quest/m2/fmp4-edit-lists.md) - the fMP4 importer applies edit lists to frame timestamps diff --git a/quest/m2/ts-rtp-egress.md b/quest/m2/ts-rtp-egress.md deleted file mode 100644 index 455974e00fa..00000000000 --- a/quest/m2/ts-rtp-egress.md +++ /dev/null @@ -1,56 +0,0 @@ -# [M] TS over RTP legs that an ST 2022-7 receiver can merge - -## Goal - -`moq export ts` can send its output as RTP (MPEG-2 TS over RTP, RFC 2250), -and two `--sync` legs of one broadcast send the same RTP packets, headers -included, for every datagram both can fill: the same sequence number, -timestamp, SSRC, and payload. A receiver applying -[ST 2022-7](https://pub.smpte.org/latest/st2022-7/st2022-7-2019.pdf) then -reconstructs the common content from whichever leg delivers each datagram, -through complementary loss on the two paths. - -Non-goals: the receiver's merge; a MoQ-side merge of two subscriptions, -which a relay's same-epoch failover already does at the object level; and -identity across a replaced broadcast. - -## Plan - -Decided 2026-10-13 by the maintainer: RTP identity is a -function of TS slot identity (`moq export ts --sync`) and the epoch, and of -nothing a leg mints. - -- Datagrams are cut on the slot grid, so their boundaries fall at the same - packets on every leg. The sequence number and the 90 kHz timestamp come - from the datagram's position on the broadcast's media timeline, modulo - their field widths, so they survive the 33-bit source wrap and a late - join. The SSRC derives from the broadcast path and the route's epoch, so - a replacement is a new RTP stream and two broadcasts sharing one epoch (a - multi-stage import clones its process epoch across paths) differ, barring - a 32-bit hash collision. RFC 3550's collision resolution would split the - legs, so it does not apply. An epochless route (a session whose version carries no - epoch) gives two legs nothing shared to derive it from, so synchronized - RTP refuses one; settle whether an explicit SSRC lifts that. -- A leg never sends a datagram it cannot fill exactly. A packetizer that - completes a datagram with stuffing, or primes a cushion with null-only - datagrams after a stall, sends them under sequence numbers its partner - fills with programme, and a receiver keeps whichever copy arrives first. - Silence lets the partner's copy through. The renderer hands the - packetizer which slots it filled with substitute stuffing, since the - bytes alone cannot tell them from the source's own null packets. -- Datagrams go out on the export's own release; no wall-clock scheduling. -- Settle the sink's shape (a destination on `export ts`, or its own sink), - and whether the passthrough export can be packetized the same way from a - constant-rate source. - -Tests: two legs fed one broadcast with different join offsets, delivery -skew, and complementary group loss send byte-identical RTP packets for every -datagram both send, and a reference ST 2022-7 selection over the two -reproduces every datagram either sent. A datagram neither leg can fill is -absent from both, never stuffing in one, and a datagram holding a slot one -leg substituted is absent from that leg. Sequence numbers and timestamps -wrap identically, a replacement or another path changes the SSRC, and an -epochless route is refused. Claims of ST 2022-7 compatibility cite the standard and an -RTP-level test, not TS identity. - -Public API: an RTP output for TS export, in the shape settled above. diff --git a/rs/moq-cli/src/args.rs b/rs/moq-cli/src/args.rs index ae4ad8b10b3..7a3128791a2 100644 --- a/rs/moq-cli/src/args.rs +++ b/rs/moq-cli/src/args.rs @@ -366,6 +366,18 @@ impl Invocation { stdout.linger.is_zero() || matches!(stdout.format, SubscribeFormat::Ts), "--linger needs an output that can mark a restart, and only `export ts` can" ); + anyhow::ensure!( + stdout.rtp.is_none() || stdout.sync, + "--rtp needs --sync: only a synced multiplex lays each packet where every leg does" + ); + anyhow::ensure!( + stdout.rtp.is_none_or(|addr| addr.port() != 0), + "--rtp needs a destination port other than 0" + ); + anyhow::ensure!( + stdout.rtp.is_none() || !stdout.max_delay.is_zero(), + "--rtp needs a --delay: without one no clock paces the datagrams" + ); if matches!(stdout.format, SubscribeFormat::H264 | SubscribeFormat::H265) { anyhow::ensure!( !export.select.no_video, @@ -1052,9 +1064,10 @@ impl ExportSink { } } - /// Whether this sink writes to stdout (the container formats and the report sinks). + /// Whether this sink writes to stdout (the container formats but `ts --rtp`, and the report + /// sinks). pub fn is_stdout(&self) -> bool { - self.stdout().is_some() || self.report().is_some() + self.stdout().is_some_and(|stdout| stdout.rtp.is_none()) || self.report().is_some() } /// The stdout container format and its options, when this sink writes to @@ -1069,6 +1082,7 @@ impl ExportSink { mux_rate: None, passthrough: false, sync: false, + rtp: None, }; let buffered = |format, args: &Buffered| Stdout { format, @@ -1079,6 +1093,7 @@ impl ExportSink { mux_rate: None, passthrough: false, sync: false, + rtp: None, }; Some(match self { Self::Fmp4(args) => Stdout { @@ -1098,6 +1113,7 @@ impl ExportSink { mux_rate: args.mux_rate, passthrough: args.passthrough, sync: args.sync, + rtp: args.rtp, }, Self::Flv(args) => buffered(SubscribeFormat::Flv, args), Self::H264(args) => container(SubscribeFormat::H264, args), @@ -1122,6 +1138,8 @@ pub struct Stdout { pub passthrough: bool, /// Render identical packets to every other `--sync` export of the broadcast (`ts` only). pub sync: bool, + /// Send RTP to this address rather than write to stdout (`ts` only). + pub rtp: Option, } /// Staleness options for stdout sinks without a fixed release delay. @@ -1251,6 +1269,15 @@ pub struct Transport { #[usage(long, conflicts("--stitch", "--passthrough"))] pub sync: bool, + /// Send the output as RTP (RFC 2250) over UDP to `ADDR` (`IP:PORT`) instead of stdout, seven + /// TS packets a datagram, with headers that follow from the packets' place on the multiplex + /// and from the broadcast, so the legs of a 1+1 pair send the same datagrams for an SMPTE + /// ST 2022-7 receiver to merge, paced at the multiplex rate on the `--delay` clock. Needs + /// `--sync`, a nonzero `--delay`, and a route carrying the publisher's epoch: moq-transport + /// draft 17 or later, or moq-lite 07, on every hop. + #[usage(long, value_name = "ADDR", conflicts("--stitch", "--passthrough"))] + pub rtp: Option, + /// The released spelling of [`Self::delay`]. #[usage(long = "max-age", hide = true)] max_age: Option, @@ -1393,6 +1420,41 @@ mod tests { } } + /// `export ts --rtp` sends what `--sync` renders, so it needs it and refuses what it + /// refuses, takes a destination with a port and a delay to pace by, refused before anything + /// dials, and leaves stdout to another stage. + #[test] + fn export_ts_rtp_needs_sync_and_leaves_stdout() { + let parse = |rtp: &str, args: &[&str]| { + let mut argv = vec!["moq", "export", "--broadcast", "b", "ts", "--rtp", rtp]; + argv.extend_from_slice(args); + Invocation::try_parse_from(argv) + }; + let cli = parse("127.0.0.1:5004", &["--sync"]).unwrap(); + cli.validate().unwrap(); + let Command::Export(export) = &cli.stages[0] else { + panic!("not an export"); + }; + assert_eq!(export.sink.stdout().unwrap().rtp, Some(([127, 0, 0, 1], 5004).into())); + assert!(!export.sink.is_stdout()); + let err = parse("127.0.0.1:5004", &[]).unwrap().validate().unwrap_err(); + assert!(err.to_string().contains("--rtp needs --sync"), "{err}"); + for flags in [&["--sync", "--stitch"][..], &["--passthrough"]] { + assert!( + parse("127.0.0.1:5004", flags).is_err(), + "{flags:?} must not start a run" + ); + } + assert!(parse("127.0.0.1", &["--sync"]).is_err(), "a destination without a port"); + let err = parse("[::1]:0", &["--sync"]).unwrap().validate().unwrap_err(); + assert!(err.to_string().contains("port other than 0"), "{err}"); + let err = parse("127.0.0.1:5004", &["--sync", "--delay", "0s"]) + .unwrap() + .validate() + .unwrap_err(); + assert!(err.to_string().contains("--rtp needs a --delay"), "{err}"); + } + fn export(flags: &[&str]) -> Result { let argv = ["moq", "--connect", "http://relay", "--broadcast", "room", "export"]; Invocation::try_parse_from(argv.iter().chain(flags).copied()) diff --git a/rs/moq-cli/src/main.rs b/rs/moq-cli/src/main.rs index aa28cdd09ce..07515df927e 100644 --- a/rs/moq-cli/src/main.rs +++ b/rs/moq-cli/src/main.rs @@ -832,6 +832,7 @@ fn spawn_export( mux_rate: stdout.mux_rate, passthrough: stdout.passthrough, sync: stdout.sync, + rtp: stdout.rtp, catalog: export.catalog_format, select: export.select, }; diff --git a/rs/moq-cli/src/subscribe.rs b/rs/moq-cli/src/subscribe.rs index 69beb57e054..f5d937de9a4 100644 --- a/rs/moq-cli/src/subscribe.rs +++ b/rs/moq-cli/src/subscribe.rs @@ -200,6 +200,9 @@ pub struct SubscribeArgs { /// Render identical packets to every other synced export of the broadcast (TS only). pub sync: bool, + /// Send the synced multiplex as RTP to this UDP address instead of stdout (TS only). + pub rtp: Option, + /// Catalog format for track discovery (default: detect from the broadcast suffix). pub catalog: Option, @@ -276,6 +279,52 @@ fn log_release(release: &moq_mux::container::ts::stats::Release, message: &'stat ); } +/// The datagrams `--rtp` could not send since it last reported them, and why the last one could +/// not. +#[derive(Default)] +struct Unsent { + count: u64, + last: Option, +} + +impl Unsent { + /// Send `datagram` now, or count it lost: a full send queue or a route briefly down loses + /// that datagram, as the network would, not the stream, and the export never waits on the + /// socket. A receiver not listening yet answers with ICMP, which a connected socket reports + /// on a later send, and is not counted. + fn send(&mut self, socket: &std::net::UdpSocket, datagram: &[u8]) { + match socket.send(datagram) { + Err(err) if err.kind() != std::io::ErrorKind::ConnectionRefused => { + self.count += 1; + self.last = Some(err); + } + _ => {} + } + } + + /// Log the datagrams lost since the last report, if any. + fn report(&mut self) { + if let Some(err) = self.last.take() { + let count = std::mem::take(&mut self.count); + tracing::warn!(count, %err, "--rtp dropped datagrams it could not send"); + } + } +} + +/// A UDP socket bound to an ephemeral port and connected to `addr`. It is nonblocking, and sent +/// on directly ([`Unsent::send`]): tokio's own readiness would refuse a send until its reactor +/// first saw the socket writable. +fn rtp_socket(addr: std::net::SocketAddr) -> std::io::Result { + let local: std::net::SocketAddr = match addr { + std::net::SocketAddr::V4(_) => (std::net::Ipv4Addr::UNSPECIFIED, 0).into(), + std::net::SocketAddr::V6(_) => (std::net::Ipv6Addr::UNSPECIFIED, 0).into(), + }; + let socket = std::net::UdpSocket::bind(local)?; + socket.connect(addr)?; + socket.set_nonblocking(true)?; + Ok(socket) +} + impl Subscribe { /// Resolve the broadcast at `path` on `origin` to export it with the resolved settings; /// [`run`](Self::run) drives it. @@ -422,6 +471,13 @@ impl Subscribe { if self.args.sync { ts = ts.with_sync(); } + let rtp = match self.args.rtp { + Some(addr) => { + ts = ts.with_rtp(); + Some(rtp_socket(addr).with_context(|| format!("could not open a UDP socket to {addr}"))?) + } + None => None, + }; // A TS byte stream carries no per-frame timing, so delivery time is the only // carrier of each frame's spacing (#2984). The export lays each slice of the PCR @@ -435,6 +491,7 @@ impl Subscribe { // the way `publish` reports one that stops arriving. let mut log = moq_mux::container::ts::stats::Log::default(); let mut sampled = moq_clock::now(); + let mut unsent = Unsent::default(); // The export's drop, gap, out-of-tolerance and `--sync` counts, as last logged. let mut reported = (0, 0, 0, 0, 0, 0); let sections = self.sections; @@ -447,16 +504,31 @@ impl Subscribe { } }; let end = loop { - let frame = match ts.next().await { - Ok(Some(frame)) => frame, - Ok(None) => break Ok(()), - Err(err) => break Err(err), + // Sampled on the interval whether a frame comes or not: an RTP leg withholding every + // run hands out none for as long as it does. + let due = sampled + moq_mux::container::ts::stats::Log::INTERVAL; + let next = tokio::select! { + next = ts.next() => Some(next), + _ = moq_clock::sleep_until(due) => None, }; - stdout.write_all(&frame.payload).await?; - stdout.flush().await?; + if let Some(next) = next { + let frame = match next { + Ok(Some(frame)) => frame, + Ok(None) => break Ok(()), + Err(err) => break Err(err), + }; + match &rtp { + Some(socket) => unsent.send(socket, &frame.payload), + None => { + stdout.write_all(&frame.payload).await?; + stdout.flush().await?; + } + } + } if sampled.elapsed() >= moq_mux::container::ts::stats::Log::INTERVAL { sampled = moq_clock::now(); + unsent.report(); let stats = ts.export().stats(); report(&stats); let release = &stats.release; @@ -475,6 +547,7 @@ impl Subscribe { log.sample(stats.into()); } }; + unsent.report(); let stats = ts.export().stats(); report(&stats); log_release(&stats.release, "TS export release clock"); @@ -539,3 +612,21 @@ impl Subscribe { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + /// A datagram the socket refuses is counted and dropped, and the export sends on. + #[test] + fn an_rtp_datagram_the_socket_refuses_is_counted_not_fatal() { + let socket = rtp_socket((std::net::Ipv4Addr::LOCALHOST, 9).into()).unwrap(); + let mut unsent = Unsent::default(); + // Longer than any UDP datagram over IPv4 can be. + unsent.send(&socket, &[0; 70_000]); + unsent.send(&socket, &[0; 1_328]); + assert_eq!(unsent.count, 1); + unsent.report(); + assert_eq!((unsent.count, unsent.last.is_none()), (0, true)); + } +} diff --git a/rs/moq-mux/benches/ts_schedule.rs b/rs/moq-mux/benches/ts_schedule.rs index 565ebd33eed..e4d94824107 100644 --- a/rs/moq-mux/benches/ts_schedule.rs +++ b/rs/moq-mux/benches/ts_schedule.rs @@ -1,5 +1,5 @@ -//! TS admission and layout across PID count and access units per PID, and padded layout -//! across queued tracks and access-unit size. +//! TS admission and layout across PID count and access units per PID, padded layout across +//! queued tracks and access-unit size, and RTP packetization across rate and unsure share. use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use std::hint::black_box; @@ -13,6 +13,9 @@ mod export { #[path = "../src/container/ts/lane.rs"] #[allow(dead_code)] mod lane; +#[path = "../src/container/ts/rtp.rs"] +#[allow(dead_code)] +mod rtp; #[path = "../src/container/ts/schedule.rs"] #[allow(dead_code)] mod schedule; @@ -135,5 +138,39 @@ fn tables(c: &mut Criterion) { } group.finish(); } -criterion_group!(benches, admission, tables); + +/// RTP packetization of a synced multiplex across its rate and the share of each slot's +/// positions the leg is unsure of: none, one in 64, every one. +fn rtp(c: &mut Criterion) { + let mut group = c.benchmark_group("ts_rtp"); + group.measurement_time(Duration::from_secs(1)); + group.warm_up_time(Duration::from_millis(200)); + group.sample_size(10); + for rate in [10_000_000u64, 100_000_000, 1_000_000_000] { + let layout = + std::sync::Arc::new(lane::Layout::new(rate, 2, 0, &[(0x100, lane::Need::Rest(u64::MAX))]).unwrap()); + let slots: Vec> = (0..64).map(|index| packet(0x100).repeat(layout.count(index))).collect(); + for every in [0usize, 64, 1] { + let unsure: Vec> = slots + .iter() + .map(|slot| match every { + 0 => Vec::new(), + every => (0..slot.len() / 188).step_by(every).collect(), + }) + .collect(); + let mbps = rate / 1_000_000; + group.bench_function(BenchmarkId::new(format!("{mbps}Mb/s"), every), |b| { + b.iter(|| { + let mut packetizer = rtp::Packetizer::new(layout.clone(), 0x1234_5678); + for (index, (slot, unsure)) in slots.iter().zip(&unsure).enumerate() { + black_box(packetizer.push(index as u128, slot, unsure)); + } + }); + }); + } + } + group.finish(); +} + +criterion_group!(benches, admission, tables, rtp); criterion_main!(benches); diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index 06163cb48a9..d0d94d6d6df 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -178,8 +178,11 @@ where /// Wakes the export when the next grid slot is due on the jitter buffer's clock. slot_timer: Option>>, /// Output frames ready to hand out, one per grid slot, each with what it tells - /// [`Self::stats`] once returned. - queue: VecDeque<(Frame, Tally)>, + /// [`Self::stats`] once returned and, for a datagram, when it is due ([`Self::pace`]). A slot + /// an RTP leg withholds whole queues its tally alone. + queue: VecDeque<(Option, Tally, Option)>, + /// Wakes the export when the datagram at the head of the queue is due. + pace_timer: Option>>, /// The rate to pad the output to with null packets, in bits per second: the /// builder override when set, else the catalog's recorded multiplex rate, else /// none and the output is unpadded ([`Schedule`]). @@ -211,6 +214,10 @@ where sync: bool, /// Under [`Self::with_sync`], the slot whose SI has been queued ([`Self::queue_si`]). si_slot: Option, + /// Hand out RTP datagrams rather than slots ([`Self::with_rtp`]). + rtp: bool, + /// Cuts the slots into datagrams, once the lanes are laid out. + packetizer: Option, } /// What a queued output frame tells [`Export::stats`], applied only once the frame is @@ -270,6 +277,10 @@ struct Synced { lane: Lane, /// The release slot of the last frame planned ([`Track::plan`]). released: Option, + /// The playhead's jumps when the last frame was planned ([`Pending::skip`]). + skip: Option, + /// Positions planned since the grid was last told ([`Grid::expect`]). + expected: Vec, } /// The PES an audio frame may share ([`Track::pack_start`]): the place of its first frame on @@ -726,6 +737,7 @@ impl Track { match plan { Ok(placed) => { lane::number(&mut packets, placed.counter); + sync.expected.extend(&placed.positions); Ok(Some(Planned { packets, positions: placed.positions, @@ -833,6 +845,11 @@ impl Track { .and_then(|ticks| Timestamp::from_scale(ticks, 90_000).ok()) .unwrap_or(frame.timestamp); let (key, pes) = self.release_key(decode, jitter.delay(), &frame)?; + if let Some(sync) = &mut self.sync + && sync.skip.replace(skip).is_some_and(|last| last != skip) + { + sync.lane.jump(); + } // Planned as it is read, so a frame the jitter buffer drops still moves its group on. let planned = match self.sync.is_some() { true => match self.plan(name, &frame, (decode, dts), description.as_ref(), group, jitter.delay())? { @@ -1371,6 +1388,7 @@ where slot_timer: None, replay: false, queue: VecDeque::new(), + pace_timer: None, liveness: Default::default(), video_start: None, video_out: None, @@ -1379,6 +1397,8 @@ where mux_rate_override: None, sync: false, si_slot: None, + rtp: false, + packetizer: None, } } @@ -1401,6 +1421,30 @@ where self } + /// Hand out MPEG-TS over RTP (RFC 2250) rather than slots: each frame is one datagram of + /// seven TS packets, its sequence number, timestamp and SSRC a function of where those + /// packets sit on the multiplex's timeline and of the broadcast's path and epoch. Two legs + /// of a broadcast then send the same datagram, headers included, wherever both render the + /// same packets, so a receiver merging them (SMPTE ST 2022-7) can take each from either. + /// + /// A leg sends a datagram only when it is sure of every packet: past the slot it joined or + /// last jumped at, and where each lane has gone past it or its source ended, so the leg + /// holds every unit any leg puts there. The rest are withheld, so another leg's copy is the + /// one that arrives. Two limits remain: SI a leg missed, and a unit of a sparse track lost + /// with no later jump in its track, go out as stuffing where another leg has them. + /// + /// Each datagram goes out on the jitter buffer's clock at its place in the slot after the one + /// it is laid in, at the multiplex's pace rather than in one burst a send queue may not hold. + /// + /// Needs [`Self::with_sync`], a [delay](Self::with_delay) for that clock, and a route with an + /// epoch: the SSRC must change with the publisher instance, and an epochless route names + /// none. Each fails at the program tables. + /// Only moq-transport draft 17 and later, and moq-lite 07, carry the epoch across a hop. + pub fn with_rtp(mut self) -> Self { + self.rtp = true; + self + } + /// Pad the output with null packets to `mux_rate` bits per second, whatever the /// catalog records. Without this the catalog's `mpegts.muxRate` decides, and a /// catalog without one leaves the output unpadded. @@ -1581,7 +1625,7 @@ where // 4. Mux each frame the jitter buffer lets go (the first carries the buffered // PAT/PMT), then lay out every grid slot whose time has come ([`Self::lay_due`]). loop { - if let Some(out) = self.pop() { + if let Some(out) = self.pop(waiter) { return Poll::Ready(Ok(Some(out))); } let ready = match self.held.take() { @@ -1599,6 +1643,11 @@ where self.held = Some(ready); continue; } + if !self.queue.is_empty() { + // That tail includes the datagrams still paced out ([`Self::pace`]). + self.held = Some(ready); + break; + } self.rewind(); self.generation = ready.generation; } @@ -1655,7 +1704,7 @@ where self.flush_packs(|_, _| true)?; } self.lay_due(waiter)?; - if let Some(out) = self.pop() { + if let Some(out) = self.pop(waiter) { return Poll::Ready(Ok(Some(out))); } @@ -1671,12 +1720,32 @@ where if self.slot_due().is_none() { self.lay(None)?; } - if let Some(out) = self.pop() { + if let Some(out) = self.pop(waiter) { return Poll::Ready(Ok(Some(out))); } - if !self.schedule.is_empty() { + if !self.schedule.is_empty() || !self.queue.is_empty() { return Poll::Pending; } + // Every lane is known past a clean end ([`Self::know`]), so the slots after it are + // alike on every leg: lay them until the datagram the media ended inside is whole. + if self.catalog_ended + && let Some(run) = self.packetizer.as_ref().and_then(|rtp| rtp.open()) + { + while self.packetizer.as_ref().and_then(|rtp| rtp.open()) == Some(run) + && let Some(next) = self.schedule.sync().and_then(|grid| grid.index()) + { + self.lay(Some(next + 1))?; + } + if let Some(rtp) = &mut self.packetizer { + rtp.close(); + } + if let Some(out) = self.pop(waiter) { + return Poll::Ready(Ok(Some(out))); + } + if !self.queue.is_empty() { + return Poll::Pending; + } + } // SI emission rides media frames, so a snapshot that arrived behind the // last one gets one trailing flush before the stream ends. if self.catalog_ended && !self.si_flushed && !self.sync { @@ -1695,19 +1764,50 @@ where Poll::Pending } - /// Return the next queued frame, committing what it carries to the output boundary. - fn pop(&mut self) -> Option { - let (out, tally) = self.queue.pop_front()?; - self.emitted_epoch = self.epoch; - let (pcr, discontinuity) = tally.pcr; - if discontinuity { - self.liveness.discontinuity(); - } - self.liveness.written_pcr(pcr); - for pid in tally.units { - self.liveness.delivered(pid, 1); + /// Return the next queued frame once it is due, committing what it carries, and what each + /// slot withheld before it carried, to the output boundary. + fn pop(&mut self, waiter: &kio::Waiter) -> Option { + loop { + if let Some(due) = self.queue.front()?.2 + && moq_clock::now() < due + { + let timer = self + .pace_timer + .get_or_insert_with(|| Box::pin(moq_clock::sleep_until(due))); + if timer.deadline() != due { + timer.as_mut().reset(due); + } + if waiter.poll_future(timer.as_mut()).is_pending() { + return None; + } + } + let (out, tally, _) = self.queue.pop_front()?; + let (pcr, discontinuity) = tally.pcr; + if discontinuity { + self.liveness.discontinuity(); + } + self.liveness.written_pcr(pcr); + for pid in tally.units { + self.liveness.delivered(pid, 1); + } + if let Some(out) = out { + self.emitted_epoch = self.epoch; + return Some(out); + } } - Some(out) + } + + /// When the datagrams of slot `index` go out: spread across the slot after the one it is laid + /// in ([`Self::slot_due`]), from its start, at the multiplex's pace rather than in one burst a + /// send queue may not hold. + fn pace(&self, index: u128) -> Option<(moq_clock::Instant, Duration)> { + let at = |slot: u128| { + let key = slot * PCR_INTERVAL.as_nanos() + self.max_drain().as_nanos() + self.delay.as_nanos(); + let key = Timestamp::from_micros(u64::try_from(key / 1_000).ok()?).ok()?; + self.jitter.at(self.generation, key) + }; + let (start, end) = (at(index + 1)?, at(index + 2)?); + Some((start, end.saturating_duration_since(start))) } /// The trailing SI frame for end of stream: every entry's current sections, @@ -2077,16 +2177,20 @@ where self.jitter.len() } - /// When the next queued frame is due, or the next grid slot while media is queued or - /// every source has finished and the tail runs out. + /// When the next queued frame is due, the next grid slot while media is queued or every + /// source has finished and the tail runs out, or the next datagram paced out. #[cfg(test)] pub(super) fn next_due(&self) -> Option { let finished = !self.tracks.is_empty() && self.tracks.values().all(|track| track.finished); + // RTP paces every slot's datagrams across it, so a slot holding nothing goes out on time too. let slot = self .slot_due() - .filter(|_| self.schedule.queued() || self.packing() || finished) + .filter(|_| self.packetizer.is_some() || self.schedule.queued() || self.packing() || finished) .map(|(_, at)| at); - self.jitter.next_deadline().into_iter().chain(slot).min() + let paced = self.queue.front().and_then(|(_, _, due)| *due); + // A frame held for the next generation waits on the paced tail, not on the jitter buffer. + let jitter = self.jitter.next_deadline().filter(|_| self.held.is_none()); + jitter.into_iter().chain(slot).chain(paced).min() } /// The discontinuity counter of the most recently returned output frame. @@ -2144,6 +2248,7 @@ where let mut next = Self::new(self.source.clone(), catalog); next.replay = self.replay; next.sync = self.sync; + next.rtp = self.rtp; next = next.with_delay(self.delay); if let Some(rate) = self.mux_rate_override { next = next.with_mux_rate(rate); @@ -2244,7 +2349,7 @@ where if let Some(flagged) = &mut self.flagged { flagged.clear(); } - self.queue.clear(); + debug_assert!(self.queue.is_empty(), "the old generation's output went out first"); self.last_psi = None; for si in self.si.values_mut() { si.last_emit = None; @@ -2310,6 +2415,14 @@ where /// Build the PAT/PMT once every track's PID and codec is known. fn build_psi(&mut self) -> anyhow::Result<()> { + anyhow::ensure!( + self.sync || !self.rtp, + "MPEG-TS over RTP needs --sync: only a synced multiplex lays each packet where every leg does" + ); + anyhow::ensure!( + !self.rtp || !self.delay.is_zero(), + "MPEG-TS over RTP needs a --delay: without one no clock paces the datagrams" + ); // A measured rate would give each leg a grid of its own. let sync = match self.sync { true => Some(self.mux_rate.context( @@ -2494,18 +2607,48 @@ where .collect(); needs.sort_unstable_by_key(|(pid, _)| *pid); let layout = Arc::new(Layout::new(rate, tables.len(), SI_LANE, &needs)?); + if self.rtp { + let epoch = self.instance.as_ref().context( + "MPEG-TS over RTP needs a route with an epoch, so the SSRC changes with the publisher instance: \ + connect every hop on moq-transport draft 17 or later, or moq-lite 07", + )?; + let ssrc = super::rtp::ssrc(self.source.path().as_str(), epoch); + self.packetizer = Some(super::rtp::Packetizer::new(layout.clone(), ssrc)); + } for track in self.tracks.values_mut() { let lane = layout.lane(track.pid).context("a track without a lane")?; + // A placed lane no receiver buffer models has no cadence: SCTE-35, a PES-packet stream. + let sparse = track.kind.placed() && track.buffer.is_none(); track.sync = Some(Synced { layout: layout.clone(), - lane: Lane::new(lane), + lane: Lane::new(lane, sparse), released: None, + skip: None, + expected: Vec::new(), }); } self.schedule.set_sync(Grid::new(layout, tables)); Ok(()) } + /// Tell the grid, before it lays a slot, what each lane holds on this leg: the units it + /// planned, and how far it knows the lane ([`Lane::known`]). A source that ended, rather + /// than went stale with its broadcast, carries nothing more on any leg. + fn know(&mut self) { + let Some(grid) = self.schedule.sync() else { + return; + }; + for (name, track) in self.tracks.iter_mut() { + let Some(sync) = &mut track.sync else { + continue; + }; + grid.expect(&sync.expected); + sync.expected.clear(); + let ended = track.finished && !self.stale.contains(name); + grid.know(sync.lane.index(), sync.lane.known(ended)); + } + } + /// Packetize one media frame and queue it on the [`Schedule`], re-emitting PAT/PMT /// before video keyframes (and periodically) so receivers can tune in mid-stream. fn mux(&mut self, name: &str, queued: Queued) -> anyhow::Result<()> { @@ -2990,6 +3133,7 @@ where { self.queue_si(index)?; } + self.know(); let Some(slot) = self.schedule.next(known)? else { break; }; @@ -3005,17 +3149,43 @@ where if !self.sync { self.number(&mut payload); } + let timestamp = slot_stamp(slot.index)?; + let tally = Tally { + pcr: (slot.pcr, discontinuity), + units: slot.units, + }; + let pace = self.pace(slot.index); + if let Some(rtp) = &mut self.packetizer { + let positions = (payload.len() / TsPacket::SIZE) as u32; + let mut tally = Some(tally); + for (last, datagram) in rtp.push(slot.index, &payload, &slot.unsure) { + let frame = Frame { + timestamp, + duration: None, + payload: datagram, + keyframe: false, + }; + let tally = tally.take().unwrap_or(Tally { + pcr: (slot.pcr, false), + units: Vec::new(), + }); + // At its last packet's place in the slot, whichever runs beside it this leg + // withholds, so legs send each copy together. + let due = pace.map(|(at, span)| at + span * last as u32 / positions); + self.queue.push_back((Some(frame), tally, due)); + } + if let Some(tally) = tally { + self.queue.push_back((None, tally, None)); + } + continue; + } let frame = Frame { - timestamp: slot_stamp(slot.index)?, + timestamp, duration: None, payload: Bytes::from(payload), keyframe: slot.keyframe, }; - let tally = Tally { - pcr: (slot.pcr, discontinuity), - units: slot.units, - }; - self.queue.push_back((frame, tally)); + self.queue.push_back((Some(frame), tally, None)); } Ok(()) } diff --git a/rs/moq-mux/src/container/ts/export_timing_test.rs b/rs/moq-mux/src/container/ts/export_timing_test.rs index a5e8a929d76..9169a5ad055 100644 --- a/rs/moq-mux/src/container/ts/export_timing_test.rs +++ b/rs/moq-mux/src/container/ts/export_timing_test.rs @@ -52,6 +52,28 @@ const JOIN: u64 = 3 * GOP + 8; const VIDEO_PID: u16 = 0x1001; const AUDIO_PID: u16 = 0x1002; +const TELETEXT_PID: u16 = 0x1010; +const SECTION_PID: u16 = 0x1020; + +/// A teletext PES of media time `micros`: its PTS in the fixed header teletext takes, and a +/// stuffing data unit (ETSI EN 300 472). +fn teletext_pes(micros: u64) -> Bytes { + let pts = micros * 9 / 100; + let mut pes = vec![0, 0, 1, 0xbd, 0, 0, 0x84, 0x80, 0x24]; + pes.extend_from_slice(&[ + 0x21 | ((pts >> 29) & 0x0e) as u8, + (pts >> 22) as u8, + 1 | ((pts >> 14) & 0xfe) as u8, + (pts >> 7) as u8, + 1 | ((pts << 1) & 0xfe) as u8, + ]); + pes.resize(9 + 0x24, 0xff); + pes.extend_from_slice(&[0x10, 0xff, 0x2c]); + pes.resize(pes.len() + 0x2c, 0xff); + let len = (pes.len() - 6) as u16; + pes[4..6].copy_from_slice(&len.to_be_bytes()); + Bytes::from(pes) +} fn length_prefixed(nals: &[&[u8]]) -> Bytes { let mut out = BytesMut::new(); @@ -64,8 +86,8 @@ fn length_prefixed(nals: &[&[u8]]) -> Bytes { /// A live H.264 + AAC broadcast whose frames are written on the source's clock. struct Live { - _broadcast: moq_net::broadcast::Producer, - catalog: crate::catalog::Producer, + broadcast: moq_net::broadcast::Producer, + catalog: crate::catalog::Producer, /// Written group by group, so a group can be left open: see [`Self::run`]. video: moq_net::track::Producer, group: Option, @@ -73,6 +95,11 @@ struct Live { stalling: bool, stalled: Vec, audio: Producer, + /// A teletext track, and the video ticks its groups stall at: see [`Self::teletext`]. + teletext: Option<(moq_net::track::Producer, std::ops::Range)>, + /// An SCTE-35 track, and the video ticks whose group opens with a malformed section: see + /// [`Self::sections`]. + sections: Option<(moq_net::track::Producer, std::ops::Range)>, source: crate::Source, /// Media time per unit of the receivers' clock: above 1 the source runs fast. scale: f64, @@ -80,6 +107,8 @@ struct Live { rate: u64, /// The receivers render with [`Export::with_sync`]. sync: bool, + /// The receivers send RTP ([`Export::with_rtp`]). + rtp: bool, /// Video is a B-pyramid: see [`Self::reorder`]. reorder: bool, /// The `initial_cpb_removal_delay` each group's first picture opens a buffering period with: @@ -96,13 +125,19 @@ struct Live { start: Instant, tick: u64, audio_index: u64, + /// The next audio frame opens a group: the first after a [restart](Self::restart_audio). + audio_opens: bool, } impl Live { async fn new(scale: f64, audio_lag: i64) -> Self { let mut broadcast = moq_net::broadcast::Info::new().produce(); let consumer = broadcast.consume(); - let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut catalog = crate::catalog::Producer::new( + &mut broadcast, + crate::catalog::Config::default().with_catalog(crate::catalog::hang::Catalog::::default()), + ) + .unwrap(); let track = broadcast .create_track( @@ -145,16 +180,19 @@ impl Live { Self { source: crate::source::announced(&consumer).await, - _broadcast: broadcast, + broadcast, catalog, video, group: None, stalling: false, stalled: Vec::new(), audio, + teletext: None, + sections: None, scale, rate: MUX_RATE, sync: false, + rtp: false, reorder: false, periods: None, burst: None, @@ -163,9 +201,70 @@ impl Live { start: moq_clock::now(), tick: 0, audio_index: 0, + audio_opens: false, } } + /// Have the receivers send RTP of the broadcast served on a route of publisher instance + /// `epoch`, as a relay serves it. + async fn rtp(&mut self, epoch: &moq_net::Epoch) { + let route = moq_net::origin::Route::default().with_epoch(epoch.clone()); + self.source = crate::source::announced_on(&self.broadcast.consume(), route).await; + self.sync = true; + self.rtp = true; + } + + /// Send a DVB teletext PES on its own PID at each video tick, a group each, but in `gap`, + /// whose groups stall as [`Self::run`]'s do. + fn teletext(&mut self, gap: std::ops::Range) { + let name = self.broadcast.unique_name(".ts"); + let track = self + .broadcast + .create_track(name, hang::container::track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let mut entry = super::Track::new(TELETEXT_PID); + entry.pes_packet = Some(super::PesPacket::new(0x06)); + entry.descriptors.push(super::Descriptor { + tag: 0x56, + data: Bytes::from_static(&[b'e', b'n', b'g', 0x09, 0x00]), + }); + self.catalog + .modify() + .unwrap() + .ext + .mpegts + .tracks + .insert(track.name().to_string(), entry); + self.teletext = Some((track, gap)); + } + + /// Send two SCTE-35 sections on their own PID at each keyframe tick, a group each, the + /// first of a group in `malformed` cut short, so the export drops it and holds the rest of + /// the group only in part. + fn sections(&mut self, malformed: std::ops::Range) { + let track = self.section_track(); + self.sections = Some((track, malformed)); + } + + /// An SCTE-35 track on its own PID, which sends nothing until written. + fn section_track(&mut self) -> moq_net::track::Producer { + let name = self.broadcast.unique_name(".ts"); + let track = self + .broadcast + .create_track(name, hang::container::track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let mut entry = super::Track::new(SECTION_PID); + entry.verbatim = Some(super::Verbatim::new(0x86, super::Framing::Section)); + self.catalog + .modify() + .unwrap() + .ext + .mpegts + .tracks + .insert(track.name().to_string(), entry); + track + } + /// The receivers' instant at which the source's clock reads media time `micros`. fn sent(&self, micros: u64) -> Instant { self.start + Duration::from_secs_f64(micros as f64 / 1e6 / self.scale) @@ -229,6 +328,22 @@ impl Live { self.stalling = true; } + /// End the video, as a broadcast whose video has stopped: its lane carries nothing more. + fn end_video(&mut self) { + if let Some(mut group) = self.group.take() { + group.finish().unwrap(); + } + self.video.finish().unwrap(); + } + + /// Declare a marker on the audio and move the timeline `by` µs ahead from there, as a + /// source that restarted it. + fn restart_audio(&mut self, by: u64) { + self.audio.discontinuity().unwrap(); + self.audio_opens = true; + self.origin += by; + } + /// Send nothing for `gap`, as a publisher that died, then resume at the live edge on the /// same timeline from the next group, as its replacement or the other half of a 1+1 pair /// would. @@ -242,7 +357,7 @@ impl Live { /// A receiver joining now. async fn join(&self, every: Duration) -> Leg { - let mut export = super::export_test::open_export(self.source.clone()) + let mut export = super::export_test::open_ts(self.source.clone(), crate::catalog::CatalogFormat::Hang) .await .unwrap() .with_delay(DELAY) @@ -250,6 +365,9 @@ impl Live { if self.sync { export = export.with_sync(); } + if self.rtp { + export = export.with_rtp(); + } Leg { export, every, @@ -321,6 +439,50 @@ impl Live { ) .unwrap(); } + if let Some((teletext, gap)) = &mut self.teletext { + let mut group = teletext.append_group().unwrap(); + if gap.contains(&tick) { + self.stalled.push(group); + } else { + let micros = self.origin + video; + HangContainer::Legacy(crate::container::Kind::Data) + .write( + &mut group, + &[Frame { + timestamp: Timestamp::from_micros(micros).unwrap(), + duration: None, + payload: teletext_pes(micros), + keyframe: true, + }], + ) + .unwrap(); + group.finish().unwrap(); + } + } + if let Some((sections, malformed)) = &mut self.sections + && keyframe + { + let mut group = sections.append_group().unwrap(); + let frames = [0, 1].map(|n: u8| { + // A splice_null, which `n` marks apart. + let mut section = vec![ + 0xfc, 0x30, 0x11, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xf0, 0, 0, 0, n, 0, 0, 0, 0, + ]; + if n == 0 && malformed.contains(&tick) { + section.truncate(10); + } + Frame { + timestamp: Timestamp::from_micros(self.origin + video + u64::from(n) * VIDEO_US).unwrap(), + duration: None, + payload: Bytes::from(section), + keyframe: n == 0, + } + }); + HangContainer::Legacy(crate::container::Kind::Data) + .write(&mut group, &frames) + .unwrap(); + group.finish().unwrap(); + } self.tick += 1; } for leg in legs.iter_mut() { @@ -337,7 +499,7 @@ impl Live { timestamp: Timestamp::from_micros(self.origin + index * AUDIO_US).unwrap(), duration: None, payload: Bytes::from_iter((0..180u16).map(|i| (i ^ index as u16) as u8)), - keyframe: index.is_multiple_of(AUDIO_GROUP), + keyframe: std::mem::take(&mut self.audio_opens) || index.is_multiple_of(AUDIO_GROUP), }) .unwrap(); self.audio_index += 1; @@ -364,6 +526,12 @@ impl Live { self.stalled.clear(); self.video.finish().unwrap(); self.audio.finish().unwrap(); + if let Some((teletext, _)) = &mut self.teletext { + teletext.finish().unwrap(); + } + if let Some((sections, _)) = &mut self.sections { + sections.finish().unwrap(); + } for leg in legs.iter_mut() { leg.drain().await; } @@ -372,7 +540,7 @@ impl Live { /// One receiver: an exporter and what it rendered, with the instant each slot went out. struct Leg { - export: Export, + export: Export>, /// How often it reads its subscription. Zero reads after every frame the source sends; a /// coarser cadence stands for a receiver behind a burstier network or scheduler. every: Duration, @@ -1121,52 +1289,8 @@ async fn legs_stopped_in_turn_stay_complementary() { /// for over a second while the relay evicts its group, else by the source stalling a group /// for that leg alone. async fn complementary_losses(reorder: bool, stop: bool) { - let lost_a = |tick: u64| (5 * GOP + 8..6 * GOP).contains(&tick); - let lost_b = |tick: u64| (7 * GOP + 5..8 * GOP).contains(&tick); - let stopped_a = |tick: u64| (5 * GOP + 8..7 * GOP).contains(&tick); - let stopped_b = |tick: u64| (8 * GOP + 5..10 * GOP).contains(&tick); for lag in LAGS { - let mut paths = Vec::new(); - for _ in 0..3 { - let mut live = Live::new(1.0, lag).await; - live.sync = true; - if reorder { - live.reorder(); - } - paths.push(live); - } - let [mut clean, mut path_a, mut path_b] = <[Live; 3]>::try_from(paths).ok().unwrap(); - assert!(clean.start == path_a.start && clean.start == path_b.start); - let mut c = clean.join(Duration::ZERO).await; - let mut a = path_a.join(Duration::ZERO).await; - let mut b = path_b.join(Duration::from_millis(100)).await; - for tick in 1..=12 * GOP { - clean.run(tick, &mut [&mut c], |_| false).await; - match (stop, stopped_a(tick)) { - (false, _) => path_a.run(tick, &mut [&mut a], lost_a).await, - (true, false) => path_a.run(tick, &mut [&mut a], |_| false).await, - (true, true) => { - if !stopped_a(tick - 1) { - path_a.evict(); - } - path_a.run(tick, &mut [], |_| false).await - } - } - match (stop, stopped_b(tick)) { - (false, _) => path_b.run(tick, &mut [&mut b], lost_b).await, - (true, false) => path_b.run(tick, &mut [&mut b], |_| false).await, - (true, true) => { - if !stopped_b(tick - 1) { - path_b.evict(); - } - path_b.run(tick, &mut [], |_| false).await - } - } - } - clean.finish(&mut [&mut c]).await; - path_a.finish(&mut [&mut a]).await; - path_b.finish(&mut [&mut b]).await; - + let [a, b, c] = lose_in_turn(lag, reorder, stop, None).await; let from = 2 * GOP * VIDEO_US; // A stopped leg may resume behind the slots it laid, and drop what it reads for them. for (leg, stopped) in [(&a, stop), (&b, stop), (&c, false)] { @@ -1197,6 +1321,525 @@ async fn complementary_losses(reorder: bool, stop: bool) { } } +/// Three legs of a broadcast on `lag`, each on a path of its own: A and B each lose part of +/// it the other receives, as [`complementary_losses`] describes, and C loses nothing. With +/// `epoch`, each path serves it on a route of that publisher instance, and the legs send RTP. +async fn lose_in_turn(lag: i64, reorder: bool, stop: bool, epoch: Option<&moq_net::Epoch>) -> [Leg; 3] { + let lost_a = |tick: u64| (5 * GOP + 8..6 * GOP).contains(&tick); + let lost_b = |tick: u64| (7 * GOP + 5..8 * GOP).contains(&tick); + let stopped_a = |tick: u64| (5 * GOP + 8..7 * GOP).contains(&tick); + let stopped_b = |tick: u64| (8 * GOP + 5..10 * GOP).contains(&tick); + let mut paths = Vec::new(); + for _ in 0..3 { + let mut live = Live::new(1.0, lag).await; + live.sync = true; + if reorder { + live.reorder(); + } + if let Some(epoch) = epoch { + live.rtp(epoch).await; + } + paths.push(live); + } + let [mut clean, mut path_a, mut path_b] = <[Live; 3]>::try_from(paths).ok().unwrap(); + assert!(clean.start == path_a.start && clean.start == path_b.start); + let mut c = clean.join(Duration::ZERO).await; + let mut a = path_a.join(Duration::ZERO).await; + let mut b = path_b.join(Duration::from_millis(100)).await; + for tick in 1..=12 * GOP { + clean.run(tick, &mut [&mut c], |_| false).await; + match (stop, stopped_a(tick)) { + (false, _) => path_a.run(tick, &mut [&mut a], lost_a).await, + (true, false) => path_a.run(tick, &mut [&mut a], |_| false).await, + (true, true) => { + if !stopped_a(tick - 1) { + path_a.evict(); + } + path_a.run(tick, &mut [], |_| false).await + } + } + match (stop, stopped_b(tick)) { + (false, _) => path_b.run(tick, &mut [&mut b], lost_b).await, + (true, false) => path_b.run(tick, &mut [&mut b], |_| false).await, + (true, true) => { + if !stopped_b(tick - 1) { + path_b.evict(); + } + path_b.run(tick, &mut [], |_| false).await + } + } + } + clean.finish(&mut [&mut c]).await; + path_a.finish(&mut [&mut a]).await; + path_b.finish(&mut [&mut b]).await; + [a, b, c] +} + +/// The RTP datagrams a leg sent from media time `from` (µs), keyed by the slot that completed +/// each and its sequence number. +fn datagrams(leg: &Leg, from: u64) -> std::collections::BTreeMap<(u64, u16), Bytes> { + leg.out + .iter() + .map(|(_, frame)| { + let seq = u16::from_be_bytes([frame.payload[2], frame.payload[3]]); + ((frame.timestamp.as_micros() as u64, seq), frame.payload.clone()) + }) + .filter(|((at, _), _)| *at >= from) + .collect() +} + +/// Assert that every datagram `a` sent that `b` sent too is the same, headers included, and +/// return how many there are. +fn same_datagrams( + a: &std::collections::BTreeMap<(u64, u16), Bytes>, + b: &std::collections::BTreeMap<(u64, u16), Bytes>, +) -> usize { + let mut both = 0; + for (key, datagram) in a { + if let Some(other) = b.get(key) { + assert_eq!(datagram, other, "the datagram at {key:?} differs"); + both += 1; + } + } + both +} + +/// RTP legs of one broadcast, joined at different points, read on different cadences and with +/// each track lagging in turn, send the same datagrams, headers included, for every run of the +/// timeline both send. Each is one RTP packet of seven TS packets, under the SSRC the path and +/// epoch give, and once the joiner holds every track whole both send the same runs. +#[tokio::test(start_paused = true)] +async fn rtp_legs_send_the_same_datagrams() { + let epoch = moq_net::Epoch::mint(); + let ssrc = super::rtp::ssrc("test", &epoch).to_be_bytes(); + for lag in LAGS { + for join in [JOIN, JOIN + GOP] { + let mut live = Live::new(1.0, lag).await; + live.rtp(&epoch).await; + let (a, b, from) = synced_with(live, join, 12 * GOP, |_| false).await; + let (x, y) = (datagrams(&a, 0), datagrams(&b, 0)); + for datagram in x.values().chain(y.values()) { + assert_eq!(datagram.len(), 12 + 7 * 188); + assert_eq!(&datagram[..2], [0x80, 33]); + assert_eq!(&datagram[8..12], ssrc); + } + let both = same_datagrams(&x, &y); + let (x, y) = (datagrams(&a, from), datagrams(&b, from)); + assert!(x.len() > 300, "lag {lag}, join {join}: {} datagrams", x.len()); + assert_eq!( + x.keys().collect::>(), + y.keys().collect::>(), + "lag {lag}, join {join}: clean legs send different runs ({both} in common)" + ); + } + } +} + +/// RTP legs that each lose part of the broadcast the other receives send only datagrams a +/// clean leg sends too, headers included, and between them every one it sends: a receiver +/// merging them (SMPTE ST 2022-7) reproduces the clean leg. Each withholds the runs it lost. +#[tokio::test(start_paused = true)] +async fn rtp_legs_that_lose_in_turn_merge_into_a_clean_leg() { + let epoch = moq_net::Epoch::mint(); + for stop in [false, true] { + for lag in LAGS { + let [a, b, c] = lose_in_turn(lag, true, stop, Some(&epoch)).await; + let from = 2 * GOP * VIDEO_US; + let (a, b, c) = (datagrams(&a, from), datagrams(&b, from), datagrams(&c, from)); + for (name, leg) in [("A", &a), ("B", &b)] { + assert_eq!( + same_datagrams(leg, &c), + leg.len(), + "stop {stop}, lag {lag}: {name} sent a run the clean leg did not" + ); + assert!(leg.len() < c.len(), "stop {stop}, lag {lag}: {name} withheld nothing"); + } + let missing: Vec<_> = c + .keys() + .filter(|key| !a.contains_key(key) && !b.contains_key(key)) + .collect(); + assert!( + missing.is_empty(), + "stop {stop}, lag {lag}: neither leg sent {missing:?}" + ); + } + } +} + +/// Two legs that lose the same group's tail, on paths of their own, send only datagrams a leg +/// holding it sends too, so the runs neither can fill are absent from both. +#[tokio::test(start_paused = true)] +async fn a_run_no_leg_can_fill_is_sent_by_neither() { + let epoch = moq_net::Epoch::mint(); + let lost = |tick: u64| (8 * GOP + 5..9 * GOP).contains(&tick); + for lag in LAGS { + let mut paths = Vec::new(); + for _ in 0..3 { + let mut live = Live::new(1.0, lag).await; + live.rtp(&epoch).await; + paths.push(live); + } + let [mut whole, mut path_a, mut path_b] = <[Live; 3]>::try_from(paths).ok().unwrap(); + let mut c = whole.join(Duration::ZERO).await; + let mut a = path_a.join(Duration::ZERO).await; + let mut b = path_b.join(Duration::from_millis(100)).await; + for tick in 1..=12 * GOP { + whole.run(tick, &mut [&mut c], |_| false).await; + path_a.run(tick, &mut [&mut a], lost).await; + path_b.run(tick, &mut [&mut b], lost).await; + } + whole.finish(&mut [&mut c]).await; + path_a.finish(&mut [&mut a]).await; + path_b.finish(&mut [&mut b]).await; + let from = 2 * GOP * VIDEO_US; + let (a, b, c) = (datagrams(&a, from), datagrams(&b, from), datagrams(&c, from)); + for (name, leg) in [("A", &a), ("B", &b)] { + assert_eq!( + same_datagrams(leg, &c), + leg.len(), + "lag {lag}: {name} sent a run the whole leg did not" + ); + } + let neither = c + .keys() + .filter(|key| !a.contains_key(key) && !b.contains_key(key)) + .count(); + assert!(neither > 0, "lag {lag}: the legs sent every run between them"); + } +} + +/// An RTP leg sends each run at its place in the slot, at the multiplex's pace rather than in +/// one burst a send queue may not hold, so a leg that withholds some of a slot's runs sends the +/// rest when a leg sending them all does, and a receiver merging them finds both copies together. +#[tokio::test(start_paused = true)] +async fn rtp_legs_send_each_run_at_its_place_in_the_slot() { + let epoch = moq_net::Epoch::mint(); + let lost = |tick: u64| (5 * GOP + 8..6 * GOP).contains(&tick); + let mut paths = Vec::new(); + for _ in 0..2 { + let mut live = Live::new(1.0, 0).await; + live.rtp(&epoch).await; + paths.push(live); + } + let [mut whole, mut lossy] = <[Live; 2]>::try_from(paths).ok().unwrap(); + assert!(whole.start == lossy.start); + let mut c = whole.join(Duration::ZERO).await; + let mut a = lossy.join(Duration::ZERO).await; + for tick in 1..=8 * GOP { + // Both legs read at each instant either would, so each sends as it falls due. + whole.run(tick, &mut [&mut c, &mut a], |_| false).await; + lossy.run(tick, &mut [&mut a], lost).await; + } + // How many of each slot's datagrams the whole leg sent at each instant. + let mut slots: std::collections::BTreeMap> = Default::default(); + for (at, frame) in &c.out { + let slot = frame.timestamp.as_micros() as u64; + if slot >= 2_000_000 { + *slots.entry(slot).or_default().entry(*at).or_default() += 1; + } + } + assert!(slots.len() > 50, "{} slots", slots.len()); + for (slot, sent) in &slots { + let (busiest, count) = (sent.values().max().unwrap(), sent.values().sum::()); + assert!( + *busiest <= 2, + "the slot at {slot} µs sent {busiest} of its {count} datagrams at once" + ); + } + let sent = |leg: &Leg| -> std::collections::BTreeMap<(u64, u16), Instant> { + leg.out + .iter() + .map(|(at, frame)| { + let seq = u16::from_be_bytes([frame.payload[2], frame.payload[3]]); + ((frame.timestamp.as_micros() as u64, seq), *at) + }) + .collect() + }; + let (a, c) = (sent(&a), sent(&c)); + assert!(a.len() < c.len(), "the lossy leg withheld nothing"); + let apart: Vec<_> = a + .iter() + .filter_map(|(key, at)| { + let other = c.get(key)?; + (at != other).then(|| (*key, at.max(other).duration_since(*at.min(other)))) + }) + .collect(); + assert!( + apart.is_empty(), + "{} runs went out apart: {:?}", + apart.len(), + &apart[..apart.len().min(5)] + ); +} + +/// A leg that withholds runs counts what they carry as it lays them, so its stats keep pace +/// with a leg that sends them however long it withholds. +#[tokio::test(start_paused = true)] +async fn an_rtp_leg_counts_what_it_withholds() { + let epoch = moq_net::Epoch::mint(); + let lost = |tick: u64| (5 * GOP + 8..6 * GOP).contains(&tick); + let mut paths = Vec::new(); + for _ in 0..2 { + let mut live = Live::new(1.0, 0).await; + live.rtp(&epoch).await; + paths.push(live); + } + let [mut whole, mut lossy] = <[Live; 2]>::try_from(paths).ok().unwrap(); + let mut c = whole.join(Duration::ZERO).await; + let mut a = lossy.join(Duration::ZERO).await; + let units = |leg: &Leg| { + leg.export + .stats() + .streams + .get(&AUDIO_PID) + .map_or(0, |stream| stream.units) + }; + let mut behind = 0; + for tick in 1..=8 * GOP { + whole.run(tick, &mut [&mut c], |_| false).await; + lossy.run(tick, &mut [&mut a], lost).await; + behind = behind.max(units(&c).saturating_sub(units(&a))); + } + assert!( + behind <= 2, + "the withholding leg's stats fell {behind} audio units behind" + ); + assert!( + datagrams(&a, 0).len() < datagrams(&c, 0).len(), + "the lossy leg withheld nothing" + ); +} + +/// A teletext lane, which a receiver buffer models, stalls on one path for longer than the +/// jitter buffer waits for it: that leg withholds the runs its lane may yet fill, so what it +/// sends a leg holding the lane sends too. +#[tokio::test(start_paused = true)] +async fn a_stalled_teletext_lane_is_withheld() { + let epoch = moq_net::Epoch::mint(); + let mut paths = Vec::new(); + for gap in [0..0, 5 * GOP..8 * GOP] { + let mut live = Live::new(1.0, 0).await; + // Room for the keyframe beside teletext's lane. + live.rate = 2 * MUX_RATE; + live.teletext(gap); + live.rtp(&epoch).await; + paths.push(live); + } + let [mut whole, mut lossy] = <[Live; 2]>::try_from(paths).ok().unwrap(); + let mut c = whole.join(Duration::ZERO).await; + let mut a = lossy.join(Duration::ZERO).await; + for tick in 1..=12 * GOP { + whole.run(tick, &mut [&mut c], |_| false).await; + lossy.run(tick, &mut [&mut a], |_| false).await; + } + whole.finish(&mut [&mut c]).await; + lossy.finish(&mut [&mut a]).await; + let (a, c) = (datagrams(&a, 0), datagrams(&c, 0)); + assert_eq!( + same_datagrams(&a, &c), + a.len(), + "the stalled leg sent a run the whole leg did not" + ); + assert!(a.len() < c.len(), "the stalled leg withheld nothing"); +} + +/// A leg that holds a group of a sparse lane only in part, its first unit unplaced, is unsure +/// of the lane from there until a group it holds whole: it withholds the runs the rest of the +/// group takes on a leg that holds it, rather than send them as stuffing. +#[tokio::test(start_paused = true)] +async fn a_group_held_in_part_leaves_its_lane_unsure() { + let epoch = moq_net::Epoch::mint(); + let mut paths = Vec::new(); + for malformed in [0..0, 5 * GOP..6 * GOP] { + let mut live = Live::new(1.0, 0).await; + live.sections(malformed); + live.rtp(&epoch).await; + paths.push(live); + } + let [mut whole, mut partial] = <[Live; 2]>::try_from(paths).ok().unwrap(); + let mut c = whole.join(Duration::ZERO).await; + let mut a = partial.join(Duration::ZERO).await; + for tick in 1..=12 * GOP { + whole.run(tick, &mut [&mut c], |_| false).await; + partial.run(tick, &mut [&mut a], |_| false).await; + } + whole.finish(&mut [&mut c]).await; + partial.finish(&mut [&mut a]).await; + let (a, c) = (datagrams(&a, 0), datagrams(&c, 0)); + assert_eq!( + same_datagrams(&a, &c), + a.len(), + "the partial leg sent a run the whole leg did not" + ); + assert!(a.len() < c.len(), "the partial leg withheld nothing"); +} + +/// A lane with nothing more to carry on any leg withholds nothing: one whose source ended +/// while the broadcast goes on, and a sparse one (SCTE-35) that never sent, whose silence no +/// leg can tell from a stall. +#[tokio::test(start_paused = true)] +async fn an_ended_or_silent_sparse_lane_withholds_nothing() { + let epoch = moq_net::Epoch::mint(); + let mut live = Live::new(1.0, 0).await; + // Room for the keyframe beside teletext's lane. + live.rate = 2 * MUX_RATE; + live.teletext(0..0); + let _silent = live.section_track(); + live.rtp(&epoch).await; + let mut leg = live.join(Duration::ZERO).await; + live.run(6 * GOP, &mut [&mut leg], |_| false).await; + let (teletext, _) = live.teletext.take().unwrap(); + teletext.finish().unwrap(); + live.run(12 * GOP, &mut [&mut leg], |_| false).await; + let seqs: Vec = datagrams(&leg, 8 * GOP * VIDEO_US) + .into_keys() + .map(|(_, seq)| seq) + .collect(); + assert!(seqs.len() > 300, "{} datagrams", seqs.len()); + let withheld = seqs.windows(2).filter(|w| w[1] != w[0].wrapping_add(1)).count(); + assert_eq!(withheld, 0, "the leg withheld runs"); +} + +/// An epochless route names no publisher instance for the SSRC to follow, so RTP refuses it. +#[tokio::test(start_paused = true)] +#[should_panic(expected = "needs a route with an epoch")] +async fn rtp_refuses_an_epochless_route() { + let mut live = Live::new(1.0, 0).await; + live.sync = true; + live.rtp = true; + let mut leg = live.join(Duration::ZERO).await; + live.run(2 * GOP, &mut [&mut leg], |_| false).await; +} + +/// Without a delay no clock paces the datagrams, so RTP refuses it rather than send each slot +/// in one burst. +#[tokio::test(start_paused = true)] +#[should_panic(expected = "needs a --delay")] +async fn rtp_refuses_a_zero_delay() { + let mut live = Live::new(1.0, 0).await; + live.rtp(&moq_net::Epoch::mint()).await; + let mut leg = live.join(Duration::ZERO).await; + leg.export = leg.export.with_delay(Duration::ZERO); + live.run(2 * GOP, &mut [&mut leg], |_| false).await; +} + +/// A restart that opens a new generation while the old one's last datagrams are still paced out +/// sends them before adopting it, and the slots it flushes then keep the old clock's pace. With +/// the video ended, the old timeline's last slots hold runs the leg is sure of beside the audio +/// lane the restart leaves unknown. +#[tokio::test(start_paused = true)] +async fn an_rtp_leg_sends_its_paced_tail_before_a_restart() { + let epoch = moq_net::Epoch::mint(); + let restart = 4_000_000; + // Past the minute a restarted grid runs on across ([`super::lane::Grid::index`]), so the new + // timeline lays no slots between the two. + let by = 120_000_000; + for lag in LAGS { + let mut live = Live::new(1.0, lag).await; + // Room for runs clear of the audio lane in every slot. + live.rate = 4 * MUX_RATE; + live.rtp(&epoch).await; + let mut leg = live.join(Duration::ZERO).await; + live.run(4 * GOP, &mut [&mut leg], |_| false).await; + live.end_video(); + live.run_audio(restart, &mut [&mut leg]).await; + live.restart_audio(by); + live.run_audio(restart + 2_000_000, &mut [&mut leg]).await; + assert_eq!( + leg.export.discontinuity(), + 1, + "lag {lag}: the restart opened no generation" + ); + let last = datagrams(&leg, 0) + .into_keys() + .map(|(at, _)| at) + .filter(|at| *at < by) + .max(); + let slot = PCR_INTERVAL.as_micros() as u64; + assert!( + last >= Some(restart - slot), + "lag {lag}: the old timeline's datagrams stop at {last:?} µs, short of {restart} µs" + ); + // Each slot's datagrams go out across the slot after it, so no instant sends two slots'. + let mut sent: std::collections::BTreeMap> = Default::default(); + for (at, frame) in &leg.out { + let slot = frame.timestamp.as_micros() as u64; + if (2_000_000..by).contains(&slot) { + sent.entry(*at).or_default().push(slot); + } + } + for slots in sent.values() { + assert!( + slots.len() <= 2 && slots.iter().all(|slot| *slot == slots[0]), + "lag {lag}: old datagrams went out at once: {slots:?}" + ); + } + } +} + +/// Once every source and the catalog finished, an RTP leg lays on past the last slot until the +/// run it ends inside is whole, wherever on a datagram's seven packets that end falls, and sends +/// it: the same datagram a later joiner sends, holding what a plain synced leg rendered there. +#[tokio::test(start_paused = true)] +async fn a_clean_end_sends_the_run_the_timeline_ends_inside() { + let epoch = moq_net::Epoch::mint(); + let mut phases = std::collections::BTreeSet::new(); + for end in 6 * GOP..7 * GOP { + let mut live = Live::new(1.0, 0).await; + live.rtp(&epoch).await; + let mut a = live.join(Duration::ZERO).await; + live.rtp = false; + let mut plain = live.join(Duration::ZERO).await; + live.rtp = true; + live.run(JOIN, &mut [&mut a, &mut plain], |_| false).await; + let mut b = live.join(Duration::from_millis(100)).await; + live.run(end, &mut [&mut a, &mut plain, &mut b], |_| false).await; + live.catalog.finish().unwrap(); + live.finish(&mut [&mut a, &mut plain, &mut b]).await; + + // The plain leg's timeline, packet by packet: 34 to a slot at this rate. + let slot = PCR_INTERVAL.as_micros() as u64; + let timeline: std::collections::BTreeMap = plain + .out + .iter() + .flat_map(|(_, frame)| { + let first = frame.timestamp.as_micros() as u64 / slot * 34; + (first..).zip(frame.payload.chunks(188)) + }) + .collect(); + let media = |packet: &[u8]| matches!(pid(packet), VIDEO_PID | AUDIO_PID); + let last = *timeline.keys().next_back().unwrap(); + phases.insert(last % 7); + + let (x, y) = (datagrams(&a, 0), datagrams(&b, 0)); + let run = (last / 7) as u16; + let find = |sent: &std::collections::BTreeMap<(u64, u16), Bytes>| { + sent.values().find(|datagram| seq(datagram) == run).cloned() + }; + let tail = find(&x).unwrap_or_else(|| panic!("end {end}: the run holding the last packet {last} was withheld")); + for (position, packet) in (u64::from(run) * 7..).zip(tail[12..].chunks(188)) { + match timeline.get(&position) { + Some(rendered) => assert_eq!(packet, *rendered, "end {end}: position {position} differs"), + None => assert!( + !media(packet), + "end {end}: position {position} holds media past the end" + ), + } + } + assert_eq!(find(&y), Some(tail), "end {end}: the joiner sent another last run"); + assert!(same_datagrams(&x, &y) > 0); + } + assert_eq!( + phases.len(), + 7, + "the timeline ended at too few places in a run: {phases:?}" + ); +} + +fn seq(datagram: &[u8]) -> u16 { + u16::from_be_bytes([datagram[2], datagram[3]]) +} + /// A group ending in a burst still fills its lane at the next group's release, so a leg holding /// it would lay the next group behind where a leg that lost it does. When the next group's /// first picture opens a buffering period, it goes out late enough for neither leg's lane to be @@ -1270,7 +1913,7 @@ async fn synced_legs_render_a_drifting_source_the_same_way() { #[should_panic(expected = "--sync needs a constant multiplex rate")] async fn sync_refuses_an_output_without_a_constant_rate() { let mut live = Live::new(1.0, 0).await; - let export = super::export_test::open_export(live.source.clone()) + let export = super::export_test::open_ts(live.source.clone(), crate::catalog::CatalogFormat::Hang) .await .unwrap() .with_delay(DELAY) diff --git a/rs/moq-mux/src/container/ts/lane.rs b/rs/moq-mux/src/container/ts/lane.rs index a70ca46f61d..0073044ec25 100644 --- a/rs/moq-mux/src/container/ts/lane.rs +++ b/rs/moq-mux/src/container/ts/lane.rs @@ -17,7 +17,7 @@ //! opens an H.264 buffering period: the export then releases it late enough for its HRD to rule //! that out. -use std::collections::{BTreeMap, HashMap, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}; use std::sync::Arc; use mpeg2ts::ts::TsPacket; @@ -279,6 +279,11 @@ pub(super) struct Lane { group: Option, /// One past the last position any group took. end: u128, + /// The first position of the first group this leg held whole since it joined or the lane + /// last [jumped](Self::jump): `None` until it holds one. + from: Option, + /// The lane carries units too seldom for a leg to tell a stall from silence ([`Self::known`]). + sparse: bool, /// Groups that began behind the one before them. overruns: u64, } @@ -294,11 +299,14 @@ struct Group { } impl Lane { - pub fn new(lane: usize) -> Self { + /// Lane `lane` of its [`Layout`], `sparse` when it has no cadence to expect a unit by. + pub fn new(lane: usize, sparse: bool) -> Self { Self { lane, group: None, end: 0, + from: sparse.then_some(0), + sparse, overruns: 0, } } @@ -313,10 +321,30 @@ impl Lane { self.overruns } + /// The positions where this leg holds what every leg holds ([`Known`]). + /// + /// A position is known once the lane has gone past it, its next unit planned beyond it, or + /// its source `ended`, from the first group held whole since the leg joined or the lane last + /// jumped. A sparse lane is known from the join without waiting for its next unit, which + /// may never come: a unit of it lost with no later jump goes out as stuffing where another + /// leg has it. + pub fn known(&self, ended: bool) -> Known { + Known { + from: self.from.unwrap_or(u128::MAX), + to: (!ended && !self.sparse).then_some(self.end), + } + } + + /// The playhead jumped, so this leg may lack units of the lane until a group it holds whole. + pub fn jump(&mut self) { + self.from = None; + } + /// Forget every group, for a timeline that starts again. pub fn reset(&mut self) { self.group = None; self.end = 0; + self.from = None; } /// Whether a unit of group `sequence`, its first when `opens`, would be the first its group @@ -356,6 +384,7 @@ impl Lane { sent: 0, whole: false, }); + self.from = None; return Err(Unplaced::Partial); } _ => { @@ -381,6 +410,7 @@ impl Lane { group.next = last + 1; group.sent += count; self.end = self.end.max(last + 1); + self.from.get_or_insert(positions[0]); Ok(Placed { positions, counter }) } } @@ -407,6 +437,26 @@ pub(super) struct Laid { pub units: Vec, /// It does not follow the slot laid before it. pub jump: bool, + /// The lane positions this leg leaves empty where another leg may carry a packet. + pub unsure: Vec, +} + +/// The positions of a lane where this leg holds every unit the lane carries, so a position it +/// leaves empty there is empty on every leg: from `from`, up to `to` when the lane may still +/// carry units this leg has not read ([`Lane::known`]). +#[derive(Clone, Copy, Debug)] +pub(super) struct Known { + pub from: u128, + pub to: Option, +} + +impl Known { + /// Every position, until the export says otherwise. + pub const ALL: Self = Self { from: 0, to: None }; + + fn covers(&self, at: u128) -> bool { + at >= self.from && self.to.is_none_or(|to| at < to) + } } /// The packets placed on a [`Layout`], laid out slot by slot. @@ -429,10 +479,15 @@ pub(super) struct Grid { last: Option, /// Packets that reached the grid after their position went out. late: u64, + /// What this leg knows of each lane's positions ([`Self::know`]). + known: Vec, + /// Positions planned for a unit that has not reached them yet ([`Self::expect`]). + expected: BTreeSet, } impl Grid { pub fn new(layout: Arc, tables: Vec<[u8; TsPacket::SIZE]>) -> Self { + let known = vec![Known::ALL; layout.lanes.len()]; Self { layout, placed: BTreeMap::new(), @@ -443,9 +498,23 @@ impl Grid { next: None, last: None, late: 0, + known, + expected: BTreeSet::new(), } } + /// What this leg knows of lane `lane`'s positions, until told otherwise. + pub fn know(&mut self, lane: usize, known: Known) { + self.known[lane] = known; + } + + /// Expect a planned unit's packets at `positions`. A unit that never reaches them (dropped as + /// late or leading) leaves them empty here and full on every leg holding it. + pub fn expect(&mut self, positions: &[u128]) { + let gone = self.next.map_or(0, |next| self.layout.first(next)); + self.expected.extend(positions.iter().filter(|&&at| at >= gone)); + } + pub fn layout(&self) -> &Arc { &self.layout } @@ -519,6 +588,7 @@ impl Grid { self.placed.clear(); self.starts.clear(); self.si.clear(); + self.expected.clear(); } /// Start the grid afresh at the next slot holding a packet, for a timeline that restarted. @@ -535,6 +605,10 @@ impl Grid { /// Tables that start a timeline number as the last repetition before them did, so a /// receiver tuning in on them counts on into the next; after a jump they flag the break /// from the repetition the stream carried last. + /// + /// A lane position left empty is [unsure](Laid::unsure) where it was planned for a unit + /// that never reached it (the jitter buffer dropped it), or lies outside what this leg + /// [knows](Self::know) of the lane. pub fn lay(&mut self, index: u128) -> Laid { self.next = Some(index + 1); let jump = self.last.is_some_and(|last| index != last + 1); @@ -553,6 +627,7 @@ impl Grid { keyframe: false, units: Vec::new(), jump, + unsure: Vec::new(), }; let emission = index / PSI_SLOTS; let spacing = self.layout.spacing as u128; @@ -574,7 +649,14 @@ impl Grid { } packet }), - Owner::Lane(_) => self.placed.remove(&at), + Owner::Lane(lane) => { + let packet = self.placed.remove(&at); + let expected = self.expected.remove(&at); + if packet.is_none() && (expected || !self.known[lane].covers(at)) { + out.unsure.push(position); + } + packet + } Owner::Free => { let ready = self.si.front().is_some_and(|(from, packet)| { *from <= at && self.si_last.get(&pid(packet)).is_none_or(|last| *last + spacing <= at) @@ -600,6 +682,7 @@ impl Grid { let end = first + count as u128; self.placed = self.placed.split_off(&end); self.starts = self.starts.split_off(&end); + self.expected = self.expected.split_off(&end); out } } @@ -760,8 +843,8 @@ mod test { fn a_group_lays_from_its_own_anchor_whatever_came_before() { let layout = layout(); let by = 10_000_000_000; - let mut whole = Lane::new(0); - let mut joiner = Lane::new(0); + let mut whole = Lane::new(0, false); + let mut joiner = Lane::new(0, false); // Group 1, which the joiner lacks. whole.plan(&layout, (1, true), start(100), by, 50).unwrap(); whole.plan(&layout, (1, false), start(101), by, 5).unwrap(); @@ -781,8 +864,8 @@ mod test { let layout = layout(); // 48 kHz AAC: frames 21.3 ms apart, two to a slot now and then. let frame = 1_000_000_000 * 1024 / 48_000; - let mut whole = Lane::new(1); - let mut joiner = Lane::new(1); + let mut whole = Lane::new(1, false); + let mut joiner = Lane::new(1, false); let last = start(100) + 2_000_000; whole.plan(&layout, (1, true), last - frame, u128::MAX, 2).unwrap(); whole.plan(&layout, (1, false), last, u128::MAX, 2).unwrap(); @@ -796,7 +879,7 @@ mod test { #[test] fn a_group_held_from_its_middle_is_not_placed() { let layout = layout(); - let mut lane = Lane::new(0); + let mut lane = Lane::new(0, false); assert_eq!( lane.plan(&layout, (1, false), start(10), u128::MAX, 2).err(), Some(Unplaced::Partial) @@ -805,13 +888,23 @@ mod test { lane.plan(&layout, (1, false), start(10), u128::MAX, 2).err(), Some(Unplaced::Partial) ); - assert!(lane.plan(&layout, (2, true), start(30), u128::MAX, 2).is_ok()); + assert_eq!(lane.known(false).from, u128::MAX, "known only from a group held whole"); + let placed = lane.plan(&layout, (2, true), start(30), u128::MAX, 2).unwrap(); + let known = lane.known(false); + assert_eq!( + (known.from, known.to), + (placed.positions[0], Some(placed.positions[1] + 1)) + ); + assert_eq!(lane.known(true).to, None, "an ended lane is known past its end"); + // A group held in part is a jump: the lane is unknown again until the next held whole. + assert!(lane.plan(&layout, (3, false), start(40), u128::MAX, 2).is_err()); + assert_eq!(lane.known(false).from, u128::MAX); } #[test] fn a_late_unit_leaves_its_positions_to_the_next() { let layout = layout(); - let mut lane = Lane::new(1); + let mut lane = Lane::new(1, false); let at = layout.time(layout.first(11)); assert_eq!( lane.plan(&layout, (1, true), start(10), at, 30).err(), diff --git a/rs/moq-mux/src/container/ts/mod.rs b/rs/moq-mux/src/container/ts/mod.rs index 94a424abac5..43680c91466 100644 --- a/rs/moq-mux/src/container/ts/mod.rs +++ b/rs/moq-mux/src/container/ts/mod.rs @@ -34,6 +34,7 @@ mod mux_rate; mod programs; mod psi; mod release; +mod rtp; mod schedule; mod si; diff --git a/rs/moq-mux/src/container/ts/rtp.rs b/rs/moq-mux/src/container/ts/rtp.rs new file mode 100644 index 00000000000..845a8a39f90 --- /dev/null +++ b/rs/moq-mux/src/container/ts/rtp.rs @@ -0,0 +1,282 @@ +//! MPEG-TS over RTP (RFC 2250) for [`Export::with_rtp`](super::Export::with_rtp). +//! +//! Each datagram carries the [`PACKETS`] TS packets of one run of the synced multiplex's +//! timeline, and every header field is a function of that run and the broadcast: the sequence +//! number is the run's index and the timestamp its first packet's time on the PCR at 90 kHz, +//! each modulo its field's width, and the SSRC a hash of the broadcast's path and epoch. So two +//! legs of one broadcast send the same datagram, headers included, for every run both render +//! alike, and a receiver merging them (SMPTE ST 2022-7) can take each from either. A leg sends +//! only the runs it is sure of: past the slot it joined or last jumped at, whose clock and +//! tables flag the break where a leg laying it in turn does not, and holding no lane position +//! another leg may fill ([`Laid::unsure`](super::lane::Laid::unsure)). The rest are withheld, +//! so another leg's copy is the one that arrives. A run the media ends inside is sent whole once +//! every source finished: past that end every leg lays the same slots, so the export lays on +//! until the run is complete. + +use std::sync::Arc; + +use bytes::Bytes; +use mpeg2ts::ts::TsPacket; + +use super::lane::Layout; + +/// TS packets a datagram carries: the most a 1500-byte MTU takes (SMPTE ST 2022-2). +pub(super) const PACKETS: usize = 7; + +/// RTP's static payload type for MPEG-2 TS (RFC 3551). +const PAYLOAD_TYPE: u8 = 33; + +/// Version 2, no padding, extension or contributing sources (RFC 3550 5.1). +const VERSION: u8 = 0x80; + +const HEADER: usize = 12; + +/// Cuts a synced export's slots into RTP datagrams. +pub(super) struct Packetizer { + layout: Arc, + ssrc: u32, + /// The slot expected next: any other is a jump. + next: Option, + /// The first position a run may start at: past the slot this leg joined or last jumped at. + from: u128, + run: Option, +} + +/// A datagram in progress. +struct Run { + index: u128, + packets: Vec, + /// No position in it is one another leg may fill otherwise. + sure: bool, +} + +impl Packetizer { + pub fn new(layout: Arc, ssrc: u32) -> Self { + Self { + layout, + ssrc, + next: None, + from: 0, + run: None, + } + } + + /// The datagrams slot `index`'s `payload` completes and this leg is sure of, each with the + /// position in the slot of its run's last packet. `unsure` lists the slot's positions + /// another leg may fill otherwise. + pub fn push(&mut self, index: u128, payload: &[u8], unsure: &[usize]) -> Vec<(usize, Bytes)> { + if self.next != Some(index) { + self.run = None; + self.from = self.layout.first(index + 1); + } + self.next = Some(index + 1); + let first = self.layout.first(index); + let packets = payload.as_chunks::<{ TsPacket::SIZE }>().0; + let mut sure = vec![true; packets.len()]; + for &position in unsure.iter().filter(|&&position| position < packets.len()) { + sure[position] = false; + } + let mut out = Vec::new(); + for (position, packet) in packets.iter().enumerate() { + let at = first + position as u128; + let (index, offset) = (at / PACKETS as u128, at % PACKETS as u128); + if offset == 0 { + self.run = (at >= self.from).then(|| Run { + index, + packets: Vec::with_capacity(PACKETS * TsPacket::SIZE), + sure: true, + }); + } + let Some(run) = self.run.as_mut() else { + continue; + }; + run.packets.extend_from_slice(packet); + run.sure &= sure[position]; + if offset + 1 == PACKETS as u128 + && let Some(run) = self.run.take() + && run.sure + { + out.push((position, self.datagram(run))); + } + } + out + } + + /// The run the slots so far leave open, while this leg is sure of it. + pub fn open(&self) -> Option { + self.run.as_ref().filter(|run| run.sure).map(|run| run.index) + } + + /// Drop the run in progress: what fills the rest of it is no longer alike on every leg. + pub fn close(&mut self) { + self.run = None; + } + + fn datagram(&self, run: Run) -> Bytes { + let mut out = Vec::with_capacity(HEADER + run.packets.len()); + out.extend_from_slice(&[VERSION, PAYLOAD_TYPE]); + out.extend_from_slice(&(run.index as u16).to_be_bytes()); + out.extend_from_slice(×tamp(&self.layout, run.index).to_be_bytes()); + out.extend_from_slice(&self.ssrc.to_be_bytes()); + out.extend_from_slice(&run.packets); + Bytes::from(out) + } +} + +/// Run `index`'s RTP timestamp: when its first packet starts on the PCR, at 90 kHz, modulo +/// 2^32, as RFC 2250 stamps the target transmission time of a datagram's first byte. +fn timestamp(layout: &Layout, index: u128) -> u32 { + (layout.time(index * PACKETS as u128) * 9 / 100_000) as u32 +} + +/// The SSRC of the broadcast at `path` from publisher instance `epoch`: FNV-1a over both, so +/// every leg derives the same one, and a replacement publisher or another path, barring a +/// 32-bit collision, another. The collision resolution of RFC 3550 8.2 is not applied: it +/// would have each leg pick a new SSRC of its own. +pub(super) fn ssrc(path: &str, epoch: &moq_net::Epoch) -> u32 { + let mut hash: u32 = 0x811c_9dc5; + for byte in path.bytes().chain([0]).chain(epoch.as_str().bytes()) { + hash ^= u32::from(byte); + hash = hash.wrapping_mul(0x0100_0193); + } + hash +} + +#[cfg(test)] +mod test { + use super::super::lane::Need; + use super::*; + + /// 34 packets a slot, so runs straddle slots. + const RATE: u64 = 34 * 188 * 8 * 40; + + fn layout() -> Arc { + Arc::new(Layout::new(RATE, 2, 0, &[(0x100, Need::Rest(u64::MAX))]).unwrap()) + } + + /// Slot `index`'s packets, each carrying its position on the timeline. + fn slot(layout: &Layout, index: u128) -> Vec { + let first = layout.first(index); + (0..layout.count(index) as u128) + .flat_map(|n| { + let mut packet = [0xffu8; TsPacket::SIZE]; + packet[0] = 0x47; + packet[4..20].copy_from_slice(&(first + n).to_be_bytes()); + packet + }) + .collect() + } + + fn positions(datagram: &[u8]) -> Vec { + datagram[HEADER..] + .chunks(TsPacket::SIZE) + .map(|packet| u128::from_be_bytes(packet[4..20].try_into().unwrap())) + .collect() + } + + fn seq(datagram: &[u8]) -> u16 { + u16::from_be_bytes([datagram[2], datagram[3]]) + } + + #[test] + fn a_datagram_carries_one_run_of_the_timeline() { + let layout = layout(); + let mut rtp = Packetizer::new(layout.clone(), 0x1234_5678); + let mut sent = Vec::new(); + for index in 0..4 { + let first = layout.first(index); + sent.extend( + rtp.push(index, &slot(&layout, index), &[]) + .into_iter() + .map(|(last, datagram)| (first + last as u128, datagram)), + ); + } + // Past the join slot, 102 packets: fourteen whole runs from run 5, and a fifteenth the + // next slot completes. + assert_eq!(sent.len(), 14); + for (n, (last, datagram)) in (5..).zip(&sent) { + assert_eq!(datagram.len(), HEADER + PACKETS * TsPacket::SIZE); + assert_eq!(&datagram[..2], [0x80, 33]); + assert_eq!(seq(datagram), n as u16); + assert_eq!(&datagram[8..12], 0x1234_5678u32.to_be_bytes()); + let first = (n * PACKETS) as u128; + assert_eq!( + positions(datagram), + (first..first + PACKETS as u128).collect::>() + ); + assert_eq!(*last, first + PACKETS as u128 - 1); + let at = u32::from_be_bytes(datagram[4..8].try_into().unwrap()); + assert_eq!(u128::from(at), layout.time(first) * 9 / 100_000); + } + } + + /// The slot a leg joins or jumps at flags the break in its clock and tables where a leg + /// laying it in turn does not, so no run holding a position of it is sent. + #[test] + fn no_run_of_a_join_or_jump_slot_is_sent() { + let layout = layout(); + let mut rtp = Packetizer::new(layout.clone(), 1); + // Slot 1 holds positions 34 to 67; run 9 (63 to 69) runs into slot 2. + assert!(rtp.push(1, &slot(&layout, 1), &[]).is_empty()); + let sent = rtp.push(2, &slot(&layout, 2), &[]); + assert_eq!(sent.iter().map(|(_, d)| seq(d)).collect::>(), [10, 11, 12, 13]); + // Slot 4 jumps over slot 3: run 14 (98 to 104), cut short, is not sent either. + assert!(rtp.push(4, &slot(&layout, 4), &[]).is_empty()); + let sent = rtp.push(5, &slot(&layout, 5), &[]); + assert_eq!(sent.iter().map(|(_, d)| seq(d)).collect::>(), [25, 26, 27, 28]); + } + + #[test] + fn a_run_holding_an_unsure_position_is_withheld() { + let layout = layout(); + let mut rtp = Packetizer::new(layout.clone(), 1); + rtp.push(0, &slot(&layout, 0), &[]); + // Positions 35 and 49, in runs 5 and 7. + let sent = rtp.push(1, &slot(&layout, 1), &[1, 15]); + assert_eq!(sent.iter().map(|(_, d)| seq(d)).collect::>(), [6, 8]); + // Position 68 opens slot 2, so its doubt reaches back into run 9. + let sent = rtp.push(2, &slot(&layout, 2), &[0]); + assert_eq!(sent.iter().map(|(_, d)| seq(d)).collect::>(), [10, 11, 12, 13]); + } + + #[test] + fn sequence_numbers_and_timestamps_wrap_at_their_widths() { + let layout = layout(); + let mut rtp = Packetizer::new(layout.clone(), 1); + // A slot whose runs cross a sequence wrap, and one whose time crosses a timestamp wrap. + let seq_wrap = layout.slot_of(65_536 * PACKETS as u128); + let seqs: Vec = (seq_wrap - 1..=seq_wrap) + .flat_map(|index| rtp.push(index, &slot(&layout, index), &[])) + .map(|(_, d)| seq(&d)) + .collect(); + assert!(seqs.windows(2).all(|w| w[1] == w[0].wrapping_add(1)), "{seqs:?}"); + assert!(seqs.contains(&0) && seqs.contains(&u16::MAX), "{seqs:?}"); + // 2^32 ticks at 90 kHz, in slots of 25 ms, which the PCR reaches a slot later. + let ts_wrap = (1u128 << 32) * 1_000 / 90 / 25; + let mut stamps = Vec::new(); + for index in ts_wrap - 2..ts_wrap + 3 { + let sent = rtp.push(index, &slot(&layout, index), &[]); + stamps.extend( + sent.iter() + .map(|(_, d)| u32::from_be_bytes(d[4..8].try_into().unwrap())), + ); + } + let steps: Vec = stamps.windows(2).map(|w| w[1].wrapping_sub(w[0])).collect(); + // Seven packets at the rate take 5.15 ms. + assert!(steps.iter().all(|step| (463..=464).contains(step)), "{steps:?}"); + assert!( + stamps.windows(2).any(|w| w[1] < w[0]), + "the timestamp wraps: {stamps:?}" + ); + } + + #[test] + fn the_ssrc_is_a_function_of_path_and_epoch() { + let epoch: moq_net::Epoch = "0199f0c2-0000-7000-8000-000000000000".parse().unwrap(); + let other: moq_net::Epoch = "0199f0c2-0000-7000-8000-000000000001".parse().unwrap(); + // Pinned, so legs built from different releases still agree. + assert_eq!(ssrc("live/one", &epoch), 0x04a6_c211); + assert_ne!(ssrc("live/one", &epoch), ssrc("live/one", &other)); + assert_ne!(ssrc("live/one", &epoch), ssrc("live/two", &epoch)); + } +} diff --git a/rs/moq-mux/src/container/ts/schedule.rs b/rs/moq-mux/src/container/ts/schedule.rs index d9f28b5d0fa..c1c23382f16 100644 --- a/rs/moq-mux/src/container/ts/schedule.rs +++ b/rs/moq-mux/src/container/ts/schedule.rs @@ -395,6 +395,9 @@ pub(super) struct Slot { pub clocked: bool, /// The slot does not follow the one laid before it, so its PCR flags the break. pub jump: bool, + /// Under sync, the positions, counting the opening clock, another leg may fill otherwise + /// ([`Laid::unsure`](super::lane::Laid::unsure)). + pub unsure: Vec, } impl Slot { @@ -919,6 +922,7 @@ impl Schedule { positions: Some(laid.positions), clocked: false, jump: laid.jump, + unsure: laid.unsure, })); } let Some(index) = self.index(known) else { @@ -976,6 +980,7 @@ impl Schedule { positions: None, clocked: false, jump: false, + unsure: Vec::new(), })) } @@ -998,6 +1003,7 @@ impl Schedule { positions: Some(Vec::new()), clocked: false, jump: false, + unsure: Vec::new(), }; let clock = self.clock.filter(|pid| self.buffers.contains_key(pid)); // A new multiplex rate changes Rsys from this slot on, and Bsys keeps the bytes it holds. @@ -2665,6 +2671,7 @@ mod tests { positions: None, clocked: false, jump: false, + unsure: Vec::new(), }; let mut null = unit(0, 1); (null[1], null[2]) = (0x1f, 0xff); @@ -2704,6 +2711,7 @@ mod tests { positions: None, clocked: false, jump: false, + unsure: Vec::new(), }; let mut null = unit(0, 1); (null[1], null[2]) = (0x1f, 0xff); diff --git a/rs/moq-mux/src/jitter.rs b/rs/moq-mux/src/jitter.rs index b3d6bbc27a3..0ca9078ed82 100644 --- a/rs/moq-mux/src/jitter.rs +++ b/rs/moq-mux/src/jitter.rs @@ -137,9 +137,11 @@ pub(crate) struct Buffer { replay: bool, /// The clock every frame is pushed under. clock: Option, - /// The generation before it, for a track the restart has not reached yet, until the - /// current generation's first frame goes out. + /// The generation before it, which times that generation's tail ([`Self::at`]) and, until + /// [`Self::crossed`], takes the frames of a track the restart has not reached yet. previous: Option, + /// The current generation's first frame went out, so the previous one takes no more frames. + crossed: bool, /// The next generation's frames, held while its clock is acquired. acquire: Option>, tracks: BTreeMap>, @@ -351,6 +353,7 @@ impl Buffer { replay: false, clock: None, previous: None, + crossed: false, acquire: None, tracks: BTreeMap::new(), horizon: None, @@ -435,7 +438,7 @@ impl Buffer { /// on that clock, and due no later than the current generation's first frame, so the /// generations still go out in turn. fn lagging(&self, key: &K, arrival: &Arrival) -> Option { - let previous = self.previous?; + let previous = self.previous.filter(|_| !self.crossed)?; let track = self.tracks.get(key)?; let deadline = self.deadline(&previous, arrival.decode); let lags = track.generation == previous.generation @@ -535,6 +538,7 @@ impl Buffer { clock.anchor += horizon - kept; } self.previous = self.clock.replace(clock); + self.crossed = false; self.steer = Steer::default(); for (key, frame) in acquire.frames { let (deadline, stale) = self.judge(&clock, &frame, acquiring); @@ -890,7 +894,7 @@ impl Buffer { self.released = true; // A previous-generation frame queued now would go out after this one. if self.clock.is_some_and(|clock| clock.generation == generation) { - self.previous = None; + self.crossed = true; } return Poll::Ready(Ready { track: key.clone(), diff --git a/rs/moq-mux/src/source.rs b/rs/moq-mux/src/source.rs index c64a28c44d0..d6a568148b9 100644 --- a/rs/moq-mux/src/source.rs +++ b/rs/moq-mux/src/source.rs @@ -464,8 +464,14 @@ pub(crate) async fn resolved(origin: &moq_net::origin::Producer, path: &str) -> /// broadcast stays reachable for the source's lifetime (harmless in a test binary). #[cfg(test)] pub(crate) async fn announced(broadcast: &moq_net::broadcast::Consumer) -> Source { + announced_on(broadcast, Default::default()).await +} + +/// Test helper: as [`announced`], served on `route`. +#[cfg(test)] +pub(crate) async fn announced_on(broadcast: &moq_net::broadcast::Consumer, route: moq_net::origin::Route) -> Source { let origin = produce_origin(); - let dynamic = origin.dynamic_route("", Default::default()).unwrap(); + let dynamic = origin.dynamic_route("", route).unwrap(); let served = broadcast.clone(); tokio::spawn(async move { while let Ok(request) = dynamic.requested_broadcast().await {