Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions quest/m1/ts-reliability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,7 @@ refused means an error or a counted drop.
Decided in the 2026-10-22 root-cause audit: the rows above kept getting
patched one at a time because two foundations were missing. Decode time is
reconstructed from history instead of decode order, and import state has no
owner that retires it. Those quests, and the shared parser that the routes
and PAT plans build on, rank ahead of the remaining rows. Rejected: carrying
owner that retires it. Those quests rank ahead of the remaining rows. Rejected: carrying
decode time in every frame container, one shared ingest byte budget,
and replacing the demux with GStreamer or another parser (it fixes neither
cause). Out of scope for now: decoupling catalog publication from mux-rate
Expand All @@ -101,7 +100,6 @@ remain with their current quests.

- [State bound](/quest/m1/ts-reliability/state-bound.md) - SI and revision state retire with what the source stops repeating
- [Decode order](/quest/m1/ts-reliability/decode-order.md) - TS, MKV and FLV author DTS from decode order and a declared `reorder` depth
- [Shared parser](/quest/m1/ts-reliability/shared-parser.md) - `--program all` scans the multiplex once
- [Access units](/quest/m1/ts-reliability/pes-access-units.md) - a PES carrying several access units publishes each at its own time
- [PAT changes](/quest/m1/ts-reliability/pat-lifecycle.md) - a revised PAT never leaves a broadcast meaning another program
- [Stream routes](/quest/m1/ts-reliability/stream-routes.md) - every PID gets a declared route, and what is not carried is counted
Expand Down
13 changes: 5 additions & 8 deletions quest/m1/ts-reliability/pat-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,11 @@ What the code does today:
- A PAT naming another PMT PID for the program adds a reassembler but never
drops the old one, and `program.pmtPid` keeps the first.
- `Programs` ignores a program a later PAT adds, without a warning.
- Under `--program all`, a later PAT that removes one program reaches every
per-program `Import`; the removed program's `handle_pat` errors,
`Programs::feed` propagates it, and the whole multiplex ingest stops,
including unrelated programs.
- Under `--program all`, the shared parser hands a later PAT that removes one
program to every program's `Import`; the removed program's `handle_pat`
errors, `Programs::decode` propagates it, and the whole multiplex ingest
stops, including unrelated programs. `Programs` sees each whole PAT first
(`Packet::pat`), so it can retire or start programs there instead.
- A PAT naming another `transport_stream_id`, or a single-program import's
PAT naming another program, already refuses the import.

Expand All @@ -42,7 +43,3 @@ Tests use the generated `fixture::Mux` builder: a PMT PID move, and a
`--program all` MPTS whose PAT adds a program, drops one and later restores
it: the other program runs uninterrupted, and the returning one publishes a
new epoch.

## Required

- [Shared parser](/quest/m1/ts-reliability/shared-parser.md) - one parser owns the PID map this plan's accounting needs
1 change: 0 additions & 1 deletion quest/m1/ts-reliability/program-cap.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,3 @@ Public API: none. Wire: none.

- [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - the TS import contract this belongs to
- [PMT lifecycle](/quest/m1/ts-reliability/pat-lifecycle.md) - the PAT handling beside it
- [Shared TS parser](/quest/m1/ts-reliability/shared-parser.md) - one parser for every program, so per-chunk cost does not grow with the programs this quest admits
30 changes: 0 additions & 30 deletions quest/m1/ts-reliability/shared-parser.md

This file was deleted.

20 changes: 8 additions & 12 deletions quest/m1/ts-reliability/stream-routes.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@ program's PMT arrives. A listed PMT that does not arrive within the 0.5 s of
program clock TR 101 290 allows (`PMT_error_2`, already counted) stops
holding the gate, so a stale or corrupt program never disables the count.

Under `--program all`, every per-program `Import` reads the whole multiplex
and `Programs::stats` merges their rows by PID, so a per-program drop row
would shadow the program that carries the PID. `Programs` owns the
Under `--program all`, one parser reads the multiplex and `Programs` routes
each PID only to the programs whose PAT entry or PMT claims it
(`Programs::readers`), while `Programs::stats` merges their rows by PID, so a
per-program drop row would shadow the program that carries the PID. `Programs` owns the
unclaimed-PID rows instead: a PID is dropped only when no program's PMT and
no captured SI claims it, counted once at the multiplex rather than per
program. A `--program N` import of an MPTS reassembles only its own PMT, so
Expand All @@ -70,11 +71,10 @@ own program's PIDs and leaves multiplex-wide accounting to `--program all`.
When the PAT lists only the selected program, it counts as above. Chosen
over passively acquiring every PMT, which adds a reassembler per program to
an import that serves one.
Classifying a PID looks it up in one map of every claimed PID, kept as
PMTs arrive, never by searching each program's PMT. A benchmark in
`rs/moq-mux/benches` feeds `Programs` a generated MPTS and sweeps program
count and PID count independently, against a baseline without the
accounting, so any slope it adds beyond the existing per-program feed shows.
Classifying a PID reads `Programs::readers`, the PID-indexed table of every
claimed PID kept as PMTs arrive: a PID no program reads is unclaimed. Extend
`rs/moq-mux/benches/ts_programs.rs` to sweep PID count beside program count
and bitrate, so any slope the accounting adds shows.

A dropped PID reuses `stats::Stream`, with no new field: a row keyed by its
PID in `Snapshot::streams`, an empty `track`, `Class::Data`, and `refused`
Expand Down Expand Up @@ -106,7 +106,3 @@ program 2's PIDs, asserting the catalog, the
stats rows (the unlisted PID's row asserted field by field), and an export
round trip of the carried streams. The MPEG-2 video arm also runs the TSDuck
round trip and an FFmpeg decode of the export in `test/ts`.

## Required

- [Shared parser](/quest/m1/ts-reliability/shared-parser.md) - one parser owns the PID map this plan's accounting needs
4 changes: 4 additions & 0 deletions rs/moq-mux/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ harness = false
name = "ts_schedule"
harness = false

[[bench]]
name = "ts_programs"
harness = false

[[bench]]
name = "export"
harness = false
Expand Down
209 changes: 209 additions & 0 deletions rs/moq-mux/benches/ts_programs.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
//! `--program all` TS ingest across program count and per-program bitrate, over one second of
//! a generated multiplex. Throughput is the whole input, so a parse that repeats per program
//! shows as a falling rate with program count.

use std::time::Duration;

use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};

const FPS: u64 = 25;
const GOP: u64 = 25;
/// PSI repeats every this many frames: 160 ms.
const PSI_EVERY: u64 = 4;
const SPS: &[u8] = &[
0x67, 0x42, 0xc0, 0x1f, 0xda, 0x01, 0x40, 0x16, 0xe9, 0xb8, 0x08, 0x08, 0x0a, 0x00, 0x00, 0x07, 0xd0, 0x00, 0x01,
0xd4, 0xc0, 0x80,
];
const PPS: &[u8] = &[0x68, 0xce, 0x3c, 0x80];

/// ISO/IEC 13818-1 CRC-32: polynomial 0x04C11DB7, no reflection, no final XOR.
fn crc32(data: &[u8]) -> u32 {
let mut crc = 0xffff_ffffu32;
for &byte in data {
crc ^= u32::from(byte) << 24;
for _ in 0..8 {
crc = if crc & 0x8000_0000 != 0 {
(crc << 1) ^ 0x04c1_1db7
} else {
crc << 1
};
}
}
crc
}

/// A long-form PSI section: `table_id`, the 16-bit extension, version 0, one section.
fn section(table_id: u8, extension: u16, body: &[u8]) -> Vec<u8> {
let length = 5 + body.len() + 4;
let mut section = vec![table_id, 0xb0 | (length >> 8) as u8, length as u8];
section.extend_from_slice(&extension.to_be_bytes());
section.extend_from_slice(&[0xc1, 0, 0]);
section.extend_from_slice(body);
section.extend_from_slice(&crc32(&section).to_be_bytes());
section
}

fn pmt_pid(program: u16) -> u16 {
0x0100 + program
}

fn video_pid(program: u16) -> u16 {
0x0200 + program
}

#[derive(Default)]
struct Mux {
out: Vec<u8>,
cc: std::collections::HashMap<u16, u8>,
}

impl Mux {
fn header(&mut self, pid: u16, start: bool, adaptation: bool) {
let cc = self.cc.entry(pid).or_default();
self.out.extend_from_slice(&[
0x47,
u8::from(start) << 6 | (pid >> 8) as u8,
pid as u8,
if adaptation { 0x30 } else { 0x10 } | *cc,
]);
*cc = (*cc + 1) & 0x0f;
}

fn psi(&mut self, pid: u16, section: &[u8]) {
assert!(section.len() < 184, "one packet per section");
self.header(pid, true, false);
let end = self.out.len() + 184;
self.out.push(0);
self.out.extend_from_slice(section);
self.out.resize(end, 0xff);
}

/// One access unit on `pid` in `packets` packets, its first carrying `pcr` (27 MHz) and a
/// PES header with PTS `pts` (90 kHz).
fn frame(&mut self, pid: u16, packets: usize, pcr: u64, pts: u64, keyframe: bool) {
let mut pes = vec![0, 0, 1, 0xe0, 0, 0, 0x80, 0x80, 5];
pes.extend_from_slice(&[
0x21 | ((pts >> 29) & 0x0e) as u8,
(pts >> 22) as u8,
0x01 | ((pts >> 14) & 0xfe) as u8,
(pts >> 7) as u8,
0x01 | ((pts << 1) & 0xfe) as u8,
]);
let nals: &[&[u8]] = match keyframe {
true => &[SPS, PPS, &[0x65, 0x88, 0x84, 0x21]],
false => &[&[0x41, 0x9a, 0x02, 0x04]],
};
for nal in nals {
pes.extend_from_slice(&[0, 0, 0, 1]);
pes.extend_from_slice(nal);
}
// The first packet gives 8 bytes to its PCR's adaptation field. Nonzero padding cannot
// add a start code.
pes.resize(packets * 184 - 8, 0xff);

self.header(pid, true, true);
let (base, ext) = (pcr / 300, pcr % 300);
self.out.extend_from_slice(&[
7,
0x10 | u8::from(keyframe) << 6,
(base >> 25) as u8,
(base >> 17) as u8,
(base >> 9) as u8,
(base >> 1) as u8,
((base & 1) << 7) as u8 | 0x7e | (ext >> 8) as u8,
ext as u8,
]);
let (first, rest) = pes.split_at(176);
self.out.extend_from_slice(first);
for chunk in rest.chunks(184) {
self.header(pid, false, false);
self.out.extend_from_slice(chunk);
}
}
}

/// One second of `programs` H.264 programs at `bitrate` bits per second each, interleaved
/// frame by frame, each program carrying its PCR on its video PID.
fn multiplex(programs: u16, bitrate: u64) -> Vec<u8> {
let packets = (bitrate / FPS / 8).div_ceil(188) as usize;
let mut mux = Mux::default();
let mut pat = Vec::new();
for program in 1..=programs {
pat.extend_from_slice(&program.to_be_bytes());
pat.extend_from_slice(&(0xe000 | pmt_pid(program)).to_be_bytes());
}
let pat = section(0x00, 1, &pat);
let pmts: Vec<_> = (1..=programs)
.map(|program| {
let video = video_pid(program);
let mut body = (0xe000 | video).to_be_bytes().to_vec();
body.extend_from_slice(&[0xf0, 0x00, 0x1b]);
body.extend_from_slice(&(0xe000 | video).to_be_bytes());
body.extend_from_slice(&[0xf0, 0x00]);
section(0x02, program, &body)
})
.collect();
for frame in 0..FPS {
if frame % PSI_EVERY == 0 {
mux.psi(0, &pat);
for (program, pmt) in (1..=programs).zip(&pmts) {
mux.psi(pmt_pid(program), pmt);
}
}
let pts = 90_000 + frame * 90_000 / FPS;
for program in 1..=programs {
mux.frame(video_pid(program), packets, pts * 300, pts, frame % GOP == 0);
}
}
mux.out
}

/// A fresh importer, with the origin driver it needs kept alive but unpolled.
fn setup() -> (moq_mux::container::ts::Programs, impl Sized) {
let (origin, driver) = moq_net::origin::Producer::new(Default::default());
let programs = moq_mux::container::ts::Programs::new(origin, "bench.hang", moq_mux::catalog::Config::default());
(programs, driver)
}

/// Import all of `input` and finish, for the counters.
fn ingest(programs: u16, input: &[u8]) -> (moq_mux::container::ts::stats::Snapshot, impl Sized) {
let (mut importer, driver) = setup();
importer.decode(input).unwrap();
let finished = importer.finish().unwrap();
assert_eq!(finished.len(), usize::from(programs));
(importer.stats(), (driver, finished))
}

fn programs(c: &mut Criterion) {
let mut group = c.benchmark_group("ts_programs");
group.measurement_time(Duration::from_secs(2));
group.warm_up_time(Duration::from_millis(300));
group.sample_size(10);
for programs in [1u16, 4, 16] {
for mbps in [1u64, 4] {
let input = multiplex(programs, mbps * 1_000_000);
let (stats, _driver) = ingest(programs, &input);
assert!(
stats.streams.len() == usize::from(programs) && stats.streams.values().all(|s| s.units == FPS),
"every program delivers every frame"
);
group.throughput(Throughput::Bytes(input.len() as u64));
group.bench_function(BenchmarkId::new(format!("{programs}"), format!("{mbps}mbps")), |b| {
b.iter_batched(
setup,
|(mut programs, driver)| {
for chunk in input.chunks(188 * 7) {
programs.decode(chunk).unwrap();
}
(programs, driver)
},
BatchSize::PerIteration,
)
});
}
}
group.finish();
}

criterion_group!(benches, programs);
criterion_main!(benches);
Loading
Loading