From 38e998d42b822d5823ff57c51459a5366d38f918 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 12:38:10 -0700 Subject: [PATCH 01/17] quest: define TS fidelity, recovery, and automated validation Co-Authored-By: Codex --- quest/m1/README.md | 4 +- quest/m1/hop-aligned-import.md | 11 ++++ quest/m1/open-gop-leading-pictures.md | 10 +++- quest/m1/plan-ts-alignment.md | 46 ++++++++++++++ quest/m1/plan-ts-pes-untimed.md | 33 ++++++++-- quest/m1/ts-passthrough-export.md | 41 +++++++++++-- quest/m1/ts-reliability/README.md | 69 +++++++++++++++++++++ quest/m1/ts-reliability/fidelity.md | 57 ++++++++++++++++++ quest/m1/ts-reliability/pes.md | 44 ++++++++++++++ quest/m1/ts-reliability/recovery.md | 62 +++++++++++++++++++ quest/m1/ts-reliability/validation.md | 73 +++++++++++++++++++++++ quest/m1/tstd/README.md | 27 +++++++-- quest/m1/tstd/send-ahead.md | 5 +- quest/m2/README.md | 2 +- quest/m2/intra-refresh/README.md | 5 ++ quest/m2/intra-refresh/consumer-warmup.md | 1 + quest/m2/intra-refresh/h264-import.md | 9 +++ quest/m2/intra-refresh/h265-import.md | 8 ++- quest/m2/intra-refresh/plan-boundary.md | 38 ++++++++++++ quest/m2/ts-hitless.md | 15 +++-- quest/m2/tstd-controls.md | 9 ++- quest/m3/msfts-convergence.md | 17 +++++- 22 files changed, 551 insertions(+), 35 deletions(-) create mode 100644 quest/m1/plan-ts-alignment.md create mode 100644 quest/m1/ts-reliability/README.md create mode 100644 quest/m1/ts-reliability/fidelity.md create mode 100644 quest/m1/ts-reliability/pes.md create mode 100644 quest/m1/ts-reliability/recovery.md create mode 100644 quest/m1/ts-reliability/validation.md create mode 100644 quest/m2/intra-refresh/plan-boundary.md diff --git a/quest/m1/README.md b/quest/m1/README.md index e213823fff0..082a3a25950 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -75,6 +75,7 @@ session outcomes, and embedder refusals inside session outcomes. - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away - [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - a sim test names any loop that holds a task poll too long, and each one yields through a budget - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` +- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle shared source coordinates and distinguish TS identity from coordinated RTP recovery - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives - [moq.sh deploys from CI](/quest/m1/moq-sh-deploy.md) - the first `release` run of the moq.sh workflow deploys with the Workers Editor token - [Plan: untimed verbatim PES](/quest/m1/plan-ts-pes-untimed.md) - decide how a verbatim TS track carries a PES that has no PTS, then write the implementation quest @@ -159,7 +160,8 @@ session outcomes, and embedder refusals inside session outcomes. - [MSF caption roles](/quest/m1/captions-msf.md) - an MSF caption or subtitle track survives conversion to a hang catalog - [Fail loud on dropped timed metadata](/quest/m1/drop-loud.md) - FLV script tags and fMP4 emsg boxes are counted and warned about instead of silently dropped, until real carriage lands - [Encoder colour](/quest/m1/color-model.md) - every moq-video encode path signals the colour its output actually has, or refuses instead of mislabelling -- [T-STD TS export](/quest/m1/tstd/README.md) - `moq export ts` is a proper remux that passes the T-STD buffer model, starting with a fixed `--delay` +- [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide +- [T-STD TS export](/quest/m1/tstd/README.md) - `moq export ts` is a proper remux that passes the supported T-STD buffer models, starting with a fixed `--delay` - [Multi-packet PMT](/quest/m1/ts-pmt-split.md) - `export ts` splits a PMT longer than one packet instead of exiting before its first packet - [ATSC AC-3](/quest/m1/ts-atsc-ac3.md) - `export ts` carries 48 kHz ATSC AC-3 at every A/52 rate up to 640 kb/s instead of aborting from 384 kb/s up - [Rust non-continuous signal](/quest/m1/rust-continuous.md) - the Rust container consumer reports a frame after a subscribe or discontinuity as non-continuous, like JS, for the tune-in and warmup trims diff --git a/quest/m1/hop-aligned-import.md b/quest/m1/hop-aligned-import.md index 5bc8469df0f..75c3654ab51 100644 --- a/quest/m1/hop-aligned-import.md +++ b/quest/m1/hop-aligned-import.md @@ -12,6 +12,13 @@ failover `export ts` sees a timestamp rewind and exits (#4354). ## Plan +The 2026-10-11 audit adds a planning prerequisite for the shared source +coordinate. The group-number and wall-anchor proposals below are provisional +until it settles how the input supplies a common anchor and wrap era. A +shared epoch or raw PTS/PCR value alone does not supply that mapping. Also +resolve raw passthrough explicitly: today's `--epoch` accepts whole MPTS +there, whereas the semantic importer scope below says one program. + Decided: every publisher of one path and epoch MUST publish the same broadcasts and tracks, since routes with the same epoch resume each other's subscriptions. A redundant pair is keyed by a shared explicit epoch (decided @@ -61,6 +68,10 @@ issues' 1+1 setup (one relay, two `import ts` sharing one `--epoch`, two - [#4352](https://github.com/moq-dev/moq/issues/4352) - close this issue when the quest finishes - [#4354](https://github.com/moq-dev/moq/issues/4354) - close this issue when the quest finishes +## Required + +- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle the source-to-wall anchor, wrap era and raw group coordinate that a shared epoch alone does not provide + ## Related - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - a redundant pair shares one epoch diff --git a/quest/m1/open-gop-leading-pictures.md b/quest/m1/open-gop-leading-pictures.md index e6fb70d2bee..6aef2a2beba 100644 --- a/quest/m1/open-gop-leading-pictures.md +++ b/quest/m1/open-gop-leading-pictures.md @@ -56,13 +56,17 @@ decode, and the JS test should drive the software decoder. export still sends a join's orphaned leading pictures, so 3 of 500 frames on the open-GOP fixture decode after they present (`dts-before-pts`). Trim them at tune-in from the same signal, on the - export's first group only, and make `just test ts --open-gop` pass under + first group after each non-continuous transition, and make + `just test ts --open-gop` pass under `--strict`. Decided in the 2026-10-05 audit: the grader changes with it. `test/ts/open-gop.py`'s decode-order check (a contiguous run of source access units) and leading-pictures check, and the `test/ts/README.md` text that says the round-trip must hand every leading picture on, both expect - every picture today; rewrite them to expect the first group's orphaned - leading pictures trimmed and every later group's kept. Rejected: dropping + every picture today; rewrite them to expect orphaned leading pictures + trimmed after each delivery break and every continuous group's kept. + The 2026-10-11 audit applies the same recovery rule to TS as to the + playback paths: the previous first-group-only wording missed later skips. + Rejected: dropping this bullet and keeping the grader. - Tests: a synthetic group with a keyframe followed by two earlier-stamped deltas is trimmed on the first group and kept on the second; and a viewer diff --git a/quest/m1/plan-ts-alignment.md b/quest/m1/plan-ts-alignment.md new file mode 100644 index 00000000000..1c9b92a9b8f --- /dev/null +++ b/quest/m1/plan-ts-alignment.md @@ -0,0 +1,46 @@ +# [S] Plan: shared coordinates for redundant TS publishers and exporters + +## Goal + +Settle what shared information makes redundant TS publishers and exporters +identify the same content and schedule. Rewrite the implementation quests +around that contract before promising failover or ST 2022-7 compatibility. + +## Plan + +The 2026-10-11 audit found two missing prerequisites: + +- Same epoch identity does not establish a source PCR/PTS-to-wall mapping, + a 33-bit timestamp wrap era, or a common raw-packet position. Importers + starting at different offsets cannot infer these from the UUID alone. + `catalog::Timebase::place` applies a supplied mapping; it does not discover + one. Raw fallback's timeout cuts also depend on acquisition history, even + though the CLI currently accepts a shared `--epoch` there. +- Identical TS packets are insufficient for ST 2022-7, which requires the + matching RTP packets, including headers. Complementary losses on redundant + paths are a reason to merge them; requiring the two legs to drop the same + content defeats that benefit. + +Present the maintainer with the actual shared-coordinate choices: an +explicit common source anchor and era, a common upstream packetizer, or an +honestly narrowed importer contract. Include what each can prove when the +standby starts after a timestamp wrap, a clock break, or a fallback group cut. +Do not derive a media clock from wall arrival and call it deterministic. + +For egress, separate deterministic TS slot/counter identity for common +content from coordinated RTP packetization, sequence/timestamp/SSRC identity +and any receiver merge. Decide the intended scope and API before implementing +`--sync`. NTP-synchronized hosts alone do not supply the missing source map. + +Produce focused implementation quests and explicit refusals for unsupported +shared-epoch modes. Their automated tests start peers at different source +offsets, cross wraps, vary delivery skew, and introduce complementary loss; +compare common content in its shared coordinate rather than whole files +whose delivered content differs. Actual ST 2022-7 claims require RTP-level +tests against the [standard](https://pub.smpte.org/latest/st2022-7/st2022-7-2019.pdf). + +## Related + +- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - currently assumes an input-derived wall anchor is available +- [TS hitless legs](/quest/m2/ts-hitless.md) - TS identity and RTP recovery need separate acceptance evidence +- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - unchanged packets do not establish identical groups from independent starts diff --git a/quest/m1/plan-ts-pes-untimed.md b/quest/m1/plan-ts-pes-untimed.md index 70d698f12fb..1a59e6d0bde 100644 --- a/quest/m1/plan-ts-pes-untimed.md +++ b/quest/m1/plan-ts-pes-untimed.md @@ -1,11 +1,12 @@ -# [S] Plan: untimed verbatim PES +# [M] Plan: private PES timing and header fidelity ## Goal Decide how a verbatim MPEG-TS track carries a PES that has no PTS without -inventing a time, from import through export. The usual case is asynchronous -private data such as KLV. The output is an implementation quest, or a -recorded reason to keep today's behaviour. +inventing a presentation time, from import through export, and which PES +header semantics it retains. The usual case is asynchronous private data +such as KLV. Produce focused implementation quests with explicit supported +profiles and refusal rules. ## Plan @@ -35,5 +36,25 @@ the decision covers: The maintainer decides the carriage, because it touches the catalog and container contract. -Test plan for the implementation quest: importer to encoded track to -exporter, for a PES without a PTS and for one with a real PTS of 0. +The 2026-10-11 audit expands the decision to avoid settling timestamps in +isolation. A source placement time used for mux scheduling is distinct from +optional PTS and DTS. Today's private PES has only payload plus `stream_id`: +the parser skips ESCR, ES rate, trick mode, copy information, CRC and extension +fields; export synthesizes a generic PTS header and cannot reproduce all +special stream-ID header forms. A track called verbatim must say which bytes +or semantics that means. Unknown stream types also need to survive export or +be refused at import, not fail later through an enum conversion. + +Settle per-PES timing presence, scheduling of asynchronous data alongside +advancing media, header preservation versus normalization, and the legacy +container migration together. The exporter currently drops non-video frames +before the initial video time; demonstrate that permitted untimed/private +content is not lost through that rule. Keep the catalog/container decision +separate from raw whole-TS fallback, whose packets already contain the headers. + +Automated CI tests for the implementation quests: importer to encoded track +to exporter for missing PTS versus real PTS 0, distinct PTS/DTS, asynchronous +private data before and between media, optional headers, and special stream +IDs. Fixtures assert either retained semantics or the precise supported-profile +refusal, using TSDuck/FFmpeg where applicable. Update Rust, JS, the MPEG-TS +draft and the guide for any catalog/container wire change. diff --git a/quest/m1/ts-passthrough-export.md b/quest/m1/ts-passthrough-export.md index 098caf4f6cc..8fca54f9517 100644 --- a/quest/m1/ts-passthrough-export.md +++ b/quest/m1/ts-passthrough-export.md @@ -6,7 +6,9 @@ `m2ts` catalog section names and writes the same bytes back out, paced on the source's own PCR and released at a fixed delay like an SRT receiver. From the first group it releases, the output is byte-identical to the input, less any -object dropped late, so it is exactly as T-STD-conformant as the source. +object dropped late. On a complete path, validate preservation of the source +schedule as well as the bytes. Missing bytes can break a decoder's acquisition +or buffers; packet identity alone does not preserve T-STD conformance under loss. There is no JS player for it. ## Plan @@ -14,6 +16,21 @@ There is no JS player for it. Decided (2026-10-01), from a discussion with t0ms, and split from the TS passthrough quest on #5003, whose import half it completes: +Confirmed in the 2026-10-11 audit: carry either one supplied SPTS or the whole +MPTS unchanged. Do not extract programs or split PIDs in this mode. Separate +SPTS inputs have separate raw broadcasts/tracks under the existing one-multiplex +catalog contract; extracting a program from an MPTS changes +the packet sequence and is not the byte-identical fallback. This matches the +distinction between `unmodified-program`, `unmodified-multiplex` and +`per-program` in [MSFTS](https://datatracker.ietf.org/doc/html/draft-gregoire-moq-msfts). + +The exactness boundary is the admitted source interval, starting at the first +released group. Account for the importer's current acquisition prefix and +188-byte framing restrictions explicitly; do not call this whole-file identity. +Test group acquisition without inserting synthetic tables or configuration +into the output. A periodic group boundary or one program's RAI does not +promise immediate decode for every program in an MPTS. + - The export reuses the TS export's jitter buffer (`rs/moq-mux/src/jitter.rs`), keyed on each object's PCR time instead of a DTS, which also spreads each object's bytes at the PCR-implied rate. It must pace on the source's PCR, @@ -36,12 +53,23 @@ passthrough quest on #5003, whose import half it completes: a coordinated RTP egress. - Rust only. -Test: an export of a broadcast capture is byte-identical to the input from the -first released group, and the strict T-STD check gives the same verdict on -output and input. Two exporters fed the same objects with different arrival +This quest owns the reusable raw packet/position and release-schedule oracle +and its basic omission fixtures; the recovery quest extends it with the wider +fault matrix rather than building a second harness. + +Automated tests in CI: an export of a supported capture is byte-identical to +the admitted input interval on a complete path, and the applicable T-STD check +gives the same verdict on output and input. Assert packet order and release +schedule as well as bytes. Two exporters fed the same objects with different arrival skew emit identical bytes; when only one misses a deadline, its output is the other's less that object's packets. A dropped object is counted, and the rest -still go out on time. Rerun the #4613 netem rig (10% loss, 120 s) against it. +still go out on their original schedule, leaving the missing time intact. +Use deterministic object/group omissions as well as a nightly real-network +impairment test: IP packet loss may be retransmitted and is not proof of +missing MoQ content. Fixtures include null/unknown PIDs, tables, adaptation +bytes, split PES, clock wrap, and MPTS with unaligned program recovery points. +Run TSDuck and FFmpeg automatically for profiles where their assertions apply; +record expected decode outages under loss separately from byte preservation. Update `doc/bin/cli.md` for the flag, which today says `export ts` does not read the passthrough track. @@ -52,3 +80,6 @@ Public API: a new export mode; nothing existing breaks. - [TS hitless](/quest/m2/ts-hitless.md) - the demultiplexed lane's 2022-7 legs and the `--sync` anchor - [T-STD TS export](/quest/m1/tstd/README.md) - the jitter buffer and clock recovery this release shares +- [TS validation](/quest/m1/ts-reliability/validation.md) - shared automatic graders and controls +- [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - actual missing content and reacquisition proof +- [Redundant alignment](/quest/m1/plan-ts-alignment.md) - identical common objects do not prove independent importers publish identical groups diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md new file mode 100644 index 00000000000..a933f3ec25a --- /dev/null +++ b/quest/m1/ts-reliability/README.md @@ -0,0 +1,69 @@ +# TS fidelity and recovery + +## Goal + +The two TS modes have explicit, software-validated contracts under clean +delivery, late join, and lost MoQ content. A dedicated guide explains which +mode to choose, supported profiles, acquisition, loss, and the automated +evidence behind each claim. + +## Plan + +Decided in the 2026-10-11 TS audit: + +- TS to Hang to TS is the semantic mode. Remove packetization and stuffing; + retain the supported source semantics through media tracks and catalog + extensions. Hang's core stays container-neutral. Byte identity is not its + contract. +- TS to MoQ to TS is the packet-preserving fallback. Keep an SPTS or an entire + MPTS unchanged. Per-program tracks mean separately supplied SPTS inputs + in separate raw broadcasts under the existing catalog contract, + not extraction from an MPTS. Reject per-PID splitting and reversible packet + scatter for this lane: both add reconstruction complexity to the fallback. +- Loss in the fallback emits unchanged surviving packets on their source + schedule and reports gaps. Never silently repair continuity counters, + insert tables or stuffing, or compress the missing interval. Exact bytes + do not imply successful decoding or T-STD conformance after loss. +- Start with explicit software-validated profiles and expand with fixtures. + Hardware receivers and commercial qualification are useful additional + evidence, not prerequisites for repeatable CI. Do not claim universal + broadcast compliance from the current subset of the buffer model. +- Every TS validation, including TSDuck and FFmpeg, is an automated test + with assertions and a CI invocation. One-off analysis is diagnostic + evidence, not completion. Long real-time tests may run nightly; short + deterministic checks belong on relevant PRs. + +The existing raw importer admits an aligned 188-byte suffix after acquisition; +it does not currently retain every byte supplied before that point. The guide +must distinguish full-input identity from identity over the admitted interval, +and distinguish a MoQ group boundary from a decoder acquisition point. RAI, +periodic cuts, and an intact object alone do not ensure available PAT/PMT, +codec configuration, clock context, or references for every MPTS program. + +The README owns the guide and the final two-mode integration proof. Reuse the +children's fixtures and CI recipes, including their known-bad controls. Publish +a coverage table linking each supported profile and loss guarantee to its +test; unsupported or unmeasured cells stay explicit. Keep CLI instructions +and the MPEG-TS draft consistent in each feature change. + +The planning children settle catalog and clock contracts before production +changes. Their output adds independently completable implementation quests to +this line; an audit finding is not authorization to guess a public API. +Existing fixes for PMT fragmentation, initial mux-rate measurement, PSI runs, +decoder-time admission, and catalog linger remain with their current quests. + +## Required + +- [Automated TS validation](/quest/m1/ts-reliability/validation.md) - software decode, structure, timing, and controls run in CI +- [Bounded PES assembly](/quest/m1/ts-reliability/pes.md) - split headers work and damaged input cannot grow staging indefinitely +- [Plan: semantic fidelity](/quest/m1/ts-reliability/fidelity.md) - explicit profiles and PMT lifecycle contracts produce implementation quests +- [Plan: PES timing and headers](/quest/m1/plan-ts-pes-untimed.md) - private PES semantics have a settled carriage and placement contract +- [Loss and recovery tests](/quest/m1/ts-reliability/recovery.md) - actual MoQ omissions exercise reacquisition and unaffected tracks +- [T-STD TS export](/quest/m1/tstd/README.md) - the semantic mux schedules the content it admits within its supported buffer models +- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - unchanged surviving packets keep their original schedule + +## Related + +- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - shared identity requires a shared source coordinate, and TS identity alone is not ST 2022-7 +- [TS health counters](/quest/m2/ts-health-stats.md) - ingress and egress evidence need distinct monitoring points +- [MSFTS mapping](/quest/m3/msfts-convergence.md) - document the exact equivalence and differences between the carriage modes diff --git a/quest/m1/ts-reliability/fidelity.md b/quest/m1/ts-reliability/fidelity.md new file mode 100644 index 00000000000..edb6f033421 --- /dev/null +++ b/quest/m1/ts-reliability/fidelity.md @@ -0,0 +1,57 @@ +# [M] Plan: explicit semantic TS profiles and live table changes + +## Goal + +Turn the semantic-lane audit into explicit supported profiles and focused +implementation quests. Each accepted mechanism is preserved, intentionally +normalized with a documented meaning, or refused. Nothing advertised as +supported disappears silently. + +## Plan + +Decided 2026-10-11: software-validated profiles first, then incremental +coverage. Broad MPEG-2/ATSC/private-data support is not a prerequisite for +shipping an honest narrower profile. The raw fallback stays an explicit +mode, not an automatic mid-stream switch of the catalog contract. + +Audit findings to reproduce and scope: + +- `Import::handle_pmt` adds streams but does not retire removed PES PIDs; + an existing stream bypasses updated types/descriptors. Several catalog + fields are captured only when a track is created. Export rejects a newly + added ES once tables exist, while removals leave the initial PSI layout. + Settle PMT version transitions, PID reuse, program/PCR changes, and track + identity together. A name must never identify different content after a + restart or PID reuse. +- MPEG-1/2 video is handled as a clock stream and its payload discarded. + Unknown stream-type bytes can import as opaque but fail export's closed + enum conversion. A known stream type is not proof of supported carriage. +- SI ingest recognizes fixed DVB PIDs. ATSC PSIP, CAT/conditional access, + dynamically signalled tables, descriptors and private sections need + explicit profile decisions. Packet passthrough can retain mechanisms the + semantic lane does not understand. +- Section export requires video, leaving audio-only programs with sections + unsupported. Keep opaque SCTE-35 fidelity separate from the later typed + browser cue API; basic carriage should not depend on that API existing. +- Private PES loses optional header semantics today; its timing/header + contract belongs to the existing planning quest linked below. + +The maintainer decides whether a supported profile handles live PMT changes +in place with fresh track identities, replaces the broadcast, or initially +refuses specific transitions. Present the observable tradeoffs and ask before +choosing the public shape. Build a mechanism-to-catalog/track/refusal table +and use it to produce independently completable quests, with Rust/JS/draft +impact recorded for every extension. Keep Hang's core TS-free. + +Every implementation quest includes generated or redistributable fixtures +and an automated round-trip or refusal test in CI. Include PMT add/remove, +version rollover and PID reuse, unknown stream types, audio-only plus +sections, missing timing, and the profile's descriptors. Preserve raw +section/PES bytes only where the settled contract actually promises that. + +## Related + +- [PES timing and headers](/quest/m1/plan-ts-pes-untimed.md) - separate placement, decode and presentation semantics +- [Catalog track identity](/quest/m1/catalog-track-id.md) - names identify the same content throughout their lifetime +- [Automated validation](/quest/m1/ts-reliability/validation.md) - profile declarations link to executable evidence +- [Typed SCTE-35](/quest/m3/2279-hang-typed-scte-35-ad-cue-signaling-carried-opaquely.md) - application cue APIs follow basic opaque carriage diff --git a/quest/m1/ts-reliability/pes.md b/quest/m1/ts-reliability/pes.md new file mode 100644 index 00000000000..66199ca6198 --- /dev/null +++ b/quest/m1/ts-reliability/pes.md @@ -0,0 +1,44 @@ +# [M] TS import assembles PES headers across packets with bounded staging + +## Goal + +Legal clear PES headers can span TS packets, and malformed or incomplete +input cannot grow importer staging indefinitely. Damage cannot concatenate +bytes across a lost packet into a plausible access unit. + +## Plan + +The audit found `Import::handle_pes` parses the first packet's payload as a +complete PES header. `PesStart::parse` needs the complete optional header, +so a legal clear header split across TS packets is discarded. PES payload +continuations and sections waiting for timing also have no explicit byte +bound in their staging paths. Raw passthrough already has explicit bounds; +semantic demux needs equally clear limits appropriate to its own units. + +Reassemble the header before parsing or delivering its payload. Keep the +state machine small and distinguish incomplete input from malformed input. +Honor declared lengths, continuity damage, discontinuity, duplicate packets, +new PUSI, and end of input. Special PES stream IDs have different header +rules and must not be forced through the ordinary optional-header layout. + +Bound retained bytes while waiting for a complete header, an unbounded PES's +next boundary, or missing clock/timestamp information. Derive protocol limits +where they exist, and use the repository's resource-limit conventions for +the rest. Refuse unsupported input explicitly instead of allowing unbounded +growth or silently inventing timing. A public configuration change needs its +own API decision if existing limits cannot express the policy. + +Regression tests fragment legal headers at every packet boundary and prove +payload/timestamps survive. Negative cases cover header length overruns, +truncation, continuity loss, and streams that never supply the awaited +boundary or clock. Assert retained memory is bounded and no damaged unit +escapes. Run these in the Rust CI tests; add a generated fragmented-header +round trip to the automated TSDuck/FFmpeg harness. + +The nonzero PES length underflow is already fixed separately in #5323; +retain its regression rather than duplicating that fix. + +## Related + +- [Automated validation](/quest/m1/ts-reliability/validation.md) - independent parsing and decoding of the reassembled output +- [PES timing and headers](/quest/m1/plan-ts-pes-untimed.md) - defines semantics retained after assembly diff --git a/quest/m1/ts-reliability/recovery.md b/quest/m1/ts-reliability/recovery.md new file mode 100644 index 00000000000..48014c19425 --- /dev/null +++ b/quest/m1/ts-reliability/recovery.md @@ -0,0 +1,62 @@ +# [L] TS recovery is tested against actual missing MoQ content + +## Goal + +Automated tests show how TS output and software decoding recover when MoQ +objects, groups, or tracks are absent or too late, including after playback +has started. Unaffected tracks continue according to the supported profile; +missing dependency chains are not presented as independently decodable. + +## Plan + +Test content loss separately from IP packet loss. QUIC may retransmit IP +loss; a netem percentage does not establish that any MoQ object was omitted. +Keep the real network impairment arm for transport and pacing evidence, and +add deterministic fault schedules at the MoQ content boundary. + +Cover a cold join, a skipped group, a partially delivered group, late objects +that miss release, one stalled/missing track, and resumed delivery. Put loss +around PAT/PMT or catalog acquisition, codec configuration and recovery +points, audio/video skew, PCR/PTS wrap and declared clock discontinuity. +Choose fixtures from declared supported profiles, not an unbounded matrix. + +Use mocked time for the Rust scheduling regressions. Integration captures +record exactly which content was delivered, omitted or late, so the oracle +does not explain every output gap as acceptable loss. Decode surviving +content with FFmpeg, compare recovered frames/samples against a continuous +source decode where applicable, and assert recovery at the profile's valid +acquisition point. No arbitrary recovery timeout becomes a substitute for +the encoded dependency rule. Exercise the graders' failure paths in CI. + +For semantic TS, validate continuity/discontinuity signalling, surviving +access-unit deadlines, decoder buffers and actual decoded output. Recovery +trims apply after a latency skip or discontinuity as well as the first join. +Coordinate the existing open-GOP and gradual-refresh quests: an H.264 +recovery-point SEI does not always promise immediate recovery, and HEVC CRA, +RADL and RASL do not have identical reference rules. Continuous subscribers +must retain valid pictures. + +For raw fallback, reuse the passthrough quest's packet/position and schedule +oracle. Extend its fault coverage to compare surviving packets against their exact positions +in the admitted source interval, including null packets, tables, unknown +PIDs and adaptation bytes. Assert preserved gaps in the release schedule, +no rewriting or duplication, and observed loss counters. A dropped object +may contain part of a PES or the next acquisition information; record when +decode is unavailable rather than claiming an unchanged-byte proof also +proves decoder conformance. Whole MPTS coverage includes programs with +unaligned recovery points and clocks. + +Short deterministic arms run on relevant PRs. Long drift and impairment +arms run nightly with failure artifacts. Keep byte-position PCR accuracy +and measured release timing as separate checks. + +## Required + +- [Automated validation](/quest/m1/ts-reliability/validation.md) - shared graders, profiles and known-bad controls +- [TS passthrough export](/quest/m1/ts-passthrough-export.md) - supplies the raw egress exercised by the loss matrix + +## Related + +- [Open-GOP leading pictures](/quest/m1/open-gop-leading-pictures.md) - trims after non-continuous transitions +- [Gradual recovery](/quest/m2/intra-refresh/README.md) - recovery counts and per-group warmup +- [Rust continuity](/quest/m1/rust-continuous.md) - consumers observe delivery breaks diff --git a/quest/m1/ts-reliability/validation.md b/quest/m1/ts-reliability/validation.md new file mode 100644 index 00000000000..0b065ec6b23 --- /dev/null +++ b/quest/m1/ts-reliability/validation.md @@ -0,0 +1,73 @@ +# [L] TS validators run as automated tests + +## Goal + +TSDuck, FFmpeg, and the repository's TS graders assert the supported output +profiles in repeatable CI tests. A green run means the expected content was +decoded and the named timing and structure checks passed, not merely that a +tool parsed a file or returned zero. + +## Plan + +Required by the maintainer in the 2026-10-11 audit: every TS validation is +automated, including any additional validator introduced later. Use the pinned +Nix tools and share local `just test` recipes with CI. Reuse `test/ts` rather +than introducing a second harness. + +Current gaps: + +- `run.sh` uses FFmpeg to generate sources but never fully decodes the + exported audio/video. `ffprobe_opus_channels` checks metadata only. +- `just test ts --pair` and `table-anchor.py` exist but no workflow invokes + that arm. Add a scheduled gate at minimum. +- `interop.yml` triggers for the harness but not `rs/moq-mux/**`. Relevant + mux, catalog, and TS CLI changes need the short integration gates. +- Plain captured runs do not reject a failed publisher, and the subscriber + is reaped without its status contributing to the verdict. A partial file + can pass analysis after the pipeline has failed. + +Add a reusable full software-decode grader for supported clean profiles, +starting with the existing closed-GOP H.264/AAC generator. FFmpeg must fail +on decode errors, decode every expected stream, and produce nonzero video +frames and audio samples where the source has them. Preserve diagnostics and +per-stream counts. A successful probe or concealment-tolerant decode is not +the assertion. Prove the grader rejects malformed coded payload and a missing +expected stream with automated controls that fail for the intended reason. + +The first declared profile also compares decoded content against the source +over the admitted interval. Match video frames and aligned audio samples rather +than requiring identical demux packet boundaries. Account explicitly for +startup, decoder priming, and end trimming; never loosen an existing duration +check to accommodate missing content. A grader that only checks decode +acceptance and nonzero progress must report that narrower result until the +source comparison passes. Open-GOP acquisition expectations land +with the existing leading-pictures work. + +Keep TSDuck's structural checks and explicit PCR-accuracy threshold, the +byte-schedule grader, and live release timing as distinct assertions. A CBR +byte-position PCR check is not a wall-clock jitter measurement. VBR and MPTS +with independent clocks need their own applicable timing model before they +can be labelled covered. Retain process failures, reports, tool versions, +input fixtures or reproducible generation parameters, and failing output as +CI artifacts. + +The custom T-STD model currently excludes the PSI system buffer, refuses +several layouts/codecs, and borrows AAC's buffer size for Opus. Report these +limits by profile. Coordinate system-buffer controls with `tstd/psi-runs` +and MB/AAC controls with the existing m2 quest; do not duplicate them or call +the current model complete. Correct the blanket claim in the harness docs +that no maintained tool implements T-STD: commercial analyzers advertise +it, while no suitable maintained open-source replacement was established. + +Independent failure reporting is already being fixed in the latecomer-control +quest. Keep its workflow changes, and ensure an earlier unrelated failure +cannot suppress these tests. All new controls have CI invocations too. + +## Related + +- [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - fixes the base capture regression without weakening the duration assertion +- [Independent Interop checks](/quest/m0/interop-latecomer-control.md) - TS tests still execute after another suite fails +- [T-STD controls](/quest/m2/tstd-controls.md) - MB and AAC positive/negative coverage +- [PSI system buffer](/quest/m1/tstd/psi-runs.md) - system-buffer proof belongs with the PSI fix +- [Open-GOP leading pictures](/quest/m1/open-gop-leading-pictures.md) - acquisition trims change the decoder oracle +- [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - extends these graders to declared omissions and recovery diff --git a/quest/m1/tstd/README.md b/quest/m1/tstd/README.md index 10a4d62c11c..ff2538b3008 100644 --- a/quest/m1/tstd/README.md +++ b/quest/m1/tstd/README.md @@ -1,12 +1,13 @@ -# [M] T-STD compliant TS export +# T-STD compliant TS export ## Goal -`moq export ts` output passes a full ISO 13818-1 T-STD buffer-model check -(transport, multiplex, and elementary buffers, with every access unit decoded -at its DTS without overflow or underflow), on a clean path and under -sustained loss. That is the bar for the media-aware lane to carry primary -distribution. Without it, only the passthrough lane can. +`moq export ts` output passes the applicable ISO 13818-1 T-STD buffer-model +checks for explicitly supported profiles, with admitted decodable access +units complete by their decode time and no modelled buffer overflow. Under +loss, validate the surviving content, declared discontinuities and recovery; +missing source units cannot all be decoded. Packet passthrough also needs +timing and recovery evidence before making a distribution-readiness claim. ## Plan @@ -33,6 +34,18 @@ This README owns the end-to-end proof: the #4613 netem rig (10% loss, a real ~10 Mb/s broadcast TS) passes the strict T-STD check, and the recipe runs nightly. +Audit clarification (2026-10-11): record what actually arrived, was +retransmitted, or missed its deadline. The IP loss percentage alone is not +a MoQ omission test. Keep this network/pacing proof and add deterministic +content-loss coverage through the TS reliability line. Run TSDuck, FFmpeg +software decode, and the custom model as automated assertions with known-bad +controls. A model pass is not independent proof of decoded content. + +The current model does not cover every codec/layout or the PSI system buffer. +Name those exclusions and assumptions in the profile report; the PSI quest +owns its system-buffer proof, and the existing controls quest owns the missing +MB/AAC controls. Do not relabel the current subset a full standards check. + ## Required - [Send-ahead within the delay](/quest/m1/tstd/send-ahead.md) - total lag is `--delay`, send-ahead included, with a 1 s default @@ -42,3 +55,5 @@ nightly. ## Related - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - the passthrough lane named in the Goal +- [Automated validation](/quest/m1/ts-reliability/validation.md) - independent decode, profile coverage and CI ownership +- [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - actual object/group omissions and recovery diff --git a/quest/m1/tstd/send-ahead.md b/quest/m1/tstd/send-ahead.md index b398cfc55f7..ddaa61b589e 100644 --- a/quest/m1/tstd/send-ahead.md +++ b/quest/m1/tstd/send-ahead.md @@ -30,4 +30,7 @@ carving alone should cut it by about one delay, not to 0.5 s. it (the existing `a_joiner_runs_at_the_delay_from_its_first_output`, tightened from two delays to one). `hrd9m` passes strict `tstd` at the 1 s default, and a delay below its buffer delay fails loud. -- Ask t0ms to re-grade the CNN capture: latency should drop by about a delay. +- Prove the latency change with generated or redistributable fixtures and + automated assertions in CI. A third-party capture re-grade can supplement + that evidence, but is not a completion dependency or a substitute for the + automated test. diff --git a/quest/m2/README.md b/quest/m2/README.md index 57fa319335f..df358b743e4 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -31,7 +31,7 @@ and bench readiness. Refusal reasons folded the same day into - [One port on the io_uring workers](/quest/m2/uring-demux.md) - `moq-uring`'s workers host the UDP demux, so a ring relay keeps WebRTC media and SRT on its QUIC port - [Synced data playback](/quest/m2/watch-data-sync.md) - js/watch releases JSON and binary payloads on the media playhead, and a slow data track holds media back - [Watch decode gate](/quest/m2/watch-decode-gate.md) - video lookahead stays encoded until it is near presentation, like audio -- [Hitless TS legs](/quest/m2/ts-hitless.md) - two `--sync` export legs emit packet-identical TS for ST 2022-7 +- [Hitless TS legs](/quest/m2/ts-hitless.md) - two export legs preserve common TS packet identity; coordinated RTP is a separate prerequisite for ST 2022-7 - [DVB E-AC-3](/quest/m2/ts-eac3.md) - E-AC-3 private data is split per access unit (independent frame plus its dependent substreams) and buffer-modelled in TS export - [T-STD controls](/quest/m2/tstd-controls.md) - the harness gains an MB-overflow control and an AAC broadcast reference - [MP4 export](/quest/m2/mp4-export.md) - `moq export mp4 --output` records crash-safe fragments, then finishes a regular MP4 with moov at the end diff --git a/quest/m2/intra-refresh/README.md b/quest/m2/intra-refresh/README.md index 17a54f4ae14..3f41d580f7e 100644 --- a/quest/m2/intra-refresh/README.md +++ b/quest/m2/intra-refresh/README.md @@ -25,6 +25,10 @@ broadcast feeds that only do intra refresh. Decisions the quests share: +The recovery-boundary planning prerequisite checks the duration contract +below before implementation. Its precision and publication ordering must +make these presentation guarantees true even when sweep lengths change. + - One group per refresh cycle. A group starts at the recovery point, the frame that begins a sweep, so a viewer joining at any group boundary is clean after exactly one cycle and relay shedding keeps its meaning. @@ -42,6 +46,7 @@ Decisions the quests share: ## Required +- [Plan: recovery boundaries](/quest/m2/intra-refresh/plan-boundary.md) - settle safe precision and publication ordering before consumers rely on a duration - [Consumer warmup](/quest/m2/intra-refresh/consumer-warmup.md) - JS and Rust viewers join `warmup` earlier and withhold display until recovery, except at a true IDR - [H.264 import](/quest/m2/intra-refresh/h264-import.md) - the splitter keeps `recovery_frame_cnt` and import publishes `warmup` from it - [H.265 import](/quest/m2/intra-refresh/h265-import.md) - the splitter reads the recovery-point SEI so an HEVC intra-refresh stream forms groups and publishes `warmup` diff --git a/quest/m2/intra-refresh/consumer-warmup.md b/quest/m2/intra-refresh/consumer-warmup.md index 536638fcad3..b860afe142b 100644 --- a/quest/m2/intra-refresh/consumer-warmup.md +++ b/quest/m2/intra-refresh/consumer-warmup.md @@ -60,6 +60,7 @@ replaces both: the rule is timestamp arithmetic on the group start. ## Required +- [Plan: recovery boundaries](/quest/m2/intra-refresh/plan-boundary.md) - consumers need a safe boundary before presenting the affected pictures - [Rust non-continuous signal](/quest/m1/rust-continuous.md) - the Rust signal this keys on ## Related diff --git a/quest/m2/intra-refresh/h264-import.md b/quest/m2/intra-refresh/h264-import.md index ea803af20d8..7dc8e8eebed 100644 --- a/quest/m2/intra-refresh/h264-import.md +++ b/quest/m2/intra-refresh/h264-import.md @@ -11,6 +11,11 @@ were immediately decodable. ## Plan +The recovery-boundary planning prerequisite must settle the precision and +publication ordering before implementing the duration proposal below. In +particular, rounding down and raising a catalog value after a longer sweep +are not yet proven safe for a viewer that already received those pictures. + - `rs/moq-mux/src/codec/h264/split.rs`: `sei_has_recovery_point` returns the `recovery_frame_cnt` the `h264-parser` crate already exposes on `SeiPayload::RecoveryPoint`, and the split frame carries it. A count of zero @@ -37,3 +42,7 @@ were immediately decodable. length-prefixed path; and a real clip generated with x264 `intra-refresh=1` added as a second round-trip in `test/ts/run.sh` next to the closed-GOP one, asserting the catalog `warmup` and the group count. + +## Required + +- [Plan: recovery boundaries](/quest/m2/intra-refresh/plan-boundary.md) - resolve the rounded and later-mutated warmup contract before implementing it diff --git a/quest/m2/intra-refresh/h265-import.md b/quest/m2/intra-refresh/h265-import.md index 278e0e1d00a..3ab09fd2f03 100644 --- a/quest/m2/intra-refresh/h265-import.md +++ b/quest/m2/intra-refresh/h265-import.md @@ -31,9 +31,11 @@ that machinery in parallel. recovery point's plus the count and measures `warmup` from its timestamp exactly as the H.264 quest does, sharing the measurement and catalog mutation. -- Fixture: synthetic NAL sequences beside the existing suffix-SEI tests, since - x265's intra refresh emits no recovery-point SEI. Verify by hand with an - externally encoded clip (ffmpeg `hevc_nvenc` with intra refresh). +- Fixtures: synthetic NAL sequences beside the existing suffix-SEI tests, + since x265's intra refresh emits no recovery-point SEI, and a + redistributable externally encoded clip when available. An NVENC-generated + fixture may supply that input, but validation must be a repeatable + software-decode test in CI, not a manual check or a hardware-only gate. ## Required diff --git a/quest/m2/intra-refresh/plan-boundary.md b/quest/m2/intra-refresh/plan-boundary.md new file mode 100644 index 00000000000..02dd77acd97 --- /dev/null +++ b/quest/m2/intra-refresh/plan-boundary.md @@ -0,0 +1,38 @@ +# [S] Plan: recovery boundaries remain safe as refresh duration changes + +## Goal + +Settle how consumers know a group's precise recovery boundary before +presenting pictures, including when later refresh sweeps take longer than +earlier ones. Rewrite the import and consumer quests around that contract. + +## Plan + +The 2026-10-11 TS audit found the proposed catalog duration is not yet a +sufficient guarantee. `h264-import` rounds a measured recovery duration down +to milliseconds and raises the global value after observing a longer sweep. +`consumer-warmup` presents any frame at or beyond that duration. A rounded +boundary can precede the recovery picture, and a later catalog update cannot +recall pictures a subscriber has already displayed using the old value. + +Ask the maintainer to choose between an exact per-group boundary and a safe +bound known and published before the affected content, including the latency +and catalog/container consequences. Test timestamp precision, reordered +pictures, frame_num wrap, variable sweep lengths, a missed recovery picture, +and a join/skip before a longer sweep is known. Decode-only pictures still +build reference state; withholding presentation is not dropping their input. + +Review the H.264 recovery-point flags and count semantics and HEVC +CRA/RADL/RASL separately. A timestamp-before-keyframe rule is a conservative +leading-picture policy, not a complete codec dependency model. State which +profiles it supports without changing an existing wire contract by inference. + +Produce implementation and automated software-decode tests in the existing +quests. Use generated fixtures, run short cases in CI, and retain full +reference-decode comparisons for continuous playback and each recovery case. + +## Related + +- [H.264 import](/quest/m2/intra-refresh/h264-import.md) - publishes recovery information +- [Consumer warmup](/quest/m2/intra-refresh/consumer-warmup.md) - uses it to withhold presentation +- [TS loss tests](/quest/m1/ts-reliability/recovery.md) - exercise actual group skips and recovered output diff --git a/quest/m2/ts-hitless.md b/quest/m2/ts-hitless.md index 40540918906..43747e2626e 100644 --- a/quest/m2/ts-hitless.md +++ b/quest/m2/ts-hitless.md @@ -4,8 +4,10 @@ Two `moq export ts --sync` legs of one broadcast, joined at different moments, emit packet-identical TS from their first common packet, across -skips and source drift, continuity counters included, within one epoch. A ST 2022-7 selector -can then switch between them hitlessly. +skips and source drift, continuity counters included, within one epoch. +This establishes TS identity only. ST 2022-7 also requires matching RTP +headers and payloads; coordinated RTP egress and receiver recovery need +their own scope and tests before claiming selector compatibility. Non-goal: identity across a replaced broadcast. A leg ends on a replacement, or starts a fresh stream with `--stitch`, @@ -28,8 +30,12 @@ requirement, so this waits until the T-STD line settles. there, so it comes from the media and not from what a leg has sent. The PCR moves to its own PID, because a discontinuity flag on the PCR PID would declare a time-base break. Loss is still detected within a group. -- Once anchored identically, both legs must make the same skip decision and - lay out the same slots. t0ms's `two_legs_render_a_skip_the_same_way` +- For content both legs release, the shared schedule must lay out the same + slots. Independent late delivery can leave complementary gaps; forcing both + legs to drop the same content would defeat redundant-path recovery. The + alignment planning prerequisite settles how the TS scheduler and any RTP + egress preserve common coordinates through those gaps. + t0ms's `two_legs_render_a_skip_the_same_way` (`rs/moq-mux/src/container/ts/export_timing_test.rs`) stays ignored until this lands. With the audio late, it also showed a PES header byte that differs at the skip; find and fix that. @@ -41,6 +47,7 @@ requirement, so this waits until the T-STD line settles. ## Required +- [Plan: redundant TS alignment](/quest/m1/plan-ts-alignment.md) - settle shared coordinates and coordinated RTP scope before implementing the public sync contract - [T-STD TS export](/quest/m1/tstd/README.md) - the export, clock recovery, and schedule these legs share ## Related diff --git a/quest/m2/tstd-controls.md b/quest/m2/tstd-controls.md index 5d8d20ffe16..7c7002e9c1f 100644 --- a/quest/m2/tstd-controls.md +++ b/quest/m2/tstd-controls.md @@ -4,8 +4,8 @@ `just test ts-tstd` has a negative control that overflows the video multiplex buffer (MB), and a positive reference that carries AAC the way a broadcast -encoder does. Every buffer and codec the export emits then has a control on -each side. +encoder does. These close the named MB/AAC gaps; they do not establish +coverage of every buffer, codec or layout the exporter can encounter. ## Plan @@ -22,3 +22,8 @@ The only positive reference carries MPEG audio, but the export emits AAC. shared. - Each control fails or passes for the stated reason, and `interop.yml` runs them. + +## Related + +- [TS validation](/quest/m1/ts-reliability/validation.md) - reports supported model coverage and assumptions without claiming full conformance +- [PSI system buffer](/quest/m1/tstd/psi-runs.md) - owns the separate system-buffer assertion and regression diff --git a/quest/m3/msfts-convergence.md b/quest/m3/msfts-convergence.md index 1e9cfdcf394..09031d91ff6 100644 --- a/quest/m3/msfts-convergence.md +++ b/quest/m3/msfts-convergence.md @@ -15,13 +15,24 @@ Decided in the 2026-09-30 audit: moved from m4 and the msfts#33 gate removed. msfts#33 closed on 2026-09-28, answered by msfts#36 (merged 2026-09-24), and egress moved to msfts#37 (closed). Two differences remain: -- **Program tables.** MSFTS carries them in tracks; here they live in the - Hang catalog's `mpegts` section. Decide whether to publish a mapping from - the catalog section to MSFTS's table tracks, or change either side. +- **Program tables.** PAT/PMT information lives in the Hang catalog's + `mpegts` section; supported SI already uses tracks. Map these mechanisms + separately to MSFTS's table tracks instead of saying all tables live in + the catalog. - **ES units.** MSFTS es-units carry the whole PES packet; here they carry the payload plus `stream_id`. Decide whether the PES header fields we drop matter to a subscriber, and converge or document the mapping. +Decided in the 2026-10-11 audit: keep the semantic lane and raw fallback +distinct. Whole SPTS maps to `unmodified-program`, whole MPTS to +`unmodified-multiplex`. MSFTS `per-program` extraction and `es-packets` +(which omit null packets) are not byte-identical whole-input carriage. +The fallback will not split PIDs or add source-position reconstruction +metadata. Use [the current draft](https://datatracker.ietf.org/doc/html/draft-gregoire-moq-msfts) +to document both semantic and scheduling differences; shared terminology +does not establish wire interoperability. Coordinate PES decisions with +[private PES timing and headers](/quest/m1/plan-ts-pes-untimed.md). + Update `drafts/draft-lcurley-moq-mpegts.md` and `doc/concept` with whatever lands. [#4577](https://github.com/moq-dev/moq/pull/4577) (per-ES access units at export) has merged, and [#4579](https://github.com/moq-dev/moq/pull/4579) From 2b3a9f2729c0b8049bdc0d02636e0b081795c8d5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 12:45:03 -0700 Subject: [PATCH 02/17] quest: name redundant TS work by its packet identity scope Co-Authored-By: Codex --- quest/m1/plan-ts-alignment.md | 2 +- quest/m1/ts-passthrough-export.md | 2 +- quest/m2/README.md | 2 +- quest/m2/ts-hitless.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/quest/m1/plan-ts-alignment.md b/quest/m1/plan-ts-alignment.md index 1c9b92a9b8f..7a53c2dbd61 100644 --- a/quest/m1/plan-ts-alignment.md +++ b/quest/m1/plan-ts-alignment.md @@ -42,5 +42,5 @@ tests against the [standard](https://pub.smpte.org/latest/st2022-7/st2022-7-2019 ## Related - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - currently assumes an input-derived wall anchor is available -- [TS hitless legs](/quest/m2/ts-hitless.md) - TS identity and RTP recovery need separate acceptance evidence +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - TS identity and RTP recovery need separate acceptance evidence - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - unchanged packets do not establish identical groups from independent starts diff --git a/quest/m1/ts-passthrough-export.md b/quest/m1/ts-passthrough-export.md index 8fca54f9517..a69e43db5fe 100644 --- a/quest/m1/ts-passthrough-export.md +++ b/quest/m1/ts-passthrough-export.md @@ -78,7 +78,7 @@ Public API: a new export mode; nothing existing breaks. ## Related -- [TS hitless](/quest/m2/ts-hitless.md) - the demultiplexed lane's 2022-7 legs and the `--sync` anchor +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - common packet identity and the proposed `--sync` anchor - [T-STD TS export](/quest/m1/tstd/README.md) - the jitter buffer and clock recovery this release shares - [TS validation](/quest/m1/ts-reliability/validation.md) - shared automatic graders and controls - [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - actual missing content and reacquisition proof diff --git a/quest/m2/README.md b/quest/m2/README.md index df358b743e4..46aa8df7acd 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -31,7 +31,7 @@ and bench readiness. Refusal reasons folded the same day into - [One port on the io_uring workers](/quest/m2/uring-demux.md) - `moq-uring`'s workers host the UDP demux, so a ring relay keeps WebRTC media and SRT on its QUIC port - [Synced data playback](/quest/m2/watch-data-sync.md) - js/watch releases JSON and binary payloads on the media playhead, and a slow data track holds media back - [Watch decode gate](/quest/m2/watch-decode-gate.md) - video lookahead stays encoded until it is near presentation, like audio -- [Hitless TS legs](/quest/m2/ts-hitless.md) - two export legs preserve common TS packet identity; coordinated RTP is a separate prerequisite for ST 2022-7 +- [Deterministic TS legs](/quest/m2/ts-hitless.md) - two export legs preserve common TS packet identity; coordinated RTP is a separate prerequisite for ST 2022-7 - [DVB E-AC-3](/quest/m2/ts-eac3.md) - E-AC-3 private data is split per access unit (independent frame plus its dependent substreams) and buffer-modelled in TS export - [T-STD controls](/quest/m2/tstd-controls.md) - the harness gains an MB-overflow control and an AAC broadcast reference - [MP4 export](/quest/m2/mp4-export.md) - `moq export mp4 --output` records crash-safe fragments, then finishes a regular MP4 with moov at the end diff --git a/quest/m2/ts-hitless.md b/quest/m2/ts-hitless.md index 43747e2626e..dc041cd3d0e 100644 --- a/quest/m2/ts-hitless.md +++ b/quest/m2/ts-hitless.md @@ -1,4 +1,4 @@ -# [L] Two TS export legs emit identical packets for ST 2022-7 +# [L] Redundant TS exports preserve common packet identity ## Goal From d7e8ea4ca20a1bcaa5aaa6016eb5855c3dd1e50f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 12:46:38 -0700 Subject: [PATCH 03/17] quest: clarify TS recovery and validation evidence Co-Authored-By: Codex --- quest/m1/open-gop-leading-pictures.md | 8 ++++---- quest/m1/plan-ts-pes-untimed.md | 5 +++-- quest/m1/tstd/README.md | 7 ++++--- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/quest/m1/open-gop-leading-pictures.md b/quest/m1/open-gop-leading-pictures.md index 6aef2a2beba..be7c8d0d01d 100644 --- a/quest/m1/open-gop-leading-pictures.md +++ b/quest/m1/open-gop-leading-pictures.md @@ -1,10 +1,10 @@ -# [M] Open-GOP leading pictures are dropped at tune-in only +# [M] Open-GOP recovery drops orphaned leading pictures after each delivery break ## Goal -A viewer joining an open-GOP broadcast at a recovery point never hands the -decoder a leading picture whose references it does not have, while a viewer -already playing keeps every frame. The rule is the same for H.264 +A viewer joining or resuming an open-GOP broadcast after a non-continuous +transition never hands the decoder a leading picture whose references it +does not have. Continuous groups keep every frame. The rule is the same for H.264 recovery-point keyframes with `recovery_frame_cnt = 0` (the broadcast contribution case, and what the reference capture carries) and H.265 CRA pictures. Gradual recovery (`recovery_frame_cnt > 0`) is out of scope: unsafe pictures diff --git a/quest/m1/plan-ts-pes-untimed.md b/quest/m1/plan-ts-pes-untimed.md index 1a59e6d0bde..4f05aec15fd 100644 --- a/quest/m1/plan-ts-pes-untimed.md +++ b/quest/m1/plan-ts-pes-untimed.md @@ -38,8 +38,9 @@ container contract. The 2026-10-11 audit expands the decision to avoid settling timestamps in isolation. A source placement time used for mux scheduling is distinct from -optional PTS and DTS. Today's private PES has only payload plus `stream_id`: -the parser skips ESCR, ES rate, trick mode, copy information, CRC and extension +optional PTS and DTS. Today's private-PES track retains payload, `stream_id`, +and present PTS as the frame timestamp, but not the original DTS or other +PES-header fields. The parser skips ESCR, ES rate, trick mode, copy information, CRC and extension fields; export synthesizes a generic PTS header and cannot reproduce all special stream-ID header forms. A track called verbatim must say which bytes or semantics that means. Unknown stream types also need to survive export or diff --git a/quest/m1/tstd/README.md b/quest/m1/tstd/README.md index ff2538b3008..d8d4e868354 100644 --- a/quest/m1/tstd/README.md +++ b/quest/m1/tstd/README.md @@ -30,9 +30,10 @@ latest. At 500 ms a 540 s run still misses a video deadline about 157 s in, which [send-ahead](/quest/m1/tstd/send-ahead.md)'s 1 s default addresses, and 10 % loss is still to pass. -This README owns the end-to-end proof: the #4613 netem rig (10% loss, a real -~10 Mb/s broadcast TS) passes the strict T-STD check, and the recipe runs -nightly. +This README owns the remaining end-to-end proof: make the #4613 netem rig +(10% loss, a redistributable or reproducible ~10 Mb/s broadcast-style TS) +pass the applicable strict T-STD checks, and run the recipe nightly. This +is a completion requirement, not a result established by the measurements above. Audit clarification (2026-10-11): record what actually arrived, was retransmitted, or missed its deadline. The IP loss percentage alone is not From 0495260a40f558c897d470ebbdcbad3887e3c6d6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 12:49:38 -0700 Subject: [PATCH 04/17] quest: claim automated TS validation Co-Authored-By: Codex From 406e3ffd851b8eeff553a2e7c0626037cb0f0368 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 13:08:09 -0700 Subject: [PATCH 05/17] test(ts): run strict FFmpeg and TSDuck validation in CI Co-Authored-By: Codex --- .github/workflows/alert.yml | 1 + .github/workflows/interop.yml | 63 +------- .github/workflows/nightly.yml | 15 -- .github/workflows/ts.yml | 156 ++++++++++++++++++++ test/justfile | 8 +- test/ts/README.md | 123 +++++++++------- test/ts/accuracy.py | 51 +++++++ test/ts/compliance.py | 8 +- test/ts/controls.py | 205 ++++++++++++++++++++++++++ test/ts/controls.sh | 9 ++ test/ts/decode.py | 263 ++++++++++++++++++++++++++++++++++ test/ts/eit-roundtrip.sh | 8 +- test/ts/process.sh | 11 ++ test/ts/process.test.sh | 29 ++++ test/ts/run.sh | 96 ++++++------- test/ts/tstd-controls.py | 50 +++---- test/ts/tstd-controls.sh | 9 ++ 17 files changed, 898 insertions(+), 207 deletions(-) create mode 100644 .github/workflows/ts.yml create mode 100644 test/ts/accuracy.py create mode 100644 test/ts/controls.py create mode 100644 test/ts/controls.sh create mode 100644 test/ts/decode.py create mode 100644 test/ts/process.sh create mode 100644 test/ts/process.test.sh create mode 100644 test/ts/tstd-controls.sh diff --git a/.github/workflows/alert.yml b/.github/workflows/alert.yml index 2776df94b7d..279bcb8ec25 100644 --- a/.github/workflows/alert.yml +++ b/.github/workflows/alert.yml @@ -59,6 +59,7 @@ on: - release-winget - rpm-repo - Swift + - TS - Video types: - completed diff --git a/.github/workflows/interop.yml b/.github/workflows/interop.yml index bf4e895eff1..52bb2860e15 100644 --- a/.github/workflows/interop.yml +++ b/.github/workflows/interop.yml @@ -3,8 +3,7 @@ name: Interop # Cross-language interop test built from THIS checkout (not the published # packages, unlike the moq-dev/smoke repo). Stands up a relay and runs the # publish x subscribe matrix across rust / python / go / browser / native-js / c, -# then browser-to-browser media QA and the TS/IRD compliance harness over the -# exporter's output. +# then browser-to-browser media QA. TS validation has its own workflow. on: # Manual on-demand runs. workflow_dispatch: @@ -19,10 +18,6 @@ on: - "test/interop/**" - "test/max-age/**" - "rs/moq-tokio/tests/max_age.rs" - # The TS/IRD compliance harness runs in the same job (see below). - - "test/ts/**" - # The live TS round trip also checks stdin ingestion keeps up with its source. - - "rs/moq-cli/src/publish.rs" # The interop job also depends on the recipe that invokes it and the # workspace membership that links the JS clients to source. - "test/justfile" @@ -133,62 +128,6 @@ jobs: env: MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs - # Round-trips a PCR-paced TS through a relay and checks the subscriber's - # `export ts` output with TSDuck (tsp/tsanalyze from the nix devShell). - - name: TS compliance - run: nix develop --command just test ts - shell: bash -leo pipefail {0} - env: - MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs - - # The same round-trip at 2 Mb/s, where every keyframe outgrows a 25 ms PCR slot, - # so the export has to spread it over the slots before its DTS and still keep - # the constant-rate schedule. - - name: TS keyframe bursts - run: nix develop --command just test ts --bitrate 2000000 - shell: bash -leo pipefail {0} - env: - MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs - - # 1080p video filling a broadcast-sized 9 Mbit CPB, at the default delay: the - # source sends pictures most of a second ahead of their decode time, and the - # export has to as well without overflowing the receiver's buffers. - - name: TS broadcast-sized CPB - run: nix develop --command just test ts --hrd - shell: bash -leo pipefail {0} - env: - MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs - - # The same video beside four audio PIDs in a multiplex faster than its Rx: each slot - # has to interleave the video with the rest, or its transport buffer overflows. - - name: TS multiplex above the video's Rx - run: nix develop --command just test ts --headroom - shell: bash -leo pipefail {0} - env: - MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs - - # The same round-trip over an open-GOP clip: every leading picture after a - # recovery point must come back, in decode order. - - name: TS open GOP - run: nix develop --command just test ts --open-gop - shell: bash -leo pipefail {0} - env: - MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs - - # Builds the sparse-EPG and pending-version fixtures and round-trips - # them, asserting the schedule survives and pending sections are dropped. - - name: EIT fixtures - run: nix develop --command just test ts-eit - shell: bash -leo pipefail {0} - env: - MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs - - # The T-STD model must pass a broadcast capture and fail the same capture - # delivered too slowly, too early, or in a burst. - - name: T-STD controls - run: nix develop --command just test ts-tstd - shell: bash -leo pipefail {0} - # A failing harness run keeps its directory, holding the process logs, the # relay config, and a Playwright trace of the failing page. Upload it for a # short while so a red cell is diagnosable; a passing run deletes its own. diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index a2276a34b68..2f9f7913047 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -213,21 +213,6 @@ jobs: if: ${{ !cancelled() }} run: nix develop --command just rs c-tests - # The TS exporter's PCR release timing. Everything else about the stream - # can be graded from a capture, but *when* the bytes were handed over - # cannot: the grader has to sit on the exporter's stdout for the length of - # the run, stamping each packet. That is a real-time window no per-PR gate - # should pay for, so it lives here (see test/ts/README.md). - # - # 120 s rather than the 20 s default, because the drift term needs it. The - # exporter builds its mux buffer once and holds it, and a window that ends - # while it is still filling cannot tell that from a pipe running slow: the - # lag reaches ~480 ms over the first ~48 s and is flat after. Nightly is the - # one arm with the wall-clock time to see the flat part. - - name: TS timing - if: ${{ !cancelled() }} - run: nix develop --command just test ts --live --duration 120 - tests: name: Tests (${{ matrix.recipe }}) runs-on: ubuntu-24.04-arm diff --git a/.github/workflows/ts.yml b/.github/workflows/ts.yml new file mode 100644 index 00000000000..310ae6faae3 --- /dev/null +++ b/.github/workflows/ts.yml @@ -0,0 +1,156 @@ +name: TS + +# Decoder, transport structure, buffer-model and release-timing gates are +# independent of browser interop. Every grader has an automated control arm. +on: + workflow_dispatch: + schedule: + - cron: "0 8 * * *" + pull_request: + paths: + - "test/ts/**" + - "test/lib/**" + - "test/justfile" + - "rs/moq-mux/**" + - "rs/hang/**" + - "rs/moq-cli/**" + - "rs/moq-net/**" + - "rs/moq-tokio/**" + - "rs/moq-relay/**" + - "rs/moq-loc/**" + - "rs/moq-json/**" + - "Cargo.toml" + - "Cargo.lock" + - "rust-toolchain.toml" + - "flake.nix" + - "flake.lock" + - ".github/actions/rust-cache/**" + - ".github/workflows/ts.yml" + +concurrency: + group: ts-${{ github.ref }} + cancel-in-progress: true + +permissions: + id-token: write + contents: read + +jobs: + ts: + name: TS validation + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Free disk space + uses: jlumbroso/free-disk-space@ceedf095f4ec1a097402bc6bd80831f2e1a6fde6 # main + with: + tool-cache: false + + - uses: DeterminateSystems/nix-installer-action@1d87d45818068401a10cf16bdc5f00b24994a83f # main + with: + determinate: false + extra-conf: | + extra-substituters = https://kixelated.cachix.org + extra-trusted-public-keys = kixelated.cachix.org-1:CmFcV0lyM6KuVM2m9mih0q4SrAa0XyCsiM7GHrz3KKk= + + - name: Rust cache + uses: ./.github/actions/rust-cache + + - name: FFmpeg and TSDuck controls + if: ${{ !cancelled() }} + run: nix develop --command just test ts-controls + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + + # Round-trips a PCR-paced TS through a relay and checks the subscriber's + # `export ts` output with TSDuck (tsp/tsanalyze from the nix devShell). + - name: TS compliance + if: ${{ !cancelled() }} + run: nix develop --command just test ts + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + shell: bash -leo pipefail {0} + + # The same round-trip at 2 Mb/s, where every keyframe outgrows a 25 ms PCR slot, + # so the export has to spread it over the slots before its DTS and still keep + # the constant-rate schedule. + - name: TS keyframe bursts + if: ${{ !cancelled() }} + run: nix develop --command just test ts --bitrate 2000000 + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + shell: bash -leo pipefail {0} + + # 1080p video filling a broadcast-sized 9 Mbit CPB, at the default delay: the + # source sends pictures most of a second ahead of their decode time, and the + # export has to as well without overflowing the receiver's buffers. + - name: TS broadcast-sized CPB + if: ${{ !cancelled() }} + run: nix develop --command just test ts --hrd + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + shell: bash -leo pipefail {0} + + # The same video beside four audio PIDs in a multiplex faster than its Rx: each slot + # has to interleave the video with the rest, or its transport buffer overflows. + - name: TS multiplex above the video's Rx + if: ${{ !cancelled() }} + run: nix develop --command just test ts --headroom + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + shell: bash -leo pipefail {0} + + # The same round-trip over an open-GOP clip: every leading picture after a + # recovery point must come back, in decode order. + - name: TS open GOP + if: ${{ !cancelled() }} + run: nix develop --command just test ts --open-gop + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + shell: bash -leo pipefail {0} + + # Builds the sparse-EPG and pending-version fixtures and round-trips + # them, asserting the schedule survives and pending sections are dropped. + - name: EIT fixtures + if: ${{ !cancelled() }} + run: nix develop --command just test ts-eit + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + shell: bash -leo pipefail {0} + + # The T-STD model must pass a broadcast capture and fail the same capture + # delivered too slowly, too early, or in a burst. + - name: T-STD controls + if: ${{ !cancelled() }} + run: nix develop --command just test ts-tstd + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + shell: bash -leo pipefail {0} + + # The pair needs enough overlap to grade the slower table cadences. + - name: TS table alignment + if: ${{ !cancelled() && github.event_name != 'pull_request' }} + run: nix develop --command just test ts --pair + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + + # Live timing needs the real-time window to distinguish drift from startup. + - name: TS live release timing + if: ${{ !cancelled() && github.event_name != 'pull_request' }} + run: nix develop --command just test ts --live --duration 120 + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + + - name: Upload failure artifacts + if: ${{ failure() }} + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ts-failure + path: ${{ runner.temp }}/moq-test-runs + retention-days: 7 + if-no-files-found: error diff --git a/test/justfile b/test/justfile index ad8a62bfa7f..e68777d8b9e 100644 --- a/test/justfile +++ b/test/justfile @@ -107,7 +107,7 @@ drill-sensitivity *args: ./drill/sensitivity.sh "$@" # Round-trips a PCR-paced TS through a relay and checks the subscriber's `export -# ts` output with TSDuck plus a custom analyzer (the T-STD buffer model, PCR +# ts` output with FFmpeg, TSDuck and a custom analyzer (the T-STD buffer model, PCR # spacing and byte schedule). Flags: `--analyze-only file.ts` # (skip the round-trip), `--strict` (fail on shape warnings), `--source file.ts` # (use a real capture), `--live` (grade PCR release timing and byte position off @@ -130,7 +130,11 @@ ts-eit *args: # The T-STD model's controls: a broadcast capture must pass, and the same capture # delivered too slowly, too early, or in a burst must fail. Needs only TSDuck. ts-tstd: - ./ts/tstd-controls.py + bash ./ts/tstd-controls.sh + +# Prove FFmpeg decode/content and TSDuck structure/PCR assertions reject damaged TS. +ts-controls: + bash ./ts/controls.sh # A JS viewer watching a live track through a relay that drains (SIGTERM, after # a stand-in for DNS stops pointing at it) migrates to a second relay behind the diff --git a/test/ts/README.md b/test/ts/README.md index 47e495ebe6c..b8484b5d6c6 100644 --- a/test/ts/README.md +++ b/test/ts/README.md @@ -5,26 +5,25 @@ what an Integrated Receiver/Decoder (IRD) expects. It round-trips a stream through a relay (`import ts` -> relay -> `export ts`), captures the output, and runs [TSDuck](https://tsduck.io) plus a custom analyzer over it. -This is a diagnostic gate, not just a pass/fail: the exporter -([`rs/moq-mux/src/container/ts/export.rs`](../../rs/moq-mux/src/container/ts/export.rs)) -pads with null packets to the multiplex rate the source recorded (or `--mux-rate`), -leaves a source without one unpadded, never delays media to fit the rate, and -puts a PCR on its own packet every 25 ms of media time, so several -broadcast-shape checks are expected to flag. The report quantifies exactly where -and by how much. - -Four instruments live here. `compliance.py` (via `run.sh`) grades a captured file -against the IRD model. [`pcr-timing.py`](#pcr-timing-pcr-timingpy) grades a live -pipe, which is the only way to see *when* the exporter released each PCR. -[`table-anchor.py`](#table-anchoring-table-anchorpy) grades two exporters of one -broadcast against each other, which is the only way to see whether a table's -emission points come from the media or from the exporter's own clock. -[`open-gop.py`](#open-gop-open-goppy) grades an open-GOP capture against its -source, access unit by access unit. +The harness keeps software decoding, transport structure, decoder-buffer models, +and release timing as separate assertions. Passing them establishes the named +software profiles below; it does not certify a hardware IRD or every TS layout. + +- `decode.py` runs strict software FFmpeg decoding and compares captured content + with the source for generated closed-GOP H.264/AAC-LC streams. +- `compliance.py` uses TSDuck for structure and a custom elementary-stream T-STD + model for supported profiles. +- `accuracy.py` requires measured PCRs within 500 ns of their CBR byte position. +- `pcr-timing.py` checks PCR values and byte schedules from files, and release + timing from a live pipe. +- `table-anchor.py` compares table emission points between two subscribers. +- `open-gop.py` checks the coded access units and leading pictures of open GOPs. ## Running ```bash +just test ts-controls # real FFmpeg/TSDuck positive and negative controls +just test ts-tstd # buffer-model controls just test ts # generate a clip, round-trip, analyze just test ts --source cap.ts # round-trip a real capture instead just test ts --analyze-only x.ts # skip the round-trip, analyze a file @@ -52,10 +51,10 @@ the first three seconds, failing on any PCR more than 500 ns off its byte positi at the rate: the `PCR_accuracy_error` an IRD or a TR 101 290 probe reports, which `pcr-schedule`'s packet of slack lets through. -The live arm passes only when the grader's verdict *and* the publisher's exit status -are clean. The grader can only speak for what reached it, and the sample floor -rejects a window that came up short, so a publisher dying late in the run leaves -enough behind to pass every check: a broken round-trip reported as a good one. +Publisher and captured subscriber processes must exit successfully. Captures drain +through EOF within the existing process deadline; a nonempty partial file cannot +hide a failed pipeline or timeout. The live arm also requires its grader to pass; +SIGPIPE is accepted only when that grader has completed its measurement window. `--analyze-only` needs only TSDuck + Python, so you can point it at any captured subscriber output: @@ -66,7 +65,7 @@ moq --connect http://localhost:4443 --broadcast live.hang export ts > sub.ts ``` Requirements: `tsp`, `tsanalyze` and `tstables` (TSDuck) and `python3` for every mode; the -round-trip modes also need `cargo`, `ffmpeg`, `curl`, and `timeout`. +round-trip modes also need `cargo`, `ffmpeg`, `ffprobe`, `curl`, and `timeout`. ## Checks @@ -89,7 +88,7 @@ Severities: **hard** checks fail the run by default; **shape** checks report as | `pcr-monotonic` | hard | PCR strictly increases (one 33-bit wrap tolerated), except into a PCR that signals `discontinuity_indicator` | | `duration-fidelity` | hard | exported PCR span tracks the source's duration (round-trip only) | | `service-descriptors` | shape | an SDT naming the service is present | -| `tstd` | hard | the full T-STD buffer model: no TB, MB, EB or B overflow, no access unit incomplete at its decoding time (see [T-STD](#t-std)) | +| `tstd` | hard | the supported elementary-stream T-STD models: no TB, MB, EB or B overflow, no access unit incomplete at its decoding time (see [T-STD](#t-std)) | Every timing check reads the stream's own PCR, so a PCR emitted on the wrong clock rate stays internally consistent and passes them all. `duration-fidelity` @@ -105,7 +104,7 @@ machine-readable report. ## T-STD -`tstd` runs every audio and video stream through the ISO 13818-1 system target +`tstd` models supported audio and video streams using the ISO 13818-1 system target decoder (2.4.2; Rec. ITU-T H.222.0, whose 10/2014 edition is a free download), fed on the stream's own PCR clock: @@ -120,9 +119,9 @@ second, where an access unit is not wholly in EB/B at its decoding time AVC/HEVC). The report gives each stream's peak fill per buffer, how late the worst underflowed access unit finished arriving, and the longest any access unit waited. -No maintained tool implements this. TSDuck has no T-STD analyzer, and nothing else -in nixpkgs does either, so the model is hand-rolled and its parameters are -transcribed from the specs: H.264 Table A-1 and H.265 Table A.8 for the level, the +TSDuck supplies the transport analysis, not this decoder-buffer model. Commercial +analyzers advertise T-STD checks; no suitable maintained open-source replacement +was established for this harness. Its parameters are transcribed from the specs: H.264 Table A-1 and H.265 Table A.8 for the level, the ADTS and "other audio" rates and sizes in H.222.0 2.4.2.3, and ATSC A/52 and A/53 Part 5 for AC-3 and E-AC-3. TSDuck does the parsing: `tstables` decodes the PMT and `tsp -P pes --avc-access-unit` the SPS, AVC and HEVC alike. @@ -141,9 +140,9 @@ Opus is graded against ADTS's buffers for the same channel count: the Opus-in-TS draft gives Rx (2 Mb/s for 1-2 channels, matching ADTS) but leaves the buffer size unset, so that size is borrowed rather than specified. A stream with no parameters at all is refused by name rather than skipped, which fails the check: MPEG-1/2 -video, DVB E-AC-3, and HEVC beyond Main/Main 10. Sections and private data -(SCTE-35, teletext) have no elementary-stream buffers and are listed as not -graded. A signalled PCR discontinuity starts fresh buffers, since the timestamps +video, DVB E-AC-3, and HEVC beyond Main/Main 10. The PSI system buffer is not implemented. Sections and private data +(SCTE-35, teletext) are listed as not graded; passing elementary-stream buffers +does not establish conformance for them. A signalled PCR discontinuity starts fresh buffers, since the timestamps on either side of it are on different clocks. Every packet on an elementary stream's PID enters TB, including adaptation-only @@ -587,27 +586,55 @@ than by malfunction; the census makes that visible instead of leaving it to be inferred from the code. Counts differ for a table that *did* survive because the exporter re-emits SI on its own repetition cadence rather than the source's. +## Decoded content (`decode.py`) + +The generated closed-GOP H.264/AAC-LC profile, at both 10 and 2 Mb/s, fully +decodes every expected video/audio stream with software FFmpeg. Missing streams, +empty output, decoder errors (including error logs with a zero exit status), and +configuration changes fail. Video frame hashes must match an intact source +suffix through EOF; frame timestamps must keep one common rebase. + +Audio timestamps must remain contiguous, allowing the 90 kHz PTS quantization +and FFmpeg's conversion to the sample clock. Audio sample counts pin the admitted +source offset against the same video rebase, so a shifted audio track or a lost +tail fails. Exact PCM comparison uses a fresh decode of source AAC starting at +the same access unit: AAC overlap and perceptual-noise state differ between a +cold decoder and a decoder that saw earlier frames. No arbitrary warmup samples +are discarded. PES boundaries and TS packetization may change. + +`just test ts-controls` exercises unchanged input, remuxing with new timestamps +and PES sizes, timestamp quantization, and a cold join. Negative controls require +the intended failure for missing audio, changed pictures, changed audio, a lost +tail, A/V skew, video clock drift, an audio gap, corrupted coded payload, and +empty input. Real TSDuck controls remove PAT, break continuity, set TEI, and check +PCR against a wrong rate or an empty measurement. Process controls also require +nonzero/timeout exit codes to fail despite nonempty output. A missing tool or +an unexpected failure category is a failing test, not a successful negative. + +Other codecs, open-GOP acquisition, loss/recovery, raw carriage, and arbitrary +source captures do not yet have decoded-content comparisons. Their existing +specific checks remain active and the harness explicitly reports the narrower +coverage. VBR and independent-clock MPTS need applicable timing models before +claiming the CBR PCR-accuracy profile. + ## CI -`.github/workflows/interop.yml` runs `just test ts`, `just test ts --bitrate -2000000`, `just test ts --hrd`, `just test ts --headroom`, -`just test ts --open-gop`, `just test ts-eit`, and `just test ts-tstd` after the -interop matrix (nightly, on demand, and on PRs touching `test/ts/`). -`ts-eit` is `eit-roundtrip.sh`: it builds the sparse-schedule and -pending-version fixtures from a generated clip, round-trips them through a -relay, and censuses the capture, so a break in the generators or in the SI -carriage they pin fails a PR instead of landing silently. TSDuck comes from the -`nix develop` shell, so the run uses the same `tsp`/`tsanalyze` a local -developer would. - -`.github/workflows/nightly.yml` runs `just test ts --live --duration 120` as -well. It stays off the PR path because release timing needs a real-time window to -measure: the grader has to sit on the pipe for the length of the capture, which -no per-PR gate should be paying for, and a scheduled arm on a quiet runner is a -better place to notice the exporter drifting off its own clock anyway. The window -is 120 s rather than the 20 s default because that is what the drift term costs -to read: under a minute it catches the mux buffer still filling and cannot tell -that from a pipe running slow. +[`.github/workflows/ts.yml`](../../.github/workflows/ts.yml) is independent of +browser interop. Relevant mux, Hang, CLI, transport, harness and toolchain PRs run +`just test ts-controls`, the default and 2 Mb/s round trips, HRD, headroom, +open-GOP, EIT, and T-STD controls. Each step runs even after another test fails. +All recipes use the same pinned Nix FFmpeg and TSDuck locally and in CI. + +Nightly and manual runs also invoke `just test ts --pair` and +`just test ts --live --duration 120`. The pair exercises late-joining subscribers; +the live window distinguishes ongoing clock drift from initial buffer filling. +These are actual test invocations, not optional tools installed on a runner. + +Failed runs retain source/capture files, process logs and statuses, tool versions, +decoder commands, frame hashes, PCM, and grader reports under the harness run +directory. CI uploads that directory for seven days. `MOQ_TEST_KEEP=1` preserves +successful local evidence too. `ts-eit` retains its sparse-schedule and pending +version fixtures; `ts-tstd` retains its existing model controls. ## Caveats diff --git a/test/ts/accuracy.py b/test/ts/accuracy.py new file mode 100644 index 00000000000..0c828286db9 --- /dev/null +++ b/test/ts/accuracy.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""Assert TSDuck PCR accuracy against a known CBR byte rate.""" + +import argparse +import re +import subprocess +from pathlib import Path + + +def check(path, bitrate, skip, log): + command = [ + "tsp", + "-I", + "file", + str(path), + "-P", + "skip", + str(skip), + "-P", + "pcrverify", + "--bitrate", + str(bitrate), + "--absolute", + "--jitter-max", + "13", + "-O", + "drop", + ] + # 13 ticks at 27 MHz is the integral threshold within +/-500 ns. Do not + # rely on pcrverify's much looser default or treat no measured PCRs as a pass. + with log.open("w") as output: + result = subprocess.run(command, stdout=output, stderr=subprocess.STDOUT, check=False) + if result.returncode: + raise RuntimeError(f"tsp exited {result.returncode}: {log}") + matches = re.findall(r"([\d,]+) PCR OK, ([\d,]+) with", log.read_text()) + if len(matches) != 1: + raise RuntimeError(f"expected one pcrverify verdict: {log}") + good, bad = (int(v.replace(",", "")) for v in matches[0]) + if not good or bad: + raise RuntimeError(f"PCR accuracy failed: {good} OK, {bad} outside threshold: {log}") + print(f"PCR accuracy: {good} OK, 0 outside +/-500 ns") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("ts", type=Path) + parser.add_argument("--bitrate", required=True, type=int) + parser.add_argument("--skip", default=0, type=int) + parser.add_argument("--log", required=True, type=Path) + args = parser.parse_args() + check(args.ts, args.bitrate, args.skip, args.log) diff --git a/test/ts/compliance.py b/test/ts/compliance.py index 0f37a6248fc..b85d2edffb1 100755 --- a/test/ts/compliance.py +++ b/test/ts/compliance.py @@ -8,15 +8,15 @@ Division of labour: TSDuck does the transport-stream parsing (we shell out to `tsanalyze --json` for PSI/service/structure and `tstables` for the PMT), and this script does what TSDuck does not cover: the ISO 13818-1 T-STD buffer -model, which no maintained tool implements, and the PCR checks that need the +model for supported elementary streams, and the PCR checks that need the time-base discontinuities a 188/204-byte header scan here recovers. PCR value intervals, byte schedule and release timing live in `pcr-timing.py`. Checks split into two severities: - HARD (structural): fail the run by default. PAT/PMT, packet size, sync, - continuity counters, PSI CRC, PCR presence, PCR monotonicity. + continuity counters, PSI CRC, PCR presence, PCR monotonicity, T-STD buffers. - SHAPE (broadcast profile): reported as WARN and only fail the run under - `--strict`. Service descriptors (SDT), T-STD buffer model. + `--strict`. Service descriptors (SDT). Timing basis is the stream's own PCR clock (an IRD locks to PCR), so the harness needs no wall-clock capture and results are deterministic for a given file. @@ -77,7 +77,7 @@ def run_tsanalyze(ts_path: str) -> dict: text=True, check=False, ) - if proc.returncode != 0 and not proc.stdout: + if proc.returncode != 0: raise RuntimeError(f"tsanalyze failed: {proc.stderr.strip()}") data = json.loads(proc.stdout) data["_stderr"] = proc.stderr diff --git a/test/ts/controls.py b/test/ts/controls.py new file mode 100644 index 00000000000..79a23e79794 --- /dev/null +++ b/test/ts/controls.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""Run real FFmpeg and TSDuck positive/negative controls without a relay.""" + +import json +import subprocess +import sys +from pathlib import Path + +import accuracy +import compliance +import decode + + +def command(args, log): + with log.open("w") as output: + subprocess.run(args, stdout=output, stderr=subprocess.STDOUT, check=True) + + +def main(root): + root.mkdir(parents=True, exist_ok=True) + source = root / "source.ts" + ffmpeg = ["ffmpeg", "-nostdin", "-y", "-hide_banner", "-v", "error"] + command( + ffmpeg + + [ + "-f", + "lavfi", + "-i", + "testsrc2=size=160x96:rate=25", + "-f", + "lavfi", + "-i", + "aevalsrc=sin(2*PI*(300*t+70*t*t)):s=48000", + "-t", + "3", + "-c:v", + "libx264", + "-pix_fmt", + "yuv420p", + "-threads", + "1", + "-x264-params", + "keyint=25:min-keyint=25:scenecut=0", + "-c:a", + "aac", + "-f", + "mpegts", + "-muxrate", + "1000000", + "-pcr_period", + "20", + "-pes_payload_size", + "0", + str(source), + ], + root / "generate.log", + ) + for tool, flag in [ + ("ffmpeg", "-version"), + ("ffprobe", "-version"), + ("tsp", "--version"), + ("tsanalyze", "--version"), + ]: + command([tool, flag], root / f"{tool}-version.txt") + + cases = [("unchanged", source, None)] + for name, options, expected in [ + ( + "remuxed", + ["-map", "0:v", "-map", "0:a", "-c", "copy", "-output_ts_offset", "5", "-pes_payload_size", "3000"], + None, + ), + ( + "audio-quantization", + ["-map", "0:v", "-map", "0:a", "-c", "copy", "-bsf:a", "setts=pts=PTS-mod(N\\,3):dts=DTS-mod(N\\,3)"], + None, + ), + ("missing-audio", ["-map", "0:v", "-c", "copy"], "streams"), + ( + "different-video", + ["-map", "0:v", "-map", "0:a", "-vf", "negate", "-c:v", "libx264", "-c:a", "copy"], + "video-content", + ), + ( + "different-audio", + ["-map", "0:v", "-map", "0:a", "-c:v", "copy", "-af", "atrim=start_sample=1024,volume=0", "-c:a", "aac"], + "audio-content", + ), + ("truncated-tail", ["-map", "0:v", "-map", "0:a", "-c", "copy", "-t", "2"], "video-content"), + ( + "audio-shift", + ["-map", "0:v", "-map", "0:a", "-c", "copy", "-bsf:a", "setts=pts=PTS+9000:dts=DTS+9000"], + "audio-length", + ), + ( + "video-rate", + ["-map", "0:v", "-map", "0:a", "-c", "copy", "-bsf:v", "setts=pts=PTS*1.01:dts=DTS*1.01"], + "video-timing", + ), + ( + "audio-gap", + [ + "-map", + "0:v", + "-map", + "0:a", + "-c", + "copy", + "-bsf:a", + "setts=pts=PTS+gte(N\\,60)*9000:dts=DTS+gte(N\\,60)*9000", + ], + "audio-timing", + ), + ]: + path = root / f"{name}.ts" + command(ffmpeg + ["-i", str(source)] + options + ["-f", "mpegts", str(path)], root / f"{name}.log") + cases.append((name, path, expected)) + + joined = root / "joined.ts" + command( + ffmpeg + ["-ss", "1", "-i", str(source), "-map", "0:v", "-map", "0:a", "-c", "copy", str(joined)], + root / "joined.log", + ) + cases.append(("joined", joined, None)) + + # Keep the transport/PES headers and parameter sets, damaging an actual + # coded slice mid-stream. This must reach the decoder, not just fail probing. + damaged = bytearray(source.read_bytes()) + changed = False + for off in range(len(damaged) // 2 // 188 * 188, len(damaged), 188): + pid = (damaged[off + 1] & 0x1F) << 8 | damaged[off + 2] + if pid != 0x100 or not damaged[off + 3] & 0x10 or damaged[off + 1] & 0x40: + continue + start = off + 4 + (1 + damaged[off + 4] if damaged[off + 3] & 0x20 else 0) + if start < off + 188: + damaged[start : off + 188] = b"\xff" * (off + 188 - start) + changed = True + break + assert changed, "fixture has no video continuation payload to corrupt" + corrupt = root / "corrupt-video.ts" + corrupt.write_bytes(damaged) + cases.append(("corrupt-video", corrupt, "decode")) + empty = root / "empty.ts" + empty.write_bytes(b"") + cases.append(("empty", empty, "decode")) + + results = [] + for name, path, expected in cases: + try: + report = decode.compare(source, path, root / name) + actual = None + except decode.Failure as error: + actual = error.check + report = {"check": actual, "detail": str(error)} + results.append({"case": name, "expected": expected, "actual": actual, "report": report}) + (root / "decode-controls.json").write_text(json.dumps(results, indent=2) + "\n") + assert actual == expected, f"{name}: expected {expected}, got {actual}: {report}" + if name == "joined": + assert report["startup_video_frames"] > 0 and report["cold_audio_reference"], report + print(f"PASS FFmpeg {name}: {actual or 'source content matches'}", flush=True) + + base = source.read_bytes() + variants = {"clean": base} + no_pat = bytearray(base) + for off in range(0, len(no_pat), 188): + if (no_pat[off + 1] & 0x1F) == 0 and no_pat[off + 2] == 0: + no_pat[off : off + 188] = b"\x47\x1f\xff\x10" + b"\xff" * 184 + variants["missing-pat"] = no_pat + cc = bytearray(base) + tei = bytearray(base) + off = len(base) // 2 // 188 * 188 + # Choose a media payload, since null PID continuity is intentionally ignored. + while ((base[off + 1] & 0x1F) << 8 | base[off + 2]) != 0x100 or not base[off + 3] & 0x10: + off += 188 + cc[off + 3] ^= 1 + tei[off + 1] |= 0x80 + variants.update(continuity=cc, transport_error=tei) + for name, data in variants.items(): + path = root / f"structure-{name}.ts" + path.write_bytes(data) + analysis = compliance.run_tsanalyze(str(path)) + checks = [ + *compliance.check_pat_pmt(analysis), + compliance.check_sync(analysis), + compliance.check_continuity(analysis), + ] + failed = {c.name for c in checks if c.status == compliance.Status.FAIL} + expected = {"missing-pat": "pat", "continuity": "continuity", "transport_error": "sync"}.get(name) + (root / f"structure-{name}.json").write_text(json.dumps(analysis, indent=2) + "\n") + assert (expected in failed) if expected else not failed, f"TSDuck {name}: {failed}" + print(f"PASS TSDuck {name}: {sorted(failed)}", flush=True) + + accuracy.check(source, 1_000_000, 0, root / "pcr-clean.log") + for name, rate, skip in [("wrong-rate", 2_000_000, 0), ("no-samples", 1_000_000, len(base) // 188)]: + try: + accuracy.check(source, rate, skip, root / f"pcr-{name}.log") + except RuntimeError as error: + assert "PCR accuracy failed" in str(error), str(error) + else: + raise AssertionError(f"pcrverify accepted {name}") + print(f"PASS TSDuck PCR {name}", flush=True) + + +if __name__ == "__main__": + main(Path(sys.argv[1])) diff --git a/test/ts/controls.sh b/test/ts/controls.sh new file mode 100644 index 00000000000..0c62efc8c87 --- /dev/null +++ b/test/ts/controls.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Real software-validator controls, with evidence retained on failure. +set -euo pipefail +DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source-path=SCRIPTDIR source=../lib/harness.sh +source "$DIR/../lib/harness.sh" +harness_begin ts-controls "just test ts-controls" +python3 "$DIR/controls.py" "$HARNESS_RUN" +bash "$DIR/process.test.sh" diff --git a/test/ts/decode.py b/test/ts/decode.py new file mode 100644 index 00000000000..1d12aedc752 --- /dev/null +++ b/test/ts/decode.py @@ -0,0 +1,263 @@ +#!/usr/bin/env python3 +"""Decode and compare a closed-GOP H.264/AAC TS round trip using software FFmpeg.""" + +import argparse +import json +import subprocess +import sys +from dataclasses import dataclass +from fractions import Fraction +from pathlib import Path + +import compliance + +TICK = Fraction(1, 90000) + + +class Failure(Exception): + """A named assertion failure, also used by the negative controls.""" + + def __init__(self, check, detail): + self.check = check + super().__init__(detail) + + +@dataclass +class Frame: + pts: Fraction + duration: int + size: int + digest: str + + +def run(command, log): + """Keep diagnostics even when an external validator fails.""" + with log.open("w") as stderr: + result = subprocess.run(command, stdout=subprocess.PIPE, stderr=stderr, check=False) + if result.returncode: + raise Failure("decode", f"{command[0]} exited {result.returncode}: {log}") + return result.stdout + + +def decode(path, output): + output.mkdir(parents=True, exist_ok=True) + probe = run(["ffprobe", "-v", "error", "-show_streams", "-of", "json", str(path)], output / "probe.log") + (output / "probe.json").write_bytes(probe) + streams = json.loads(probe)["streams"] + media = [s for s in streams if s["codec_type"] in ("video", "audio")] + video = [s for s in media if s["codec_type"] == "video"] + audio = [s for s in media if s["codec_type"] == "audio"] + if len(video) != 1 or len(audio) != 1: + raise Failure("streams", f"expected one video and one audio stream, got {len(video)} and {len(audio)}: {path}") + video, audio = video[0], audio[0] + if video["codec_name"] != "h264" or audio["codec_name"] != "aac" or audio.get("profile") != "LC": + raise Failure("profile", f"only the closed-GOP H.264/AAC-LC profile is covered: {path}") + + # Disable timestamp repair, frame duplication, hardware decode and tolerant + # error handling. PCM is also retained so comparison is sample-aligned, not + # tied to the framehash muxer's audio packet boundaries. + command = [ + "ffmpeg", + "-nostdin", + "-y", + "-hide_banner", + "-v", + "error", + "-xerror", + "-err_detect", + "explode", + "-abort_on", + "empty_output+empty_output_stream", + "-copyts", + "-hwaccel", + "none", + "-threads", + "1", + "-i", + str(path), + "-map", + f"0:{video['index']}", + "-map", + f"0:{audio['index']}", + "-c:v", + "rawvideo", + "-threads:v", + "1", + "-fps_mode", + "passthrough", + "-enc_time_base:v", + "1:90000", + "-c:a", + "pcm_s16le", + "-f", + "framehash", + str(output / "frames.sha256"), + "-map", + f"0:{audio['index']}", + "-c:a", + "pcm_s16le", + "-f", + "s16le", + str(output / "audio.pcm"), + "-map", + f"0:{audio['index']}", + "-c:a", + "copy", + "-f", + "adts", + str(output / "audio.aac"), + ] + (output / "command.json").write_text(json.dumps(command, indent=2) + "\n") + run(command, output / "decode.log") + # Some codec errors are logged without affecting the process exit status. + if (output / "decode.log").stat().st_size: + raise Failure("decode", f"FFmpeg logged errors: {output / 'decode.log'}") + frames = [[], []] + timebases = {} + for line in (output / "frames.sha256").read_text().splitlines(): + if line.startswith("#tb "): + index, value = line[4:].split(": ") + timebases[int(index)] = Fraction(value) + elif line and not line.startswith("#"): + index, _dts, pts, duration, size, digest = [x.strip() for x in line.split(",")] + index = int(index) + frames[index].append(Frame(int(pts) * timebases[index], int(duration), int(size), digest)) + if not all(frames): + raise Failure("decode", f"a declared stream decoded no frames: {path}") + return video, audio, frames, (output / "audio.pcm").read_bytes() + + +def audio_reference(output, samples, rate): + """Decode source audio with the same cold-start state as the subscriber.""" + data = (output / "source/audio.aac").read_bytes() + pos = count = 0 + while count < samples: + try: + size, duration = compliance.adts_frame(data, pos) + except compliance.Refused as error: + raise Failure("profile", str(error)) from error + if size < 7 or pos + size > len(data): + raise Failure("profile", "truncated source ADTS frame") + pos += size + count += round(duration * rate) + if count != samples: + raise Failure("audio-timing", "admitted audio does not start at a source AAC access-unit boundary") + reference = output / "reference.aac" + reference.write_bytes(data[pos:]) + log = output / "reference.log" + pcm = run( + [ + "ffmpeg", + "-nostdin", + "-hide_banner", + "-v", + "error", + "-xerror", + "-err_detect", + "explode", + "-threads", + "1", + "-i", + str(reference), + "-map", + "0:a:0", + "-c:a", + "pcm_s16le", + "-f", + "s16le", + "pipe:1", + ], + log, + ) + if log.stat().st_size: + raise Failure("decode", f"reference AAC decoder logged errors: {log}") + return pcm + + +def compare(source, capture, output): + """Compare the captured suffix, including its tail, with one shared A/V rebase.""" + sv, sa, sf, spcm = decode(source, output / "source") + cv, ca, cf, cpcm = decode(capture, output / "capture") + for keys, expected, actual in [ + (("width", "height", "pix_fmt"), sv, cv), + (("sample_rate", "channels", "channel_layout"), sa, ca), + ]: + if any(expected.get(k) != actual.get(k) for k in keys): + raise Failure("streams", f"stream configuration changed: {keys}") + + # Startup may admit a suffix. EOF is not a licence to lose the last frames: + # matching backwards pins the tail and detects missing/duplicated pictures. + start = len(sf[0]) - len(cf[0]) + if start < 0 or [f.digest for f in sf[0][start:]] != [f.digest for f in cf[0]]: + raise Failure("video-content", "decoded pictures are not an intact suffix of the source") + offset = cf[0][0].pts - sf[0][start].pts + if any(abs(c.pts - s.pts - offset) > TICK for s, c in zip(sf[0][start:], cf[0])): + raise Failure("video-timing", "video timing changed within the admitted interval") + + rate = int(sa["sample_rate"]) + stride = 2 * sa["channels"] + for label, rows, pcm in [("source", sf[1], spcm), ("capture", cf[1], cpcm)]: + samples = 0 + for frame in rows: + # PTS is quantized at 90 kHz, then FFmpeg rounds it onto the + # sample clock. Relative timestamps can differ by one sample plus + # one TS tick even when every AAC access unit is contiguous. + if abs(frame.pts - rows[0].pts - Fraction(samples, rate)) > TICK + Fraction(1, rate): + raise Failure("audio-timing", f"{label} has a gap or overlap in decoded audio") + if frame.size % stride: + raise Failure("audio-content", f"{label} has an incomplete PCM sample") + samples += frame.size // stride + if not samples or samples * stride != len(pcm): + raise Failure("audio-content", f"{label} PCM length disagrees with decoded frame sizes") + + audio_start = (cf[1][0].pts - offset - sf[1][0].pts) * rate + # PTS quantization and FFmpeg's sample timebase can differ by one sample. + # Content, length and the common video rebase must all agree on that offset. + candidates = [ + n + for n in range(round(audio_start) - 1, round(audio_start) + 2) + if n >= 0 and abs(Fraction(n, rate) - audio_start / rate) <= TICK + Fraction(1, rate) + ] + candidates = [n for n in candidates if len(spcm) - n * stride == len(cpcm)] + if not candidates: + raise Failure("audio-length", "audio does not reach the source tail at the shared A/V timestamp offset") + # Length pins a unique sample offset. AAC's MDCT overlap and perceptual + # noise state depend on earlier frames, so replay the admitted source AUs + # from a cold decoder too. No arbitrary warmup samples are excluded. + audio_start = candidates[0] + reference = audio_reference(output, audio_start, rate) if audio_start else spcm + if cpcm != reference: + raise Failure("audio-content", "aligned decoded PCM samples differ from the source") + return { + "profile": "closed-gop-h264-aac-lc", + "video_frames": len(cf[0]), + "source_video_frames": len(sf[0]), + "startup_video_frames": start, + "audio_samples": len(cpcm) // stride, + "startup_audio_samples": audio_start, + "cold_audio_reference": bool(audio_start), + "compared_audio_samples": len(cpcm) // stride, + "timestamp_offset_seconds": float(offset), + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("source", type=Path) + parser.add_argument("capture", type=Path) + parser.add_argument("--output", required=True, type=Path, help="directory for decoder evidence and report.json") + args = parser.parse_args() + args.output.mkdir(parents=True, exist_ok=True) + report = {"source": str(args.source), "capture": str(args.capture)} + try: + report.update(compare(args.source, args.capture, args.output)) + report["status"] = "PASS" + except Failure as error: + report.update(status="FAIL", check=error.check, detail=str(error)) + (args.output / "report.json").write_text(json.dumps(report, indent=2) + "\n") + print(json.dumps(report, indent=2)) + return int(report["status"] != "PASS") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test/ts/eit-roundtrip.sh b/test/ts/eit-roundtrip.sh index 1f6821f713b..35f4da7e903 100755 --- a/test/ts/eit-roundtrip.sh +++ b/test/ts/eit-roundtrip.sh @@ -17,8 +17,12 @@ set -euo pipefail DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -TMP=$(mktemp -d) -trap 'rm -rf "$TMP"' EXIT +# shellcheck source-path=SCRIPTDIR source=../lib/harness.sh +source "$DIR/../lib/harness.sh" +harness_begin ts-eit "$(harness_env TSC_EIT_DURATION)just test ts-eit$(harness_argv "$@")" +TMP="$HARNESS_RUN" +ffmpeg -version >"$TMP/ffmpeg-version.txt" +tsp --version >"$TMP/tsduck-version.txt" 2>&1 DURATION="${TSC_EIT_DURATION:-20}" diff --git a/test/ts/process.sh b/test/ts/process.sh new file mode 100644 index 00000000000..436ca28e9b0 --- /dev/null +++ b/test/ts/process.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# A capture is evidence only when its producer finished successfully. +ts_wait() { + local pid="$1" label="$2" status=0 + harness_wait "$pid" || status=$? + printf '%s: %s\n' "$label" "$status" >>"$HARNESS_RUN/process-status.log" + if [[ "$status" -ne 0 ]]; then + echo "error: $label exited $status; partial output is not a successful round trip" >&2 + return 1 + fi +} diff --git a/test/ts/process.test.sh b/test/ts/process.test.sh new file mode 100644 index 00000000000..77cd42a3c82 --- /dev/null +++ b/test/ts/process.test.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +set -euo pipefail +DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source-path=SCRIPTDIR source=../lib/harness.sh +source "$DIR/../lib/harness.sh" +# shellcheck source-path=SCRIPTDIR source=process.sh +source "$DIR/process.sh" +harness_begin ts-process "just test ts-controls" + +for code in 0 42 124; do + # shellcheck disable=SC2016 # $1 belongs to the child shell. + harness_spawn writer "$HARNESS_RUN/writer-$code.log" bash -c 'printf "captured content\n"; exit "$1"' _ "$code" + if ts_wait "$HARNESS_PID" "writer-$code"; then + [[ "$code" -eq 0 ]] || { + echo "accepted failed writer $code" >&2 + exit 1 + } + else + [[ "$code" -ne 0 ]] || { + echo "rejected successful writer" >&2 + exit 1 + } + fi + [[ -s "$HARNESS_RUN/writer-$code.log" ]] || { + echo "control produced no partial output" >&2 + exit 1 + } +done +echo "PASS process status: partial output cannot hide a failure or timeout" diff --git a/test/ts/run.sh b/test/ts/run.sh index 530449b453d..4f6401654e7 100755 --- a/test/ts/run.sh +++ b/test/ts/run.sh @@ -29,7 +29,7 @@ # checks can see *when* the exporter handed the bytes over. pcr-timing.py reads # the subscriber's stdout a packet at a time and stamps each read, so it grades # release timing and byte position alongside the values. Nightly runs this arm -# (.github/workflows/nightly.yml); it is not a per-PR gate, because it needs a +# (.github/workflows/ts.yml); it is not a per-PR gate, because it needs a # real-time window to measure at all. # # `--pair` changes the rig rather than the analyzer: it subscribes twice to one @@ -46,6 +46,8 @@ WORKSPACE=$(cd "$DIR/../.." && pwd) # Run directory, reserved ports, and process-group ownership. See test/README.md. # shellcheck source-path=SCRIPTDIR source=../lib/harness.sh source "$DIR/../lib/harness.sh" +# shellcheck source-path=SCRIPTDIR source=process.sh +source "$DIR/process.sh" # Captured before the parse below consumes it, so the rerun command carries the # source capture, mux rate, EIT fixture, and analyzer thresholds this run used, @@ -196,7 +198,7 @@ require_tools() { done # ffmpeg + cargo are only needed for the round-trip, not for --analyze-only. if [[ -z "$ANALYZE_ONLY" ]]; then - for t in cargo ffmpeg curl timeout; do have "$t" || missing+=("$t"); done + for t in cargo ffmpeg ffprobe curl timeout; do have "$t" || missing+=("$t"); done fi # The EIT fixture reads the service triplet out of the stream and may need to pad a # stuffing-free clip to make room for the table. @@ -214,9 +216,10 @@ analyze() { # Single source of truth for the verdict: compliance.py runs the TSDuck # tools itself and prints the PASS/WARN/FAIL summary. A second argument is # the source TS, which enables the duration-fidelity check (round-trip only). - local ref=() + local ref=() report=() + [[ -n "${HARNESS_RUN:-}" ]] && report=(--report-json "$HARNESS_RUN/compliance.json") [[ -n "${2:-}" ]] && ref=(--reference "$2") - python3 "$DIR/compliance.py" --ts "$1" ${ref[@]+"${ref[@]}"} $STRICT ${PASSTHRU[@]+"${PASSTHRU[@]}"} + python3 "$DIR/compliance.py" --ts "$1" ${ref[@]+"${ref[@]}"} ${report[@]+"${report[@]}"} $STRICT ${PASSTHRU[@]+"${PASSTHRU[@]}"} } require_tools @@ -235,6 +238,9 @@ fi # Only the round-trip stands anything up, so `--analyze-only` above needs no run # directory, no reserved port, and nothing to reap. harness_begin ts "$RERUN" +ffmpeg -version >"$HARNESS_RUN/ffmpeg-version.txt" +ffprobe -version >"$HARNESS_RUN/ffprobe-version.txt" +tsp --version >"$HARNESS_RUN/tsduck-version.txt" 2>&1 TARGET_BASE=$(cargo metadata --format-version 1 --manifest-path "$WORKSPACE/Cargo.toml" --no-deps | sed -n 's/.*"target_directory":"\([^"]*\)".*/\1/p') @@ -427,7 +433,7 @@ echo "### publishing PCR-paced TS -> $BROADCAST" # shellcheck disable=SC2329 # invoked indirectly via 'harness_spawn' publish() { # shellcheck disable=SC2016 # $1..$4 are the child bash -c positionals, not ours. - timeout -k 3 $((DURATION + 20)) bash -c ' + timeout -k 3 $((DURATION + 20)) bash -o pipefail -c ' tsp -I file "$1" -P regulate --pcr-synchronous --wait-min 5 | "$2" --connect "$3" --broadcast "$4" import ts ' _ "$SRC_TS" "$MOQ" "$URL" "$BROADCAST" @@ -435,18 +441,18 @@ publish() { harness_spawn pub "$HARNESS_RUN/pub.log" publish PUB_PID="$HARNESS_PID" -# Keep the publisher's exit status: `timeout` returns 124 when it had to kill a -# stalled `moq import ts`, non-zero/non-124 means the import itself errored. Both -# explain a truncated capture, so surface it alongside the logs on failure. -harness_wait "$PUB_PID" && PUB_RC=0 || PUB_RC=$? - # shellcheck disable=SC2329 # invoked from multiple failure paths below dump_logs() { - echo " publisher exit status: $PUB_RC" >&2 + cat "$HARNESS_RUN/process-status.log" >&2 || true sed 's/^/ pub: /' "$HARNESS_RUN/pub.log" >&2 || true sed 's/^/ sub: /' "$HARNESS_RUN/sub.log" >&2 || true } +if ! ts_wait "$PUB_PID" publisher; then + dump_logs + exit 1 +fi + # ── live: the grader owns the verdict ─────────────────────────────────────── if [[ -n "$LIVE" ]]; then harness_wait "$SUB_PID" || true @@ -458,16 +464,13 @@ if [[ -n "$LIVE" ]]; then read -r EXPORT_RC GRADE_RC <"$HARNESS_RUN/timing.rc" echo cat "$HARNESS_RUN/timing.out" - # 124 is `timeout` reaching the end of the window, which is how the exporter is - # meant to stop; SIGPIPE (141) is the grader closing the pipe on its own window. - # Anything else ended the stream under the grader, so say so: it graded a - # shorter window than it was asked for, and the report alone doesn't show that. - # Not fatal, because the grader's verdict on what it did see is still the - # verdict, and a short window can only make the sample smaller, not kinder. - if [[ "$EXPORT_RC" -ne 0 && "$EXPORT_RC" -ne 124 && "$EXPORT_RC" -ne 141 ]]; then + # SIGPIPE is expected when the live grader closes a completed measurement + # window. A timeout or another failure is not a successful export. + if [[ "$EXPORT_RC" -ne 0 && "$EXPORT_RC" -ne 141 ]]; then echo >&2 - echo "warning: the exporter exited $EXPORT_RC before the window closed" >&2 - sed 's/^/ sub: /' "$HARNESS_RUN/sub.log" >&2 || true + echo "error: the exporter exited $EXPORT_RC before the window closed" >&2 + dump_logs + exit 1 fi if [[ "$GRADE_RC" -ne 0 ]]; then echo >&2 @@ -475,23 +478,15 @@ if [[ -n "$LIVE" ]]; then dump_logs exit "$GRADE_RC" fi - # The grader's verdict is not the whole run. It grades whatever reached it, and - # the sample floor only rejects a window that came up short: a publisher that - # dies late still leaves enough behind to pass every check. That is a broken - # round-trip reported as a good one, so the publisher's own status is a gate - # too. 124 is `timeout` killing a stalled `moq import ts`, which is a failure - # of the same kind rather than a clean end, so it is not excused here. - if [[ "$PUB_RC" -ne 0 ]]; then - echo >&2 - echo "error: the publisher exited $PUB_RC; the graded stream is not a whole round-trip" >&2 - dump_logs - exit 1 - fi exit 0 fi -sleep 3 -harness_reap "$SUB_PID" +# Let the exporter drain to EOF within its existing timeout. Killing it after a +# sleep can both truncate a valid tail and conceal an earlier process failure. +if ! ts_wait "$SUB_PID" exporter; then + dump_logs + exit 1 +fi if [[ ! -s "$SUB_TS" ]]; then echo "error: subscriber captured no data" >&2 @@ -501,7 +496,11 @@ fi # ── pair: the two captures are the measurement ────────────────────────────── if [[ -n "$PAIR" ]]; then - harness_reap "$SUB_B_PID" + if ! ts_wait "$SUB_B_PID" exporter-b; then + dump_logs + cat "$HARNESS_RUN/sub-b.log" >&2 + exit 1 + fi if [[ ! -s "$SUB_B_TS" ]]; then echo "error: the second subscriber captured no data" >&2 sed 's/^/ sub-b: /' "$HARNESS_RUN/sub-b.log" >&2 || true @@ -510,7 +509,7 @@ if [[ -n "$PAIR" ]]; then fi echo "### captured $(wc -c <"$SUB_TS" | tr -d ' ') + $(wc -c <"$SUB_B_TS" | tr -d ' ') bytes -> comparing table anchors" echo - if ! python3 "$DIR/table-anchor.py" "$SUB_TS" "$SUB_B_TS" $STRICT \ + if ! python3 "$DIR/table-anchor.py" "$SUB_TS" "$SUB_B_TS" --report-json "$HARNESS_RUN/table-anchor.json" $STRICT \ ${PASSTHRU[@]+"${PASSTHRU[@]}"}; then echo >&2 echo "error: table anchor analysis failed (see round-trip logs below)" >&2 @@ -518,14 +517,6 @@ if [[ -n "$PAIR" ]]; then dump_logs exit 1 fi - # As in --live: a grader can only speak for what reached it, so a publisher that - # died mid-run must not be reported as a clean pair. - if [[ "$PUB_RC" -ne 0 ]]; then - echo >&2 - echo "error: the publisher exited $PUB_RC; the graded pair is not a whole round-trip" >&2 - dump_logs - exit 1 - fi exit 0 fi @@ -535,6 +526,15 @@ fi echo "### captured $(wc -c <"$SUB_TS" | tr -d ' ') bytes -> analyzing" +# The initial decoded-content profile is the generated closed-GOP H.264/AAC +# pair. Other arms retain their specific structural/model oracles until their +# acquisition and decoder-priming contracts are covered by fixtures. +if [[ -z "$SOURCE$OPEN_GOP$HRD$HEADROOM" ]]; then + python3 "$DIR/decode.py" "$SRC_TS" "$SUB_TS" --output "$HARNESS_RUN/decode" +else + echo "decoded-content comparison: not covered for this profile" +fi + # Which SI survived the round-trip. Informational: the exporter rebuilds SI from the # catalog, so a PID the import path does not route simply is not there, and that is a # statement about SI_PIDS rather than a malformed stream. compliance.py grades the stream @@ -580,7 +580,7 @@ fi # capture goes through pcr-timing.py as well. Its hard checks gate here as they do under # --live, and so does pcr-schedule when the rate is known (SCHEDULE). echo -if ! python3 "$DIR/pcr-timing.py" "$SUB_TS" ${SCHEDULE[@]+"${SCHEDULE[@]}"} $STRICT; then +if ! python3 "$DIR/pcr-timing.py" "$SUB_TS" --report-json "$HARNESS_RUN/pcr-timing.json" ${SCHEDULE[@]+"${SCHEDULE[@]}"} $STRICT; then echo >&2 echo "error: PCR timing analysis failed (see round-trip logs below)" >&2 dump_logs @@ -595,11 +595,7 @@ if [[ ${#SCHEDULE[@]} -gt 0 ]]; then echo echo "### PCR accuracy (tsp -P pcrverify, +/-500 ns at ${BITRATE} b/s)" SKIP=$((3 * BITRATE / 8 / 188)) - tsp -I file "$SUB_TS" -P skip "$SKIP" -P pcrverify --bitrate "$BITRATE" --absolute --jitter-max 13 \ - -O drop >"$HARNESS_RUN/pcrverify.log" 2>&1 || true - VERDICT=$(grep -E "PCR OK, " "$HARNESS_RUN/pcrverify.log" | tail -1) - echo " ${VERDICT:-no verdict}" - if [[ ! "$VERDICT" =~ ([0-9,]+)\ PCR\ OK,\ 0\ with ]]; then + if ! python3 "$DIR/accuracy.py" "$SUB_TS" --bitrate "$BITRATE" --skip "$SKIP" --log "$HARNESS_RUN/pcrverify.log"; then echo >&2 echo "error: PCRs stray from their byte position (see $HARNESS_RUN/pcrverify.log)" >&2 dump_logs diff --git a/test/ts/tstd-controls.py b/test/ts/tstd-controls.py index b1f94c558bb..aa336b4eaed 100755 --- a/test/ts/tstd-controls.py +++ b/test/ts/tstd-controls.py @@ -41,10 +41,10 @@ A model that passes everything, or fails everything, cannot tell these apart. """ +import json import os import subprocess import sys -import tempfile DIR = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(0, DIR) @@ -276,30 +276,32 @@ def grade(path: str) -> dict[str, compliance.Check]: return {check.name: check for check in compliance.analyze(path)} -def main() -> int: +def main(tmp: str) -> int: """Run every case and exit non-zero if any verdict is not the expected one.""" failed = 0 - with tempfile.TemporaryDirectory() as tmp: - for n, (name, build, expected) in enumerate(CASES): - checks = grade(build(os.path.join(tmp, f"{n}.ts"))) - check = checks["tstd"] - streams = check.metrics.get("streams", {}) - seen = {v for s in streams.values() for v in s["violations"]} - if expected == "pass": - # Not vacuous: every stream must have had access units to grade. - ok = check.status == compliance.Status.PASS and all(s["access_units"] for s in streams.values()) - ok = ok and checks["pcr-presence"].status == compliance.Status.PASS - elif isinstance(expected, tuple): - check = checks[expected[1]] - ok = check.status == compliance.Status.FAIL - elif isinstance(expected, str): - ok = any(expected in why for why in check.metrics.get("refused", {}).values()) - else: - ok = expected <= seen - failed += not ok - want = " ".join(reversed(expected)) if isinstance(expected, tuple) else expected - want = want if isinstance(want, str) else ", ".join(sorted(want)) - print(f" {'ok ' if ok else 'FAIL'} {name:<18} want {want:<40} got {check.detail}") + for n, (name, build, expected) in enumerate(CASES): + checks = grade(build(os.path.join(tmp, f"{n}.ts"))) + with open(os.path.join(tmp, f"{n}.json"), "w") as report: + json.dump({key: {"status": value.status.value, "detail": value.detail, "metrics": value.metrics} + for key, value in checks.items()}, report, indent=2) + check = checks["tstd"] + streams = check.metrics.get("streams", {}) + seen = {v for s in streams.values() for v in s["violations"]} + if expected == "pass": + # Not vacuous: every stream must have had access units to grade. + ok = check.status == compliance.Status.PASS and all(s["access_units"] for s in streams.values()) + ok = ok and checks["pcr-presence"].status == compliance.Status.PASS + elif isinstance(expected, tuple): + check = checks[expected[1]] + ok = check.status == compliance.Status.FAIL + elif isinstance(expected, str): + ok = any(expected in why for why in check.metrics.get("refused", {}).values()) + else: + ok = expected <= seen + failed += not ok + want = " ".join(reversed(expected)) if isinstance(expected, tuple) else expected + want = want if isinstance(want, str) else ", ".join(sorted(want)) + print(f" {'ok ' if ok else 'FAIL'} {name:<18} want {want:<40} got {check.detail}") if failed: print(f"tstd controls: {failed} of {len(CASES)} cases wrong", file=sys.stderr) return 1 @@ -308,4 +310,4 @@ def main() -> int: if __name__ == "__main__": - sys.exit(main()) + sys.exit(main(sys.argv[1])) diff --git a/test/ts/tstd-controls.sh b/test/ts/tstd-controls.sh new file mode 100644 index 00000000000..6c667786d41 --- /dev/null +++ b/test/ts/tstd-controls.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Keep the model's synthetic fixtures and reports when any control fails. +set -euo pipefail +DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source-path=SCRIPTDIR source=../lib/harness.sh +source "$DIR/../lib/harness.sh" +harness_begin ts-tstd "just test ts-tstd" +tsp --version >"$HARNESS_RUN/tsduck-version.txt" 2>&1 +python3 "$DIR/tstd-controls.py" "$HARNESS_RUN" From 790622d7378d92d3981094fb4870535fa6ea6fbe Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 13:13:56 -0700 Subject: [PATCH 06/17] test(ts): drain live measurements and complete validation quest Co-Authored-By: Codex --- quest/m1/plan-ts-alignment.md | 8 +++ quest/m1/ts-passthrough-export.md | 2 +- quest/m1/ts-reliability/README.md | 1 - quest/m1/ts-reliability/fidelity.md | 2 +- quest/m1/ts-reliability/pes.md | 2 +- quest/m1/ts-reliability/recovery.md | 3 +- quest/m1/ts-reliability/validation.md | 73 --------------------------- quest/m1/tstd/README.md | 1 - quest/m2/tstd-controls.md | 2 +- test/ts/README.md | 3 +- test/ts/process.sh | 8 +++ test/ts/process.test.sh | 11 ++++ test/ts/run.sh | 11 ++-- 13 files changed, 41 insertions(+), 86 deletions(-) delete mode 100644 quest/m1/ts-reliability/validation.md diff --git a/quest/m1/plan-ts-alignment.md b/quest/m1/plan-ts-alignment.md index 7a53c2dbd61..311535a3565 100644 --- a/quest/m1/plan-ts-alignment.md +++ b/quest/m1/plan-ts-alignment.md @@ -44,3 +44,11 @@ tests against the [standard](https://pub.smpte.org/latest/st2022-7/st2022-7-2019 - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - currently assumes an input-derived wall anchor is available - [Deterministic TS legs](/quest/m2/ts-hitless.md) - TS identity and RTP recovery need separate acceptance evidence - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - unchanged packets do not establish identical groups from independent starts + +The automated validation work (#5328) enables the existing `just test ts --pair` +arm nightly. Its local 45-second clean-delivery run found 100% PAT/PMT agreement +but 86.96% SDT agreement over 41 seconds, below the existing 90% gate. Preserve +that evidence while settling the alignment contract. The grader associates a +table with the next PES PTS; verify that assumption against the current slot +scheduler before attributing every mismatch to table emission. Do not loosen +the threshold or claim packet identity from table agreement alone. diff --git a/quest/m1/ts-passthrough-export.md b/quest/m1/ts-passthrough-export.md index a69e43db5fe..2d620257eb2 100644 --- a/quest/m1/ts-passthrough-export.md +++ b/quest/m1/ts-passthrough-export.md @@ -80,6 +80,6 @@ Public API: a new export mode; nothing existing breaks. - [Deterministic TS legs](/quest/m2/ts-hitless.md) - common packet identity and the proposed `--sync` anchor - [T-STD TS export](/quest/m1/tstd/README.md) - the jitter buffer and clock recovery this release shares -- [TS validation](/quest/m1/ts-reliability/validation.md) - shared automatic graders and controls +- [TS validation](/test/ts/README.md) - shared automatic graders and controls - [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - actual missing content and reacquisition proof - [Redundant alignment](/quest/m1/plan-ts-alignment.md) - identical common objects do not prove independent importers publish identical groups diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index a933f3ec25a..50d944b45d8 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -54,7 +54,6 @@ decoder-time admission, and catalog linger remain with their current quests. ## Required -- [Automated TS validation](/quest/m1/ts-reliability/validation.md) - software decode, structure, timing, and controls run in CI - [Bounded PES assembly](/quest/m1/ts-reliability/pes.md) - split headers work and damaged input cannot grow staging indefinitely - [Plan: semantic fidelity](/quest/m1/ts-reliability/fidelity.md) - explicit profiles and PMT lifecycle contracts produce implementation quests - [Plan: PES timing and headers](/quest/m1/plan-ts-pes-untimed.md) - private PES semantics have a settled carriage and placement contract diff --git a/quest/m1/ts-reliability/fidelity.md b/quest/m1/ts-reliability/fidelity.md index edb6f033421..767c13d1367 100644 --- a/quest/m1/ts-reliability/fidelity.md +++ b/quest/m1/ts-reliability/fidelity.md @@ -53,5 +53,5 @@ section/PES bytes only where the settled contract actually promises that. - [PES timing and headers](/quest/m1/plan-ts-pes-untimed.md) - separate placement, decode and presentation semantics - [Catalog track identity](/quest/m1/catalog-track-id.md) - names identify the same content throughout their lifetime -- [Automated validation](/quest/m1/ts-reliability/validation.md) - profile declarations link to executable evidence +- [Automated validation](/test/ts/README.md) - profile declarations link to executable evidence - [Typed SCTE-35](/quest/m3/2279-hang-typed-scte-35-ad-cue-signaling-carried-opaquely.md) - application cue APIs follow basic opaque carriage diff --git a/quest/m1/ts-reliability/pes.md b/quest/m1/ts-reliability/pes.md index 66199ca6198..87c98177f83 100644 --- a/quest/m1/ts-reliability/pes.md +++ b/quest/m1/ts-reliability/pes.md @@ -40,5 +40,5 @@ retain its regression rather than duplicating that fix. ## Related -- [Automated validation](/quest/m1/ts-reliability/validation.md) - independent parsing and decoding of the reassembled output +- [Automated validation](/test/ts/README.md) - independent parsing and decoding of the reassembled output - [PES timing and headers](/quest/m1/plan-ts-pes-untimed.md) - defines semantics retained after assembly diff --git a/quest/m1/ts-reliability/recovery.md b/quest/m1/ts-reliability/recovery.md index 48014c19425..06e696b7046 100644 --- a/quest/m1/ts-reliability/recovery.md +++ b/quest/m1/ts-reliability/recovery.md @@ -52,11 +52,12 @@ and measured release timing as separate checks. ## Required -- [Automated validation](/quest/m1/ts-reliability/validation.md) - shared graders, profiles and known-bad controls - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - supplies the raw egress exercised by the loss matrix ## Related +- [Automated validation](/test/ts/README.md) - shared graders, profiles and known-bad controls + - [Open-GOP leading pictures](/quest/m1/open-gop-leading-pictures.md) - trims after non-continuous transitions - [Gradual recovery](/quest/m2/intra-refresh/README.md) - recovery counts and per-group warmup - [Rust continuity](/quest/m1/rust-continuous.md) - consumers observe delivery breaks diff --git a/quest/m1/ts-reliability/validation.md b/quest/m1/ts-reliability/validation.md deleted file mode 100644 index 0b065ec6b23..00000000000 --- a/quest/m1/ts-reliability/validation.md +++ /dev/null @@ -1,73 +0,0 @@ -# [L] TS validators run as automated tests - -## Goal - -TSDuck, FFmpeg, and the repository's TS graders assert the supported output -profiles in repeatable CI tests. A green run means the expected content was -decoded and the named timing and structure checks passed, not merely that a -tool parsed a file or returned zero. - -## Plan - -Required by the maintainer in the 2026-10-11 audit: every TS validation is -automated, including any additional validator introduced later. Use the pinned -Nix tools and share local `just test` recipes with CI. Reuse `test/ts` rather -than introducing a second harness. - -Current gaps: - -- `run.sh` uses FFmpeg to generate sources but never fully decodes the - exported audio/video. `ffprobe_opus_channels` checks metadata only. -- `just test ts --pair` and `table-anchor.py` exist but no workflow invokes - that arm. Add a scheduled gate at minimum. -- `interop.yml` triggers for the harness but not `rs/moq-mux/**`. Relevant - mux, catalog, and TS CLI changes need the short integration gates. -- Plain captured runs do not reject a failed publisher, and the subscriber - is reaped without its status contributing to the verdict. A partial file - can pass analysis after the pipeline has failed. - -Add a reusable full software-decode grader for supported clean profiles, -starting with the existing closed-GOP H.264/AAC generator. FFmpeg must fail -on decode errors, decode every expected stream, and produce nonzero video -frames and audio samples where the source has them. Preserve diagnostics and -per-stream counts. A successful probe or concealment-tolerant decode is not -the assertion. Prove the grader rejects malformed coded payload and a missing -expected stream with automated controls that fail for the intended reason. - -The first declared profile also compares decoded content against the source -over the admitted interval. Match video frames and aligned audio samples rather -than requiring identical demux packet boundaries. Account explicitly for -startup, decoder priming, and end trimming; never loosen an existing duration -check to accommodate missing content. A grader that only checks decode -acceptance and nonzero progress must report that narrower result until the -source comparison passes. Open-GOP acquisition expectations land -with the existing leading-pictures work. - -Keep TSDuck's structural checks and explicit PCR-accuracy threshold, the -byte-schedule grader, and live release timing as distinct assertions. A CBR -byte-position PCR check is not a wall-clock jitter measurement. VBR and MPTS -with independent clocks need their own applicable timing model before they -can be labelled covered. Retain process failures, reports, tool versions, -input fixtures or reproducible generation parameters, and failing output as -CI artifacts. - -The custom T-STD model currently excludes the PSI system buffer, refuses -several layouts/codecs, and borrows AAC's buffer size for Opus. Report these -limits by profile. Coordinate system-buffer controls with `tstd/psi-runs` -and MB/AAC controls with the existing m2 quest; do not duplicate them or call -the current model complete. Correct the blanket claim in the harness docs -that no maintained tool implements T-STD: commercial analyzers advertise -it, while no suitable maintained open-source replacement was established. - -Independent failure reporting is already being fixed in the latecomer-control -quest. Keep its workflow changes, and ensure an earlier unrelated failure -cannot suppress these tests. All new controls have CI invocations too. - -## Related - -- [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - fixes the base capture regression without weakening the duration assertion -- [Independent Interop checks](/quest/m0/interop-latecomer-control.md) - TS tests still execute after another suite fails -- [T-STD controls](/quest/m2/tstd-controls.md) - MB and AAC positive/negative coverage -- [PSI system buffer](/quest/m1/tstd/psi-runs.md) - system-buffer proof belongs with the PSI fix -- [Open-GOP leading pictures](/quest/m1/open-gop-leading-pictures.md) - acquisition trims change the decoder oracle -- [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - extends these graders to declared omissions and recovery diff --git a/quest/m1/tstd/README.md b/quest/m1/tstd/README.md index d8d4e868354..7f0c4d0f58e 100644 --- a/quest/m1/tstd/README.md +++ b/quest/m1/tstd/README.md @@ -56,5 +56,4 @@ MB/AAC controls. Do not relabel the current subset a full standards check. ## Related - [TS passthrough export](/quest/m1/ts-passthrough-export.md) - the passthrough lane named in the Goal -- [Automated validation](/quest/m1/ts-reliability/validation.md) - independent decode, profile coverage and CI ownership - [Loss and recovery](/quest/m1/ts-reliability/recovery.md) - actual object/group omissions and recovery diff --git a/quest/m2/tstd-controls.md b/quest/m2/tstd-controls.md index 7c7002e9c1f..83a9d8b03ef 100644 --- a/quest/m2/tstd-controls.md +++ b/quest/m2/tstd-controls.md @@ -25,5 +25,5 @@ The only positive reference carries MPEG audio, but the export emits AAC. ## Related -- [TS validation](/quest/m1/ts-reliability/validation.md) - reports supported model coverage and assumptions without claiming full conformance +- [TS validation](/test/ts/README.md) - reports supported model coverage and assumptions without claiming full conformance - [PSI system buffer](/quest/m1/tstd/psi-runs.md) - owns the separate system-buffer assertion and regression diff --git a/test/ts/README.md b/test/ts/README.md index b8484b5d6c6..4ba7fd742a6 100644 --- a/test/ts/README.md +++ b/test/ts/README.md @@ -54,7 +54,8 @@ at the rate: the `PCR_accuracy_error` an IRD or a TR 101 290 probe reports, whic Publisher and captured subscriber processes must exit successfully. Captures drain through EOF within the existing process deadline; a nonempty partial file cannot hide a failed pipeline or timeout. The live arm also requires its grader to pass; -SIGPIPE is accepted only when that grader has completed its measurement window. +after its measurement window, the reader drains through source EOF so neither a +broken pipe nor another exporter failure needs to be ignored. `--analyze-only` needs only TSDuck + Python, so you can point it at any captured subscriber output: diff --git a/test/ts/process.sh b/test/ts/process.sh index 436ca28e9b0..fe61f9863c5 100644 --- a/test/ts/process.sh +++ b/test/ts/process.sh @@ -9,3 +9,11 @@ ts_wait() { return 1 fi } + +# A finite measurement must not break its producer's pipe before source EOF. +ts_drain() { + local status=0 + "$@" || status=$? + cat >/dev/null || return $? + return "$status" +} diff --git a/test/ts/process.test.sh b/test/ts/process.test.sh index 77cd42a3c82..06965ec56c5 100644 --- a/test/ts/process.test.sh +++ b/test/ts/process.test.sh @@ -25,5 +25,16 @@ for code in 0 42 124; do echo "control produced no partial output" >&2 exit 1 } + # The grader may stop early, but the writer must finish and the grader's + # failure must survive draining. No sleeps or wall-clock timeouts are needed. + set +e + # shellcheck disable=SC2016 # $1 belongs to the child shell. + printf 'window\n%1000000s' '' | ts_drain bash -c 'read -r line; exit "$1"' _ "$code" + statuses=("${PIPESTATUS[@]}") + set -e + [[ "${statuses[0]}" -eq 0 && "${statuses[1]}" -eq "$code" ]] || { + echo "draining changed writer/grader statuses: ${statuses[*]}" >&2 + exit 1 + } done echo "PASS process status: partial output cannot hide a failure or timeout" diff --git a/test/ts/run.sh b/test/ts/run.sh index 4f6401654e7..ee5e373fc78 100755 --- a/test/ts/run.sh +++ b/test/ts/run.sh @@ -382,7 +382,7 @@ grade_live() { set +e timeout -k 3 $((DURATION + 20)) \ "$MOQ" --connect "$URL" --broadcast "$BROADCAST" export ts ${EXPORT[@]+"${EXPORT[@]}"} 2>"$HARNESS_RUN/sub.log" | - python3 "$DIR/pcr-timing.py" --live --seconds "$DURATION" --release-pct-max 1 $STRICT \ + ts_drain python3 "$DIR/pcr-timing.py" --live --seconds "$DURATION" --release-pct-max 1 $STRICT \ ${SCHEDULE[@]+"${SCHEDULE[@]}"} ${PASSTHRU[@]+"${PASSTHRU[@]}"} >"$HARNESS_RUN/timing.out" 2>&1 printf '%s\n' "${PIPESTATUS[0]} ${PIPESTATUS[1]}" >"$HARNESS_RUN/timing.rc" } @@ -464,11 +464,12 @@ if [[ -n "$LIVE" ]]; then read -r EXPORT_RC GRADE_RC <"$HARNESS_RUN/timing.rc" echo cat "$HARNESS_RUN/timing.out" - # SIGPIPE is expected when the live grader closes a completed measurement - # window. A timeout or another failure is not a successful export. - if [[ "$EXPORT_RC" -ne 0 && "$EXPORT_RC" -ne 141 ]]; then + # The grader drains after its measurement window, so no broken pipe or + # timeout is an expected successful export. + printf 'exporter: %s\ngrader: %s\n' "$EXPORT_RC" "$GRADE_RC" >>"$HARNESS_RUN/process-status.log" + if [[ "$EXPORT_RC" -ne 0 ]]; then echo >&2 - echo "error: the exporter exited $EXPORT_RC before the window closed" >&2 + echo "error: the exporter exited $EXPORT_RC" >&2 dump_logs exit 1 fi From eba0751bb7928e47c3d9fa8a1d79344a790459fb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 14:27:10 -0700 Subject: [PATCH 07/17] Claim TS pair acquisition quest Co-Authored-By: Codex From 75f3baf31e4a1246f25a0c4227fc599e15b36ff7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 14:37:29 -0700 Subject: [PATCH 08/17] fix(test): measure TS pair alignment on the acquired PCR timeline Co-Authored-By: Codex --- .github/workflows/interop.yml | 5 + .github/workflows/nightly.yml | 4 + quest/m1/README.md | 1 - quest/m1/initial-catalog-routing.md | 2 +- quest/m1/ts-pair-acquisition.md | 60 ---- test/justfile | 4 + test/ts/README.md | 102 +++---- test/ts/run.sh | 3 +- test/ts/table-anchor.py | 445 +++++++++++----------------- test/ts/table-anchor.test.py | 125 ++++++++ 10 files changed, 358 insertions(+), 393 deletions(-) delete mode 100644 quest/m1/ts-pair-acquisition.md create mode 100644 test/ts/table-anchor.test.py diff --git a/.github/workflows/interop.yml b/.github/workflows/interop.yml index b00cf7a19af..1ac645ff8cb 100644 --- a/.github/workflows/interop.yml +++ b/.github/workflows/interop.yml @@ -141,6 +141,11 @@ jobs: env: MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs + - name: TS pair controls + if: ${{ !cancelled() && steps.setup.outcome == 'success' }} + run: nix develop --command just test ts-pair-controls + shell: bash -leo pipefail {0} + # Round-trips a PCR-paced TS through a relay and checks the subscriber's # `export ts` output with TSDuck (tsp/tsanalyze from the nix devShell). - name: TS compliance diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index a98a76d8594..0392eb7ca8c 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -217,6 +217,10 @@ jobs: if: ${{ !cancelled() }} run: nix develop --command just rs c-tests + - name: TS table alignment + if: ${{ !cancelled() }} + run: nix develop --command just test ts --pair + # The TS exporter's PCR release timing. Everything else about the stream # can be graded from a capture, but *when* the bytes were handed over # cannot: the grader has to sit on the exporter's stdout for the length of diff --git a/quest/m1/README.md b/quest/m1/README.md index e55b407d71c..18e9c19e8a4 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -67,7 +67,6 @@ session outcomes, and embedder refusals inside session outcomes. - [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 carries an untimed track in both languages; lite-05/06 write send time - [A watch and publish release ships assets()](/quest/m1/assets-release.md) - the release that lets the sites host the worklets - [Dogfood hosted worklets](/quest/m1/dogfood-assets.md) - the moq.pro dashboard hosts the worklets and calls `assets()` after the release -- [TS pair acquisition](/quest/m1/ts-pair-acquisition.md) - fix the clean baseline and prove the grader's controls before #5328 enables its scheduled hard gate - [Initial catalog routing](/quest/m1/initial-catalog-routing.md) - a usable fallback does not strand initial catalog acquisition with a spurious Unroutable - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [END_OF_TRACK placement](/quest/m1/ietf-end-of-track-placement.md) - END_OF_TRACK rides the upstream's Location, and Rust's header stops claiming END_OF_GROUP diff --git a/quest/m1/initial-catalog-routing.md b/quest/m1/initial-catalog-routing.md index a747ec0997e..1b26a0d9fef 100644 --- a/quest/m1/initial-catalog-routing.md +++ b/quest/m1/initial-catalog-routing.md @@ -59,4 +59,4 @@ decision rather than an invented retry or compatibility layer. - [Transport upgrade](/quest/m1/transport-upgrade/README.md) - existing handover and epoch contracts - [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - previously captured only a short suffix, not zero-packet initial acquisition - [Replacement catalog linger](/quest/m1/ts-follow-catalog-bound.md) - bounds a midstream stitch, not initial discovery -- [Pair acquisition](/quest/m1/ts-pair-acquisition.md) - independent grader/mux finding +- [Pair validation](/test/ts/README.md#table-anchoring-table-anchorpy) - independent grader/acquisition checks diff --git a/quest/m1/ts-pair-acquisition.md b/quest/m1/ts-pair-acquisition.md deleted file mode 100644 index c630b83afdc..00000000000 --- a/quest/m1/ts-pair-acquisition.md +++ /dev/null @@ -1,60 +0,0 @@ -# [M] TS pair validation distinguishes acquisition from persistent misalignment - -## Goal - -The late-joining pair test accepts supported clean acquisition and detects -persistent table misalignment without mistaking interleave for emission time. -Its positive baseline and negative controls run in CI before the scheduled -hard gate in #5328 is merged. - -## Plan - -Decided with the maintainer on 2026-10-11: fix the finding, not just diagnose -it. Keep this separate from initial catalog routing and the broader redundant -TS coordinate/API planning in #5327. Fix the baseline before #5328 merges; -do not temporarily make the pair test non-gating. - -The validation run in #5328 used `just test ts --pair`: two exports of one -45-second generated broadcast, the second joining five seconds late. Both -exporters and the publisher exited successfully. PAT/PMT agreement was 100%; -SDT had 20 common placements out of 23 in the union, or 86.96%, below the -existing 90% gate over 41 seconds of shared media. The mismatches were near -acquisition; 20 consecutive later SDT placements matched. These observations -do not establish permanent phase drift. - -`test/ts/table-anchor.py` tags a table with the next PES PTS and infers a -timer-driven phase difference from median cadence. Check those assumptions -against the current slot scheduler and the capture's actual table placement. -Correct the grader, the mux, or both as the evidence requires; do not assume -the exporter is defective because that diagnostic says so. - -Define and document the acquisition behavior being graded for the generated -profile. Any distinction between acquisition tables and recurring tables must -follow an observable stream condition with automated evidence. Do not ignore -an arbitrary number of seconds, reduce the agreement or coverage thresholds, -or shrink the shared window to the surviving table emissions. Keep missing -expected tables and inadequate evidence distinguishable from a passing check. - -Add deterministic controls for supported late acquisition and changed PES -interleave, plus persistent phase offset, drift and table loss within shared -media. The controls must fail for their intended reasons and must prevent a -grader that accepts everything from passing. Keep table carriage, cadence and -packet identity separate: this quest does not promise byte-identical TS legs, -introduce `--sync`, or claim ST 2022-7 compatibility. - -Run the short controls on relevant PRs and the real pair arm at least nightly, -using shared `just test` recipes. Preserve captures and reports on failure. -Require a passing clean pair baseline and failing negative controls before -unblocking #5328; a green PR run that skips the pair arm is not that evidence. -Update `test/ts/README.md` and misleading grader comments/diagnoses inline. - -Public API and wire: none expected. New shared-coordinate or synchronization -contracts remain with the broader alignment planning. - -## Related - -- [Validation PR #5328](https://github.com/moq-dev/moq/pull/5328) - its scheduled pair gate must wait for this fix -- [Pair gate review](https://github.com/moq-dev/moq/pull/5328#discussion_r4243103700) - a known-failing baseline would generate daily alerts -- [TS roadmap #5327](https://github.com/moq-dev/moq/pull/5327) - owns the broader redundant alignment planning -- [Deterministic TS legs](/quest/m2/ts-hitless.md) - packet identity requires a separate contract and proof -- [Initial catalog routing](/quest/m1/initial-catalog-routing.md) - distinct acquisition failure found by the same validation work diff --git a/test/justfile b/test/justfile index ad8a62bfa7f..98512d40741 100644 --- a/test/justfile +++ b/test/justfile @@ -121,6 +121,10 @@ drill-sensitivity *args: ts *args: ./ts/run.sh "$@" +# Table-placement grader: acquisition, interleave, phase, drift and missing-table controls. +ts-pair-controls: + python3 ./ts/table-anchor.test.py + # Round-trip the EIT fixtures: the sparse schedule must survive and # pending-version sections must be dropped at import. [positional-arguments] diff --git a/test/ts/README.md b/test/ts/README.md index 47e495ebe6c..52d872010ed 100644 --- a/test/ts/README.md +++ b/test/ts/README.md @@ -389,76 +389,52 @@ checks to hard. ## Table anchoring (`table-anchor.py`) -Every other check here grades one stream in isolation, and no single stream can -answer the question this one asks: **are a table's emission points a property of -the broadcast, or of the exporter that happened to emit them?** - -It matters because two exporters of one broadcast are how redundancy is built. A -receiver merging two legs, or cutting from one to the other, needs them to agree -about where the tables sit. If each leg emits to its own clock the two outputs -are not interchangeable, and the divergence is permanent rather than something a -longer run settles. - -One stream cannot show this because a timer and an anchor look identical from -inside: both produce a table every so often. Only a second leg, joining at a -different moment, separates them. +Two exporters of the same broadcast must converge after acquisition. `--pair` +starts the second subscriber five seconds late and checks table placement on the +shared PCR timeline. This measures table alignment; it does not establish packet +identity, `--sync`, or ST 2022-7 compatibility. ```bash -just test ts --pair # round-trip twice, grade the pair -just test ts --pair --pair-join 20 --duration 60 # join the second leg 20s in -./table-anchor.py a.ts b.ts # grade two captures you already have +just test ts --pair +just test ts --pair --pair-join 20 --duration 60 +just test ts-pair-controls +./table-anchor.py a.ts b.ts --report-json pair.json ``` -The second subscriber joins late on purpose. Two exporters started together can -agree on a cadence by having started together, which is the confound; a leg that -joins mid-broadcast has to derive its emission points from the media, because it -has no shared history to derive them from. - -**What gets graded is the overlap, not the run**, so `--pair` defaults to a 45 s -duration rather than the 20 s the other arms use, and refuses a run leaving less -than 25 s of it. At 20 s with a 5 s join the legs share 15 s, which is about -seven SDT emissions against a floor of eight — the table this mode exists to -check would quietly drop to report-only. Raise `--duration` when you raise -`--pair-join`. `--pair` cannot be combined with `--live`: they grade different -things. - -The measurement is the PTS of the frame each table was emitted against. `export -ts` writes the tables that are due and then the frame's PES packets into one -buffer, so the first PES header after a table gives the PTS of the frame that -triggered it. **Agreement** is the emission points both legs used over those -either used, counted only inside the media time the two captures share, so a -late join costs nothing. 100 % means the emission points are a function of the -broadcast. +The CBR slot scheduler interleaves tables and PES, so the next PES timestamp does not +identify when a table was sent. The grader interpolates each table's packet +position between PCR samples. A match allows one packet of placement quantization +plus one PCR tick of rounding. Matching is one-to-one, so duplicates count. + +The comparison starts at the later leg's **second video random-access PES**, +after its first acquired group, matching the exporter's late-join convergence +regression. This boundary comes from media markers, never from the tables under +test or a fixed time allowance. The end is the earlier leg's final PCR. Tables +that disappear cannot shorten their own scoring window. Captures with multiple +program clocks, variable bitrate, midstream PCR discontinuities, missing acquisition markers, or +broken packet framing are refused. PCR wrap is supported. + +The default run lasts 45 seconds and requires 20 seconds of acquired shared +clock. PAT, declared PMTs, and every observed non-clock table must each have at +least eight emissions **on both legs**, with at least 90% agreement. Missing or +sparse tables fail rather than quietly becoming report-only. TDT/TOT remains +informational because its content follows wall-clock time; `--strict` grades it +as well. `--source` also requires the source's SI tables, catching loss on both legs. These checks do not diagnose a process timer from cadence alone. | Option | Meaning | |---|---| -| `--min-agreement` | percent of shared emission points required per table (default 90) | -| `--min-emissions` | a table with fewer emissions in the overlap is reported, not graded (default 8) | -| `--min-window` | seconds of shared media required before any verdict is given (default 20) | -| `--strict` | fail on report-only tables too | -| `--report-json` | write the full report | - -`--min-window` exists because a capture that came up short is the commonest way -this grades clean: too little shared media puts the slower tables under the -emission floor, and a table reported without a verdict reads as a pass. The -window is taken from the media the two captures carry, **not** from the -emissions being scored — deriving it from the emissions is itself a false pass, -since a leg that stops emitting a table halfway through would pull the upper -bound back to its own last emission and score its desertion as 100 %. - -Two things are deliberately not graded. A table seen on only one leg is reported -without a verdict, because that is a carriage question rather than an anchoring -one. TDT/TOT is report-only whatever its agreement, since it carries wall-clock -time and is *supposed* to track a clock rather than the media. - -Where both legs run the same period at different phase, the report says so -rather than leaving a percentage to interpret: - -``` - FAIL SDT/BAT 10 10 19 1 5.26% - both legs emit every 2.000s, 0.480s out of phase - -> a timer started with the exporter, not an anchor in the media -``` +| `--min-agreement` | Required percentage of matched emissions (default 90) | +| `--min-emissions` | Required emissions per leg and table (default 8) | +| `--min-window` | Required seconds of acquired shared clock (default 20) | +| `--source` | Require the source's SI tables on both legs | +| `--strict` | Grade TDT/TOT too | +| `--report-json` | Save the window, tolerance, counts, and verdicts | + +`just test ts-pair-controls` runs deterministic transport fixtures through the +actual grader. Identical streams, different PES interleave, late acquisition with +bootstrap tables, and PCR wrap must pass. Persistent phase shifts, accumulating +drift, disappearing/absent/sparse tables, and insufficient shared media must fail. +Controls run in PR CI; the real pair runs nightly and on manual TS validation. ## Open GOP (`open-gop.py`) diff --git a/test/ts/run.sh b/test/ts/run.sh index 530449b453d..25222be2273 100755 --- a/test/ts/run.sh +++ b/test/ts/run.sh @@ -510,7 +510,8 @@ if [[ -n "$PAIR" ]]; then fi echo "### captured $(wc -c <"$SUB_TS" | tr -d ' ') + $(wc -c <"$SUB_B_TS" | tr -d ' ') bytes -> comparing table anchors" echo - if ! python3 "$DIR/table-anchor.py" "$SUB_TS" "$SUB_B_TS" $STRICT \ + if ! python3 "$DIR/table-anchor.py" "$SUB_TS" "$SUB_B_TS" --source "$SRC_TS" \ + --report-json "$HARNESS_RUN/table-anchor.json" $STRICT \ ${PASSTHRU[@]+"${PASSTHRU[@]}"}; then echo >&2 echo "error: table anchor analysis failed (see round-trip logs below)" >&2 diff --git a/test/ts/table-anchor.py b/test/ts/table-anchor.py index 58f1d3c9e1b..ffb4844fad6 100644 --- a/test/ts/table-anchor.py +++ b/test/ts/table-anchor.py @@ -1,44 +1,31 @@ #!/usr/bin/env python3 -"""Grade whether a table's emission points belong to the broadcast or to the process. - - ./table-anchor.py a.ts b.ts - -`export ts` writes a frame by emitting any due tables and then that frame's PES -packets into one buffer, so the first PES header after a table belongs to the -frame that triggered it. Tagging each table emission with that frame's PTS says -what two exporters of one broadcast cannot otherwise be asked: whether they put -a table in the same place in *media* time, or merely at the same rate. - -That distinction is the whole property. A cadence driven by the exporter's own -clock still produces a correct stream on each leg and still emits at the right -average interval; it just emits at different points, so the legs are two -transports of one programme rather than one transport sent twice. Anything that -merges or switches between them - an ST 2022-7 receiver, a standby packager, a -late joiner expected to converge on a running peer - needs the second. - -Why not compare bytes. Two legs order the same frames differently, because the -exporter emits whichever track has a frame in hand and arrival order is a -property of the network path rather than of the broadcast. A packet-by-packet -walk desynchronises on that regardless of where the tables land, so it cannot -answer this question even in principle. Indexing by PTS is immune to it, which -isolates the cadence from the interleave. - -Comparison is restricted to the media-time window the two legs share, so a late -joiner is graded on the overlap rather than penalised for the start it missed. - -Exit status is 0 when every hard check passes, 1 otherwise. `--strict` promotes -the report-only (shape) checks to hard. +"""Compare table placement on the PCR timeline after a complete acquisition group. + +Tables and PES are interleaved by the slot scheduler. The next PES PTS is not the +emission time: interpolate packet position between PCRs instead. Start at the +second video random-access PES on each leg, when its first acquired group is out, +as the exporter's late-join convergence tests do. The end comes from the clock, +never from the tables, so disappearing tables cannot shorten their own test. + +This checks table placement, not packet identity or hitless switching. It supports +one continuous program clock and video carrying random_access_indicator; refuse +captures that cannot establish those conditions rather than guess a grace period. """ import argparse import json import sys +from bisect import bisect_right from collections import defaultdict +from dataclasses import dataclass +from importlib import import_module + +parse_pcr = import_module("pcr-timing").parse_pcr +CLOCK = 27_000_000 +WRAP = (1 << 33) * 300 PKT = 188 SYNC = 0x47 -HARD, SHAPE = "hard", "shape" -NULL_PID = 0x1FFF # PIDs whose table is fixed by the standard. A PMT sits on a PID the PAT chooses, # so it is discovered rather than listed. @@ -57,16 +44,16 @@ def packets(path): - """188-byte packets from a TS file, resynchronising on the first stable sync run.""" + """Read the exporter's complete 188-byte packets, refusing broken framing.""" with open(path, "rb") as f: data = f.read() - start = 0 - while start + PKT * 8 < len(data): - if all(data[start + i * PKT] == SYNC for i in range(8)): - break - start += 1 - end = len(data) - (len(data) - start) % PKT - return (data[i : i + PKT] for i in range(start, end, PKT)) + if not data or len(data) % PKT: + raise ValueError(f"{path}: empty or truncated transport") + for at in range(0, len(data), PKT): + pkt = data[at : at + PKT] + if pkt[0] != SYNC or pkt[1] & 0x80 or not pkt[3] & 0x30: + raise ValueError(f"{path}: malformed packet at {at}") + yield pkt def pid_of(pkt): @@ -97,13 +84,7 @@ def parse_pts(pkt): if len(p) < 14 or not (p[7] & 0x80): # PTS_DTS_flags return None b = p[9:14] - return ( - ((b[0] >> 1) & 0x07) << 30 - | b[1] << 22 - | ((b[2] >> 1) & 0x7F) << 15 - | b[3] << 7 - | ((b[4] >> 1) & 0x7F) - ) + return ((b[0] >> 1) & 0x07) << 30 | b[1] << 22 | ((b[2] >> 1) & 0x7F) << 15 | b[3] << 7 | ((b[4] >> 1) & 0x7F) def pmt_pids(path): @@ -111,7 +92,7 @@ def pmt_pids(path): Sections are reassembled across packets. A PAT carrying more than about forty programmes does not fit in one packet, and stopping at the packet boundary would - silently find only the programmes that happened to land in the first one — which + silently find only the programmes that happened to land in the first one; which on a full multiplex is a subset, with no error to say so. """ found = set() @@ -150,242 +131,172 @@ def pmt_pids(path): return found -def anchors(path, pmts): - """Frame PTS values each table was emitted at, and the capture's media bounds. +@dataclass +class Capture: + tables: dict + start: float + end: float + positions: list[int] + clocks: list[int] + + def rate(self, window): + """Packet duration in the shared window, refusing non-CBR interpolation.""" + lo = max(0, bisect_right(self.clocks, window[0]) - 1) + hi = bisect_right(self.clocks, window[1]) + clocks, positions = self.clocks[lo:hi], self.positions[lo:hi] + if len(clocks) < 2: + raise ValueError("insufficient shared PCR samples") + ticks = (clocks[-1] - clocks[0]) / (positions[-1] - positions[0]) + # Two integer PCR samples bound the rounding error in the fitted slope. + if any(abs(t - clocks[0] - (i - positions[0]) * ticks) > 2 for i, t in zip(positions, clocks)): + raise ValueError("pair grading requires a constant-rate shared PCR/packet timeline") + return ticks - The bounds are returned alongside because the scoring window has to come from - the media the capture covers, not from the emissions themselves — see - `agreement()`. - """ - out = defaultdict(list) - pending = [] - lo = hi = None + +def anchors(path, pmts): + """Table positions and the independently observed, acquired clock window.""" + clocks, positions = [], [] + tables, access = defaultdict(list), defaultdict(list) + clock_pid = None watched = set(WELL_KNOWN) | pmts - for pkt in packets(path): + for index, pkt in enumerate(packets(path)): pid = pid_of(pkt) - if pid == NULL_PID: - continue - if pid in watched: - if pkt[1] & 0x40: - pending.append(pid) - continue - pts = parse_pts(pkt) - if pts is None: - continue - lo = pts if lo is None or pts < lo else lo - hi = pts if hi is None or pts > hi else hi - if pending: - for table in pending: - out[table].append(pts) - pending = [] - return out, (lo, hi) + pcr = parse_pcr(pkt) + if pcr is not None: + if clock_pid is not None and pid != clock_pid: + raise ValueError("pair grading requires a single program clock") + clock_pid = pid + if clocks: + if pkt[5] & 0x80: + raise ValueError("PCR discontinuity inside pair capture") + delta = (pcr - clocks[-1]) % WRAP + if not 0 < delta < WRAP // 2: + raise ValueError("non-increasing program clock") + pcr = clocks[-1] + delta + clocks.append(pcr) + positions.append(index) + if pid in watched and pkt[1] & 0x40: + tables[pid].append(index) + off = payload_offset(pkt) + if parse_pts(pkt) is not None and 0xE0 <= pkt[off + 3] <= 0xEF: + # Record the PID even when every random-access flag is missing. + points = access[pid] + if pkt[3] & 0x20 and pkt[4] and pkt[5] & 0x40: + points.append(index) + if len(clocks) < 2 or not access or any(len(v) < 2 for v in access.values()): + raise ValueError("need PCR and two video random-access points per video PID") + + def at(index): + i = bisect_right(positions, index) - 1 + if i < 0 or i >= len(clocks) - 1: + raise ValueError("acquisition point is outside the measured PCR window") + return clocks[i] + (index - positions[i]) * (clocks[i + 1] - clocks[i]) / (positions[i + 1] - positions[i]) + + start = max(at(v[1]) for v in access.values()) + return Capture( + {pid: [at(i) for i in points if positions[0] <= i < positions[-1]] for pid, points in tables.items()}, + start, + clocks[-1], + positions, + clocks, + ) def name(pid, pmts): - if pid in WELL_KNOWN: - return WELL_KNOWN[pid] - return f"PMT {pid:#06x}" if pid in pmts else f"PID {pid:#06x}" - - -def agreement(a_pts, b_pts, window): - """Share of emission points the legs share, over the media time they both cover. + return WELL_KNOWN.get(pid, f"PMT {pid:#06x}" if pid in pmts else f"PID {pid:#06x}") - The window is the media both captures carry, and it must come from the captures - rather than from the emissions being scored. Deriving it from the emissions - instead is a false pass: a leg that stops emitting a table halfway through pulls - the upper bound back to its own last emission, so the partner's later emissions - fall outside the window and the desertion scores 100 %. - Returns None when the captures do not overlap in media time at all, which is a - different answer from "they overlap and disagree" and must not be scored as 0. - """ +def agreement(a, b, window, tolerance): + """One-to-one matches within one packet, retaining losses and duplicates.""" lo, hi = window - if lo is None or hi is None or lo > hi: - return None - oa = {t for t in a_pts if lo <= t <= hi} - ob = {t for t in b_pts if lo <= t <= hi} - union = oa | ob - if not union: - return None - shared = oa & ob - return len(oa), len(ob), len(union), len(shared), 100.0 * len(shared) / len(union) - - -def cadence(pts): - """The period a leg emits on, in seconds, or None if it keeps no steady one. - - "Steady" has to tolerate a few odd gaps rather than demand none: a leg that - re-phases once on joining, or drops an emission, is still running a timer, and a - spread test over the extremes would miss exactly the case worth catching. So the - period is the median gap and the leg counts as steady when most gaps sit on it. - """ - pts = sorted(pts) - if len(pts) < 4: - return None - gaps = sorted((pts[i + 1] - pts[i]) / 90000.0 for i in range(len(pts) - 1)) - period = gaps[len(gaps) // 2] - if period <= 0: - return None - on_period = sum(1 for g in gaps if abs(g - period) <= 0.05 * period) - return period if on_period >= 0.6 * len(gaps) else None - - -def phase(pts, period): - """Where a leg's grid sits inside one period, as a robust median. - - Taken relative to the first point so that a grid straddling the modulus does not - split into two clusters and median to a value in neither. - """ - base = min(pts) / 90000.0 - offs = sorted(((t / 90000.0 - base) % period) for t in pts) - return offs[len(offs) // 2] + base - - -def diagnose(a_pts, b_pts): - """Say why two legs disagree, when the shape of the disagreement is legible. - - The case worth naming is a table both legs emit on the same steady period but at - different phase: that is a timer started when the exporter started, and no amount - of running time will bring the two legs back together. - """ - pa, pb = cadence(a_pts), cadence(b_pts) - if pa is None or pb is None or abs(pa - pb) > 0.05 * max(pa, pb): - return None - offset = abs(phase(a_pts, pa) - phase(b_pts, pa)) % pa - offset = min(offset, pa - offset) - if offset <= 0.01 * pa: - return None - return ( - f"both legs emit every {pa:.3f}s, {offset:.3f}s out of phase " - f"-> a timer started with the exporter, not an anchor in the media" - ) + a = [t for t in a if lo <= t <= hi] + b = [t for t in b if lo <= t <= hi] + i = j = shared = 0 + while i < len(a) and j < len(b): + if abs(a[i] - b[j]) <= tolerance: + shared += 1 + i += 1 + j += 1 + elif a[i] < b[j]: + i += 1 + else: + j += 1 + union = len(a) + len(b) - shared + return len(a), len(b), union, shared, 100 * shared / union if union else 0 def main(): - ap = argparse.ArgumentParser(description="Grade table emission points across two exporters.") - ap.add_argument("a", help="TS captured from the first exporter") - ap.add_argument("b", help="TS captured from the second exporter") - ap.add_argument( - "--min-agreement", - type=float, - default=90.0, - help="percent of shared emission points required per table (default 90)", - ) - ap.add_argument( - "--min-emissions", - type=int, - default=8, - help="a table with fewer emissions in the overlap is reported, not graded (default 8)", - ) - ap.add_argument( - "--min-window", - type=float, - default=20.0, - help="seconds of shared media required before any verdict is given (default 20)", - ) - ap.add_argument("--strict", action="store_true", help="fail on shape checks too") - ap.add_argument("--report-json", help="write the full report here") + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("a") + ap.add_argument("b") + ap.add_argument("--min-agreement", type=float, default=90.0) + ap.add_argument("--min-emissions", type=int, default=8) + ap.add_argument("--min-window", type=float, default=20.0) + ap.add_argument("--source", help="require the source's non-clock SI tables on both legs") + ap.add_argument("--strict", action="store_true", help="also grade clock tables") + ap.add_argument("--report-json") args = ap.parse_args() - - pmts = pmt_pids(args.a) | pmt_pids(args.b) - a, (a_lo, a_hi) = anchors(args.a, pmts) - b, (b_lo, b_hi) = anchors(args.b, pmts) - if not a and not b: - print("error: neither capture carries a table on a known PID", file=sys.stderr) - return 1 - if None in (a_lo, a_hi, b_lo, b_hi): - print("error: a capture carries no PTS, so there is no media time to compare in", file=sys.stderr) - return 1 - # The media both captures carry. Every table is scored inside this one window, so a - # table one leg abandons is scored against the partner's emissions rather than - # silently shrinking the window to hide them. - window = (max(a_lo, b_lo), min(a_hi, b_hi)) - shared_s = (window[1] - window[0]) / 90000.0 - print(f"### shared media window: {shared_s:.1f}s") - # A window too short to carry the slower tables would put them under the emission - # floor and report them without a verdict, which reads as a pass. Refuse the run - # instead: a capture that came up short is the commonest way this grades clean. - if shared_s < args.min_window: - print( - f"error: the captures share only {shared_s:.1f}s of media, below the " - f"{args.min_window:.0f}s needed for a verdict", - file=sys.stderr, - ) - print(" run for longer, join earlier, or lower --min-window deliberately", file=sys.stderr) - return 1 - - rows = [] - for pid in sorted(set(a) | set(b)): - label = name(pid, pmts) - scored = agreement(a.get(pid, []), b.get(pid, []), window) - if scored is None: - rows.append((pid, label, len(a.get(pid, [])), len(b.get(pid, [])), None, None, "one leg only")) - continue - na, nb, union, shared, pct = scored - # A table emitted a handful of times cannot separate "anchored to the media" - # from "coincidence", so it is reported without a verdict rather than graded - # on a sample too small to mean anything. - severity = SHAPE if pid in NOT_GRADED or union < args.min_emissions else HARD - rows.append((pid, label, na, nb, union, shared, pct, severity)) - - width = max(len(r[1]) for r in rows) - print(f"### table anchor report - {len(rows)} table(s)") - print() - print(f" {'':4} {'table':<{width}} {'A':>7} {'B':>7} {'either':>7} {'both':>7} agreement") - failed = 0 - report = [] - for row in rows: - if len(row) == 7: - pid, label, na, nb, _, _, note = row - print(f" {'--':4} {label:<{width}} {na:>7,} {nb:>7,} {'-':>7} {'-':>7} {note}") - report.append({"pid": pid, "table": label, "a": na, "b": nb, "graded": False, "note": note}) - continue - pid, label, na, nb, union, shared, pct, severity = row - ok = pct >= args.min_agreement - if ok: - verdict = "PASS" - elif severity == HARD or args.strict: - verdict = "FAIL" - failed += 1 - else: - verdict = "WARN" - print( - f" {verdict:4} {label:<{width}} {na:>7,} {nb:>7,} {union:>7,} {shared:>7,} {pct:8.2f}%" - ) - why = None if ok else diagnose(a.get(pid, []), b.get(pid, [])) - if why: - print(f" {'':4} {'':<{width}} {why}") - report.append( - { - "pid": pid, - "table": label, - "a": na, - "b": nb, - "either": union, - "both": shared, - "agreement_pct": round(pct, 4), - "severity": severity, - "graded": True, - "ok": ok, - "diagnosis": why, - } + try: + pmts = pmt_pids(args.a) | pmt_pids(args.b) + if not pmts: + raise ValueError("no PAT program mapping") + expected = {pid_of(pkt) for pkt in packets(args.source) if pid_of(pkt) in WELL_KNOWN} if args.source else set() + a, b = anchors(args.a, pmts), anchors(args.b, pmts) + # Captures may start on opposite sides of the 33-bit wrap. + shift = round((a.start - b.start) / WRAP) * WRAP + b.start += shift + b.end += shift + b.clocks = [t + shift for t in b.clocks] + b.tables = {pid: [t + shift for t in points] for pid, points in b.tables.items()} + window = max(a.start, b.start), min(a.end, b.end) + shared_s = (window[1] - window[0]) / CLOCK + if shared_s < args.min_window: + raise ValueError(f"only {shared_s:.3f}s of acquired shared clock; need {args.min_window}s") + tolerance = max(a.rate(window), b.rate(window)) + 1 + rows = [] + # PAT and its declared PMTs are mandatory, even when absent after acquisition. + for pid in sorted({0} | pmts | expected | set(a.tables) | set(b.tables)): + na, nb, union, shared, pct = agreement(a.tables.get(pid, []), b.tables.get(pid, []), window, tolerance) + graded = pid not in NOT_GRADED or args.strict + enough = min(na, nb) >= args.min_emissions + ok = enough and pct >= args.min_agreement + rows.append( + dict( + pid=pid, + table=name(pid, pmts), + a=na, + b=nb, + either=union, + both=shared, + agreement_pct=round(pct, 4), + graded=graded, + ok=ok, + note=None if enough else "insufficient emissions", + ) + ) + report = dict( + window_seconds=[t / CLOCK for t in window], + shared_seconds=shared_s, + tolerance_ticks=tolerance, + min_agreement=args.min_agreement, + tables=rows, ) - - print() - print( - "agreement = emission points both legs used, over those either used, inside the\n" - "media time they share. 100% means the emission points are a function of the\n" - "broadcast; a low figure with a healthy count on both legs means each leg is\n" - "emitting to its own clock." - ) - print() - print(f"{failed} check(s) failed" if failed else "all checks passed") - + print(f"### acquired shared clock: {shared_s:.3f}s") + for row in rows: + verdict = "PASS" if row["ok"] else "FAIL" if row["graded"] else "INFO" + print( + f"{verdict}: {row['table']}: {row['both']}/{row['either']} ({row['agreement_pct']:.2f}%)" + + (f"; {row['note']}" if row["note"] else "") + ) + failed = any(row["graded"] and not row["ok"] for row in rows) + except ValueError as error: + report = dict(error=str(error)) + print(f"error: {error}", file=sys.stderr) + failed = True if args.report_json: with open(args.report_json, "w") as f: - json.dump({"min_agreement": args.min_agreement, "tables": report}, f, indent=2) - - return 1 if failed else 0 + json.dump(report, f, indent=2) + return int(failed) if __name__ == "__main__": diff --git a/test/ts/table-anchor.test.py b/test/ts/table-anchor.test.py new file mode 100644 index 00000000000..cc1b05c16de --- /dev/null +++ b/test/ts/table-anchor.test.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""Exercise the pair grader through complete synthetic transport captures.""" + +import json +import subprocess +import sys +import tempfile +import unittest +from importlib import import_module +from pathlib import Path + +fixtures = import_module("tstd-controls") +PKT, PMT, VIDEO = fixtures.packet, 0x1000, 0x100 +CLOCK = 27_000_000 + + +def capture(path, *, start=0, phase=0, drift=0, loss=None, interleave=False, bootstrap=False, wrap=False): + pat = fixtures.section(0, 1, bytes.fromhex("0001f000")) + pmt = fixtures.section(2, 1, bytes.fromhex("e100f0001be100f000")) + sdt = fixtures.section(0x42, 1, bytes.fromhex("0001ff")) + # A packet per millisecond; PCR every 25 packets, independently of tables/PES. + events = {} + for at in range(7, 30000, 2000): + events[at + phase + (at // 2000) * drift] = 0x11 + if bootstrap: + events[start + 7] = 0x11 + with open(path, "wb") as out: + for at in range(start, 30001): + pid, payload, pusi, adaptation = 0x1FFF, b"", False, None + if at % 25 == 0: + ticks = (at * CLOCK // 1000 + (((1 << 33) * 300 - 10 * CLOCK) if wrap else 0)) % ((1 << 33) * 300) + base, ext = divmod(ticks, 300) + adaptation = b"\x10" + ((base << 15) | (0x3F << 9) | ext).to_bytes(6, "big") + pid = VIDEO + elif at % 500 == 2: + pid, payload, pusi = 0, b"\0" + pat, True + elif at % 500 == 3: + pid, payload, pusi = PMT, b"\0" + pmt, True + elif at in events: + pid, payload, pusi = events[at], b"\0" + sdt, True + elif at % 20 == 10: + # Changing which PES follows a table does not change table placement. + audio = interleave and at % 1000 != 10 + pid = 0x101 if audio else VIDEO + pts = (at + (600 if audio else 200)) * 90 + payload = bytes.fromhex("000001") + bytes([0xC0 if audio else 0xE0]) + payload += bytes.fromhex("0000808005") + fixtures.stamp(2, pts) + b"media" + pusi = True + adaptation = b"\x40" if not audio and at % 1000 == 10 else None + if pid == 0x11 and (loss == "all" or loss == "tail" and at > 14000 or loss == "sparse" and at % 4000 != 7): + pid, payload, pusi = 0x1FFF, b"", False + out.write(PKT(pid, 0, payload, pusi, adaptation)) + + +class Controls(unittest.TestCase): + def test_controls(self): + with tempfile.TemporaryDirectory(prefix="ts-pair-controls-") as directory: + directory = Path(directory) + a, b, report = directory / "a.ts", directory / "b.ts", directory / "report.json" + capture(a) + source = directory / "source.ts" + source.write_bytes(a.read_bytes()) + cases = [ + ("identical", {}, True), + ("different PES interleave", dict(interleave=True), True), + ("late acquisition and bootstrap", dict(start=3500, bootstrap=True, interleave=True), True), + ("persistent phase", dict(start=3500, phase=100), False), + ("drift", dict(start=3500, drift=10), False), + ("table disappears in shared media", dict(loss="tail"), False), + ("table absent", dict(loss="all"), False), + ("table absent on both legs", dict(loss="all"), False), + ("sparse emissions", dict(loss="sparse"), False), + ("sparse on both legs", dict(loss="sparse"), False), + ("insufficient shared media", dict(start=15000), False), + ] + for label, options, expected in cases: + with self.subTest(label=label): + capture( + a, + loss=options["loss"] if label in ("table absent on both legs", "sparse on both legs") else None, + ) + capture(b, **options) + result = subprocess.run( + [ + sys.executable, + str(Path(__file__).with_name("table-anchor.py")), + str(a), + str(b), + "--source", + str(source), + "--report-json", + str(report), + ], + capture_output=True, + text=True, + ) + self.assertEqual(result.returncode, 0 if expected else 1, result.stdout + result.stderr) + parsed = json.loads(report.read_text()) + if not expected and "error" not in parsed: + sdt = next(r for r in parsed["tables"] if r["pid"] == 0x11) + self.assertTrue(sdt["graded"]) + self.assertFalse(sdt["ok"]) + print(f"PASS control: {label}") + capture(a, wrap=True) + capture(b, start=11500, wrap=True) + # A shorter requested evidence window isolates wrap alignment; normal pair + # runs retain the default 20-second minimum. + result = subprocess.run( + [ + sys.executable, + str(Path(__file__).with_name("table-anchor.py")), + str(a), + str(b), + "--min-window", + "15", + ], + capture_output=True, + text=True, + ) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + print("PASS control: captures starting across PCR wrap") + + +if __name__ == "__main__": + unittest.main() From e5a2d7391c1cad9c41ed6c9ec5b0b7d2dd5efa33 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 14:39:35 -0700 Subject: [PATCH 09/17] chore: claim initial catalog routing quest Co-Authored-By: Codex From e04c0fa5164c91abc588dddb0002771b1f6276eb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 14:45:19 -0700 Subject: [PATCH 10/17] test(ts): require every source-declared PMT in pair captures Co-Authored-By: Codex --- test/ts/README.md | 2 +- test/ts/table-anchor.py | 2 ++ test/ts/table-anchor.test.py | 34 ++++++++++++++++++++++++++++++++-- 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/test/ts/README.md b/test/ts/README.md index 52d872010ed..1b1a5ebe592 100644 --- a/test/ts/README.md +++ b/test/ts/README.md @@ -419,7 +419,7 @@ clock. PAT, declared PMTs, and every observed non-clock table must each have at least eight emissions **on both legs**, with at least 90% agreement. Missing or sparse tables fail rather than quietly becoming report-only. TDT/TOT remains informational because its content follows wall-clock time; `--strict` grades it -as well. `--source` also requires the source's SI tables, catching loss on both legs. These checks do not diagnose a process timer from cadence alone. +as well. `--source` also requires the source's SI tables and PAT-declared PMTs, catching loss on both legs. These checks do not diagnose a process timer from cadence alone. | Option | Meaning | |---|---| diff --git a/test/ts/table-anchor.py b/test/ts/table-anchor.py index ffb4844fad6..75dee3beef4 100644 --- a/test/ts/table-anchor.py +++ b/test/ts/table-anchor.py @@ -238,6 +238,8 @@ def main(): args = ap.parse_args() try: pmts = pmt_pids(args.a) | pmt_pids(args.b) + if args.source: + pmts |= pmt_pids(args.source) if not pmts: raise ValueError("no PAT program mapping") expected = {pid_of(pkt) for pkt in packets(args.source) if pid_of(pkt) in WELL_KNOWN} if args.source else set() diff --git a/test/ts/table-anchor.test.py b/test/ts/table-anchor.test.py index cc1b05c16de..36af9a43179 100644 --- a/test/ts/table-anchor.test.py +++ b/test/ts/table-anchor.test.py @@ -14,8 +14,10 @@ CLOCK = 27_000_000 -def capture(path, *, start=0, phase=0, drift=0, loss=None, interleave=False, bootstrap=False, wrap=False): - pat = fixtures.section(0, 1, bytes.fromhex("0001f000")) +def capture( + path, *, start=0, phase=0, drift=0, loss=None, interleave=False, bootstrap=False, wrap=False, extra_program=False +): + pat = fixtures.section(0, 1, bytes.fromhex("0001f0000002f001" if extra_program else "0001f000")) pmt = fixtures.section(2, 1, bytes.fromhex("e100f0001be100f000")) sdt = fixtures.section(0x42, 1, bytes.fromhex("0001ff")) # A packet per millisecond; PCR every 25 packets, independently of tables/PES. @@ -36,6 +38,8 @@ def capture(path, *, start=0, phase=0, drift=0, loss=None, interleave=False, boo pid, payload, pusi = 0, b"\0" + pat, True elif at % 500 == 3: pid, payload, pusi = PMT, b"\0" + pmt, True + elif extra_program and at % 500 == 4: + pid, payload, pusi = 0x1001, b"\0" + fixtures.section(2, 2, bytes.fromhex("e100f0001be100f000")), True elif at in events: pid, payload, pusi = events[at], b"\0" + sdt, True elif at % 20 == 10: @@ -53,6 +57,32 @@ def capture(path, *, start=0, phase=0, drift=0, loss=None, interleave=False, boo class Controls(unittest.TestCase): + def test_source_declared_program(self): + with tempfile.TemporaryDirectory(prefix="ts-pair-source-") as directory: + directory = Path(directory) + source, output, report = directory / "source.ts", directory / "output.ts", directory / "report.json" + capture(source, extra_program=True) + capture(output) + result = subprocess.run( + [ + sys.executable, + str(Path(__file__).with_name("table-anchor.py")), + str(output), + str(output), + "--source", + str(source), + "--report-json", + str(report), + ], + capture_output=True, + text=True, + ) + self.assertEqual(result.returncode, 1, result.stdout + result.stderr) + missing = next(row for row in json.loads(report.read_text())["tables"] if row["pid"] == 0x1001) + self.assertTrue(missing["graded"]) + self.assertFalse(missing["ok"]) + self.assertEqual((missing["a"], missing["b"]), (0, 0)) + def test_controls(self): with tempfile.TemporaryDirectory(prefix="ts-pair-controls-") as directory: directory = Path(directory) From bc1dd1128df75ec38164305aae914991196c511a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 14:48:33 -0700 Subject: [PATCH 11/17] test(net): reproduce catalog acquisition across session replacement Co-Authored-By: Codex --- quest/m1/initial-catalog-routing.md | 27 +++++++++ rs/moq-net/tests/catalog_acquisition.rs | 75 +++++++++++++++++++++++++ 2 files changed, 102 insertions(+) create mode 100644 rs/moq-net/tests/catalog_acquisition.rs diff --git a/quest/m1/initial-catalog-routing.md b/quest/m1/initial-catalog-routing.md index a747ec0997e..077a26bca8a 100644 --- a/quest/m1/initial-catalog-routing.md +++ b/quest/m1/initial-catalog-routing.md @@ -51,6 +51,33 @@ Update comments and affected feature docs with the fix. No new guide, public API or wire change is expected; a newly required contract needs a maintainer decision rather than an invented retry or compatibility layer. +### Evidence and remaining decision + +Controlled reproduction in `rs/moq-net/tests/catalog_acquisition.rs`: hold the +publisher's initial catalog track request, introduce a cheaper second session to +the same publisher, observe the route change, then answer the held request. +Lite-06 returns `Unroutable`; Lite-07 with the publisher's explicit epoch acquires +and reads the catalog. The old session and publisher both remain alive. No +wall-clock timing, retry or transport teardown is needed. + +This follows the current identity contract: an epochless route replacement is a +new instance, and a pending `TrackInfoFetch` checks its captured +`RequestValidity` before accepting a response. Default negotiation excludes the +work-in-progress Lite-07 wire. #5284's closed-fallback case is different: this +reproduction has no closed session. The reproduction establishes a sufficient +ordering for the reported error, not proof that all retained historical captures +followed exactly that ordering. + +A fresh pair run also failed with the explicit replacement-broadcast error during +upgrade; a subsequent traced direct-WebTransport run passed table alignment. +The pair grader fix is #5331 and does not change routing semantics. + +The maintainer must settle the newly exposed contract before a production fix: +use epoch-capable negotiation for seamless acquisition/continuation, or permit a +fresh initial acquisition on a replacement before output begins. Existing handles +must never silently cross publisher identities. Keep #5328 draft; do not paper +over the default behavior with retries, `--stitch`, or disabled fallback. + ## Related - [Validation PR #5328](https://github.com/moq-dev/moq/pull/5328) - records the process failures and successful PR CI baseline diff --git a/rs/moq-net/tests/catalog_acquisition.rs b/rs/moq-net/tests/catalog_acquisition.rs new file mode 100644 index 00000000000..d9b15183c70 --- /dev/null +++ b/rs/moq-net/tests/catalog_acquisition.rs @@ -0,0 +1,75 @@ +//! Initial catalog acquisition while the same publisher gains a second session. + +mod support; + +use moq_net::{Error, Hop, Version, origin}; +use support::harness::{MockConnectOptions, MockPair, connect_mock}; + +fn origin(hop: u64) -> origin::Producer { + let (producer, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); + support::harness::spawn(driver); + producer +} + +async fn link(version: Version, from: &origin::Producer, to: &origin::Producer, cost: u64) -> MockPair { + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(from.consume()); + options.client_subscribe = Some(to.clone()); + options.cost = Some(cost); + connect_mock(options).await +} + +async fn acquisition(version: &str) { + let version: Version = version.parse().unwrap(); + let publisher = origin(1); + let subscriber = origin(2); + let broadcast = publisher.create_broadcast("live").unwrap(); + let mut dynamic = broadcast.dynamic(); + broadcast + .announce(origin::Route::default().with_epoch(moq_net::Epoch::mint())) + .unwrap(); + let _old = link(version, &publisher, &subscriber, 100).await; + let remote = subscriber.consume().routed_broadcast("live").await.unwrap(); + let mut announced = subscriber.consume().follow("live").unwrap(); + assert!(matches!( + announced.next().await, + Some(moq_net::announce::Event::Start(_)) + )); + let mut pending = Box::pin(remote.track("catalog.json").unwrap().subscribe(None)); + let request = match futures::future::select(Box::pin(dynamic.requested_track()), &mut pending).await { + futures::future::Either::Left((request, _)) => request.unwrap(), + _ => panic!("catalog resolved before the publisher answered"), + }; + // The publisher is still available and the old session remains alive. Holding + // TRACK_INFO makes the replacement happen during acquisition without sleeps. + let _new = link(version, &publisher, &subscriber, 1).await; + announced.next().await.unwrap(); + let track = request.accept(None); + let mut group = track.append_group().unwrap(); + group + .write_frame(moq_net::Timestamp::ZERO, b"catalog".to_vec()) + .unwrap(); + group.finish().unwrap(); + let result = pending.await; + if version == "moq-lite-07-wip".parse().unwrap() { + let mut subscription = result.expect("same epoch must acquire"); + let mut group = subscription.recv_group().await.unwrap().unwrap(); + assert_eq!(group.read_frame().await.unwrap().unwrap().payload.as_ref(), b"catalog"); + } else { + assert!( + matches!(result, Err(Error::Unroutable)), + "epochless replacement: {:?}", + result.err() + ); + } +} + +#[moq_net_sim::test] +async fn initial_catalog_acquisition_lite06() { + acquisition("moq-lite-06").await; +} + +#[moq_net_sim::test] +async fn initial_catalog_acquisition_lite07() { + acquisition("moq-lite-07-wip").await; +} From aab6a88e43c94320d35df7056276e1114dfaefd0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 14:59:02 -0700 Subject: [PATCH 12/17] test(net): retain strict identity during catalog acquisition Complete the routing investigation with deterministic epoch-capable success and epochless refusal controls. Production negotiation and retry policy remain unchanged. Co-Authored-By: Codex --- quest/m1/README.md | 1 - quest/m1/initial-catalog-routing.md | 89 ------------------------- quest/m1/ts-pair-acquisition.md | 1 - rs/moq-net/tests/catalog_acquisition.rs | 4 +- 4 files changed, 2 insertions(+), 93 deletions(-) delete mode 100644 quest/m1/initial-catalog-routing.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 004d82441be..df84f1f11ac 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -68,7 +68,6 @@ session outcomes, and embedder refusals inside session outcomes. - [A watch and publish release ships assets()](/quest/m1/assets-release.md) - the release that lets the sites host the worklets - [Dogfood hosted worklets](/quest/m1/dogfood-assets.md) - the moq.pro dashboard hosts the worklets and calls `assets()` after the release - [TS pair acquisition](/quest/m1/ts-pair-acquisition.md) - fix the clean baseline and prove the grader's controls before #5328 enables its scheduled hard gate -- [Initial catalog routing](/quest/m1/initial-catalog-routing.md) - a usable fallback does not strand initial catalog acquisition with a spurious Unroutable - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [END_OF_TRACK placement](/quest/m1/ietf-end-of-track-placement.md) - END_OF_TRACK rides the upstream's Location, and Rust's header stops claiming END_OF_GROUP - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone diff --git a/quest/m1/initial-catalog-routing.md b/quest/m1/initial-catalog-routing.md deleted file mode 100644 index 077a26bca8a..00000000000 --- a/quest/m1/initial-catalog-routing.md +++ /dev/null @@ -1,89 +0,0 @@ -# [M] Initial catalog acquisition survives a usable transport fallback - -## Goal - -Fix the initial `catalog.json` routing failure observed by TS validation, -with a deterministic regression and an automated round trip. A publisher -remaining available on a usable path must not leave the subscriber with a -spurious `Unroutable`; genuine refusal or disappearance still fails explicitly. - -## Plan - -Decided with the maintainer on 2026-10-11: reproduce and fix the cause, not -stop at diagnosis. This is independent of the pair grader's acquisition -contract. It is not a general transport redesign or a TS-specific retry path. - -Local live, EIT and HRD runs during #5328 failed their initial `catalog.json` -track-info request shortly after selecting WebSocket fallback. The exporter -exited one with `moq: unroutable`; the publisher continued to normal EOF and -exited zero. The live grader received zero packets. Live and EIT ran -concurrently; HRD reproduced later. Do not count these as three independent -sequential reproductions. The full PR TS job passed, including EIT and HRD; -its event skipped the live and pair arms. No transport root cause is established. - -Trace the failing initial request across publication discovery, catalog -availability, route selection and any session replacement. Preserve enough -session/broadcast/request identity to distinguish a real unavailable route -from a stale handle or handover race. First check current main and #5284: -that PR preserves a pending QUIC dial after an early WebSocket close, but -similar timing is not proof that it fixes this failure. If it owns the same -cause, add the missing regression there and verify the landed fix instead of -creating competing implementations. Otherwise keep the fix focused on the -layer that loses the usable route. - -Turn the observed ordering into a controlled regression before changing -behavior. Hold or order the relevant events and mock time where needed; -do not rely on load, sleeps, repeated reruns or larger timeouts to reproduce -or hide the failure. Cover the successful fallback/acquisition path and the -adjacent terminal path so a fix cannot silently retry genuine refusal or -keep a disappeared publication alive. Preserve existing broadcast/track -identity and epoch semantics. - -Verify the affected path with the TS round-trip harness and its existing -process-status, nonempty-content and decoder assertions. A successful rerun -alone does not finish the quest. Wire every regression and any new controls -into CI, with long live coverage at least nightly; retain diagnostic logs -and failing fixtures. Keep the publisher and exporter statuses in the verdict. - -Keep the older short-duration capture regression and midstream `--stitch` -catalog-linger work separate unless reproduction establishes the same cause. -Update comments and affected feature docs with the fix. No new guide, public -API or wire change is expected; a newly required contract needs a maintainer -decision rather than an invented retry or compatibility layer. - -### Evidence and remaining decision - -Controlled reproduction in `rs/moq-net/tests/catalog_acquisition.rs`: hold the -publisher's initial catalog track request, introduce a cheaper second session to -the same publisher, observe the route change, then answer the held request. -Lite-06 returns `Unroutable`; Lite-07 with the publisher's explicit epoch acquires -and reads the catalog. The old session and publisher both remain alive. No -wall-clock timing, retry or transport teardown is needed. - -This follows the current identity contract: an epochless route replacement is a -new instance, and a pending `TrackInfoFetch` checks its captured -`RequestValidity` before accepting a response. Default negotiation excludes the -work-in-progress Lite-07 wire. #5284's closed-fallback case is different: this -reproduction has no closed session. The reproduction establishes a sufficient -ordering for the reported error, not proof that all retained historical captures -followed exactly that ordering. - -A fresh pair run also failed with the explicit replacement-broadcast error during -upgrade; a subsequent traced direct-WebTransport run passed table alignment. -The pair grader fix is #5331 and does not change routing semantics. - -The maintainer must settle the newly exposed contract before a production fix: -use epoch-capable negotiation for seamless acquisition/continuation, or permit a -fresh initial acquisition on a replacement before output begins. Existing handles -must never silently cross publisher identities. Keep #5328 draft; do not paper -over the default behavior with retries, `--stitch`, or disabled fallback. - -## Related - -- [Validation PR #5328](https://github.com/moq-dev/moq/pull/5328) - records the process failures and successful PR CI baseline -- [Closed fallback](/quest/m1/transport-upgrade/closed-fallback.md) - #5284 owns the pending QUIC dial after an early WebSocket close -- [Fallback PR #5284](https://github.com/moq-dev/moq/pull/5284) - check its actual effect on this reproduction before duplicating work -- [Transport upgrade](/quest/m1/transport-upgrade/README.md) - existing handover and epoch contracts -- [TS duration fidelity](/quest/m0/ts-duration-fidelity.md) - previously captured only a short suffix, not zero-packet initial acquisition -- [Replacement catalog linger](/quest/m1/ts-follow-catalog-bound.md) - bounds a midstream stitch, not initial discovery -- [Pair acquisition](/quest/m1/ts-pair-acquisition.md) - independent grader/mux finding diff --git a/quest/m1/ts-pair-acquisition.md b/quest/m1/ts-pair-acquisition.md index c630b83afdc..05c164b5923 100644 --- a/quest/m1/ts-pair-acquisition.md +++ b/quest/m1/ts-pair-acquisition.md @@ -57,4 +57,3 @@ contracts remain with the broader alignment planning. - [Pair gate review](https://github.com/moq-dev/moq/pull/5328#discussion_r4243103700) - a known-failing baseline would generate daily alerts - [TS roadmap #5327](https://github.com/moq-dev/moq/pull/5327) - owns the broader redundant alignment planning - [Deterministic TS legs](/quest/m2/ts-hitless.md) - packet identity requires a separate contract and proof -- [Initial catalog routing](/quest/m1/initial-catalog-routing.md) - distinct acquisition failure found by the same validation work diff --git a/rs/moq-net/tests/catalog_acquisition.rs b/rs/moq-net/tests/catalog_acquisition.rs index d9b15183c70..5eae22f0ccc 100644 --- a/rs/moq-net/tests/catalog_acquisition.rs +++ b/rs/moq-net/tests/catalog_acquisition.rs @@ -65,11 +65,11 @@ async fn acquisition(version: &str) { } #[moq_net_sim::test] -async fn initial_catalog_acquisition_lite06() { +async fn epochless_replacement_refuses_pending_catalog() { acquisition("moq-lite-06").await; } #[moq_net_sim::test] -async fn initial_catalog_acquisition_lite07() { +async fn same_epoch_replacement_acquires_pending_catalog() { acquisition("moq-lite-07-wip").await; } From 51d56b38609a2388b027ec14076d716d6c1cb9bb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 14:59:13 -0700 Subject: [PATCH 13/17] test(ts): negotiate epochs for seamless acquisition Pin the validation relay and clients to Lite-07 without changing production defaults. Document the strict Lite-06 refusal contract and existing transport teardown coverage. Co-Authored-By: Codex --- test/ts/README.md | 11 ++++++++++- test/ts/run.sh | 21 +++++++++++++-------- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/test/ts/README.md b/test/ts/README.md index 4bda4b07892..15ab1e31f3f 100644 --- a/test/ts/README.md +++ b/test/ts/README.md @@ -596,9 +596,18 @@ claiming the CBR PCR-accuracy profile. ## CI +The round-trip rig explicitly negotiates `moq-lite-07-wip` on the relay, +publisher and subscribers. Its publisher epochs allow acquisition to continue +when WebSocket upgrades to QUIC. Production defaults remain unchanged: Lite-06 +cannot prove publisher identity across session replacement and must refuse a +pending acquisition. `rs/moq-net/tests/catalog_acquisition.rs` covers both +outcomes deterministically in Rust CI, without retries or implicit `--stitch`. +Actual transport teardown coverage belongs to the transport-upgrade tests; a +clean TS run alone does not prove that an upgrade occurred. + [`.github/workflows/ts.yml`](../../.github/workflows/ts.yml) is independent of browser interop. Relevant mux, Hang, CLI, transport, harness and toolchain PRs run -`just test ts-controls`, the default and 2 Mb/s round trips, HRD, headroom, +`just test ts-controls`, `just test ts-pair-controls`, the default and 2 Mb/s round trips, HRD, headroom, open-GOP, EIT, and T-STD controls. Each step runs even after another test fails. All recipes use the same pinned Nix FFmpeg and TSDuck locally and in CI. diff --git a/test/ts/run.sh b/test/ts/run.sh index 6fad266edbf..d519c9de4d8 100755 --- a/test/ts/run.sh +++ b/test/ts/run.sh @@ -54,6 +54,10 @@ source "$DIR/process.sh" # whether they arrived as flags or as environment overrides. RERUN="$(harness_env TSC_DURATION TSC_BITRATE TSC_PORT TSC_PROFILE)just test ts$(harness_argv "$@")" +# Seamless session replacement needs an explicit publisher epoch. Keep this +# opt-in local to the validation rig; epochless replacement is tested as refusal. +VERSION=moq-lite-07-wip + SOURCE="" # real capture to publish instead of a generated clip ANALYZE_ONLY="" # existing TS to analyze without a round-trip CAPTURE_OUT="" # keep the subscriber capture here (for callers asserting on it) @@ -343,9 +347,10 @@ if harness_probe "$URL/certificate.sha256"; then exit 1 fi -echo "### starting relay on 127.0.0.1:${PORT}" +echo "### starting relay on 127.0.0.1:${PORT} ($VERSION)" +printf '%s\n' "$VERSION" >"$HARNESS_RUN/moq-version.txt" harness_spawn relay "$HARNESS_RUN/relay.log" "$RELAY" "$DIR/relay.toml" \ - --listen "127.0.0.1:${PORT}" --web-http-listen "127.0.0.1:${PORT}" + --listen-version "$VERSION" --listen "127.0.0.1:${PORT}" --web-http-listen "127.0.0.1:${PORT}" if ! harness_ready "$URL/certificate.sha256" 30 "$HARNESS_PID"; then echo "error: relay never became ready" >&2 sed 's/^/ relay: /' "$HARNESS_RUN/relay.log" >&2 || true @@ -381,7 +386,7 @@ grade_live() { # status we are here to record. set +e timeout -k 3 $((DURATION + 20)) \ - "$MOQ" --connect "$URL" --broadcast "$BROADCAST" export ts ${EXPORT[@]+"${EXPORT[@]}"} 2>"$HARNESS_RUN/sub.log" | + "$MOQ" --connect "$URL" --connect-version "$VERSION" --broadcast "$BROADCAST" export ts ${EXPORT[@]+"${EXPORT[@]}"} 2>"$HARNESS_RUN/sub.log" | ts_drain python3 "$DIR/pcr-timing.py" --live --seconds "$DURATION" --release-pct-max 1 $STRICT \ ${SCHEDULE[@]+"${SCHEDULE[@]}"} ${PASSTHRU[@]+"${PASSTHRU[@]}"} >"$HARNESS_RUN/timing.out" 2>&1 printf '%s\n' "${PIPESTATUS[0]} ${PIPESTATUS[1]}" >"$HARNESS_RUN/timing.rc" @@ -390,7 +395,7 @@ grade_live() { # shellcheck disable=SC2329 # invoked indirectly via 'harness_spawn' capture() { timeout -k 3 $((DURATION + 20)) \ - "$MOQ" --connect "$URL" --broadcast "$BROADCAST" export ts ${EXPORT[@]+"${EXPORT[@]}"} >"$SUB_TS" 2>"$HARNESS_RUN/sub.log" + "$MOQ" --connect "$URL" --connect-version "$VERSION" --broadcast "$BROADCAST" export ts ${EXPORT[@]+"${EXPORT[@]}"} >"$SUB_TS" 2>"$HARNESS_RUN/sub.log" } # shellcheck disable=SC2329 # invoked indirectly via 'harness_spawn' @@ -401,7 +406,7 @@ capture_b() { # the media, because it has no shared history to derive them from. sleep "$PAIR_JOIN" timeout -k 3 $((DURATION + 20)) \ - "$MOQ" --connect "$URL" --broadcast "$BROADCAST" export ts ${EXPORT[@]+"${EXPORT[@]}"} >"$SUB_B_TS" 2>"$HARNESS_RUN/sub-b.log" + "$MOQ" --connect "$URL" --connect-version "$VERSION" --broadcast "$BROADCAST" export ts ${EXPORT[@]+"${EXPORT[@]}"} >"$SUB_B_TS" 2>"$HARNESS_RUN/sub-b.log" } if [[ -n "$LIVE" ]]; then @@ -432,11 +437,11 @@ sleep 1 echo "### publishing PCR-paced TS -> $BROADCAST" # shellcheck disable=SC2329 # invoked indirectly via 'harness_spawn' publish() { - # shellcheck disable=SC2016 # $1..$4 are the child bash -c positionals, not ours. + # shellcheck disable=SC2016 # $1..$5 are the child bash -c positionals, not ours. timeout -k 3 $((DURATION + 20)) bash -o pipefail -c ' tsp -I file "$1" -P regulate --pcr-synchronous --wait-min 5 | - "$2" --connect "$3" --broadcast "$4" import ts - ' _ "$SRC_TS" "$MOQ" "$URL" "$BROADCAST" + "$2" --connect "$3" --connect-version "$5" --broadcast "$4" import ts + ' _ "$SRC_TS" "$MOQ" "$URL" "$BROADCAST" "$VERSION" } harness_spawn pub "$HARNESS_RUN/pub.log" publish PUB_PID="$HARNESS_PID" From 19fb45287639f7bc534ed14dddd873f7d2fdaf66 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 15:03:10 -0700 Subject: [PATCH 14/17] test(ts): require specific pair failures and retain evidence Co-Authored-By: Codex --- .github/workflows/nightly.yml | 11 +++++++++++ test/ts/table-anchor.test.py | 20 ++++++++++++++++++-- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 0392eb7ca8c..801b84d1712 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -220,6 +220,8 @@ jobs: - name: TS table alignment if: ${{ !cancelled() }} run: nix develop --command just test ts --pair + env: + MOQ_TEST_RUNS: ${{ runner.temp }}/moq-test-runs # The TS exporter's PCR release timing. Everything else about the stream # can be graded from a capture, but *when* the bytes were handed over @@ -236,6 +238,15 @@ jobs: if: ${{ !cancelled() }} run: nix develop --command just test ts --live --duration 120 + - name: Upload TS pair failure artifacts + if: failure() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: nightly-ts-pair + path: ${{ runner.temp }}/moq-test-runs + if-no-files-found: ignore + retention-days: 7 + tests: name: Tests (${{ matrix.recipe }}) runs-on: ubuntu-24.04-arm diff --git a/test/ts/table-anchor.test.py b/test/ts/table-anchor.test.py index 36af9a43179..23fe30b1778 100644 --- a/test/ts/table-anchor.test.py +++ b/test/ts/table-anchor.test.py @@ -126,10 +126,26 @@ def test_controls(self): ) self.assertEqual(result.returncode, 0 if expected else 1, result.stdout + result.stderr) parsed = json.loads(report.read_text()) - if not expected and "error" not in parsed: + if label == "insufficient shared media": + self.assertEqual(parsed["error"], "only 13.990s of acquired shared clock; need 20.0s") + else: + self.assertNotIn("error", parsed) sdt = next(r for r in parsed["tables"] if r["pid"] == 0x11) self.assertTrue(sdt["graded"]) - self.assertFalse(sdt["ok"]) + self.assertEqual(sdt["ok"], expected) + self.assertTrue(all(r["ok"] for r in parsed["tables"] if r["pid"] != 0x11)) + if label in ("persistent phase", "drift"): + self.assertEqual((sdt["a"], sdt["b"]), (12, 12)) + self.assertLess(sdt["agreement_pct"], 90) + elif not expected: + counts = { + "table disappears in shared media": (14, 6), + "table absent": (14, 0), + "table absent on both legs": (0, 0), + "sparse emissions": (14, 7), + "sparse on both legs": (7, 7), + } + self.assertEqual((sdt["a"], sdt["b"]), counts[label]) print(f"PASS control: {label}") capture(a, wrap=True) capture(b, start=11500, wrap=True) From 2f9d022ac3ca2b41bc7b14db12788e440e0864e3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 11:00:42 -0700 Subject: [PATCH 15/17] test(ts): keep the PCR accuracy skip for the no-samples control The round trip grades from the first PCR, but the control still needs to skip every packet to prove an unmeasured capture fails. Co-Authored-By: Claude Opus 5.5 --- test/ts/accuracy.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/test/ts/accuracy.py b/test/ts/accuracy.py index 211b083e07e..0c828286db9 100644 --- a/test/ts/accuracy.py +++ b/test/ts/accuracy.py @@ -7,13 +7,16 @@ from pathlib import Path -def check(path, bitrate, log): +def check(path, bitrate, skip, log): command = [ "tsp", "-I", "file", str(path), "-P", + "skip", + str(skip), + "-P", "pcrverify", "--bitrate", str(bitrate), @@ -42,6 +45,7 @@ def check(path, bitrate, log): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("ts", type=Path) parser.add_argument("--bitrate", required=True, type=int) + parser.add_argument("--skip", default=0, type=int) parser.add_argument("--log", required=True, type=Path) args = parser.parse_args() - check(args.ts, args.bitrate, args.log) + check(args.ts, args.bitrate, args.skip, args.log) From 78283ba6bdf624906161bfe1e1e99dac6d2fb621 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 11:50:50 -0700 Subject: [PATCH 16/17] quest: record the #5328 linger waiver and a local upgrade repro Co-Authored-By: Claude Opus 5.5 --- quest/m0/upgrade-unroutable.md | 7 +++++-- quest/m1/ts-same-instance-resume.md | 11 ++++++++--- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/quest/m0/upgrade-unroutable.md b/quest/m0/upgrade-unroutable.md index 89a3f2ec830..17e10e1d59f 100644 --- a/quest/m0/upgrade-unroutable.md +++ b/quest/m0/upgrade-unroutable.md @@ -124,8 +124,11 @@ upgrading epochless sessions, were rejected. `Restart` for `cam`, and that following it (a fresh `routed_broadcast`) serves the new track. Add the same case on an epoch route, where the old handle keeps working and no `Restart` is sent. Then run `just test interop --all`. - The race can't be forced locally (a fast machine lets QUIC win), so the - unit test carries the proof. + A fast machine lets QUIC win, but setting the QUIC head start to zero + forces the race: `MOQ_CONNECT_WEBSOCKET_DELAY=0s just test interop + --publishers js,python --subscribers rust` fails `js -> rust` and `CLI + FETCH` with `unroutable` on every run on `main`. Use it to confirm the + fix end to end; the unit test still carries the proof. If a failing client still leaves its connection to idle out, fix that client's error path to close its session, as a separate commit. The C diff --git a/quest/m1/ts-same-instance-resume.md b/quest/m1/ts-same-instance-resume.md index eb454055c42..5e248b009ea 100644 --- a/quest/m1/ts-same-instance-resume.md +++ b/quest/m1/ts-same-instance-resume.md @@ -49,9 +49,14 @@ as acceptable. Every regression runs in CI, at least nightly for long integratio coverage. Completion requires the causal test and the relevant full checks, not a successful isolated rerun after a loaded failure. -Keep validation PR #5328 and catalog PR #5333 draft until their readiness -findings are resolved. Update affected test documentation and comments -inline. No new public API, wire format, or user guide is expected; a newly +Decided by the maintainer 2026-10-13: this quest no longer gates validation +PR #5328, and #5333 has merged. None of 137 runs reproduced the exit 1. With the +exporter's stderr kept in the assertion, 97 passed here: batches of 16 and 48 concurrent, 24 +concurrent runs of the whole `export` binary, 8 with the WebSocket race forced +(`MOQ_CONNECT_WEBSOCKET_DELAY=0s`), and a full `just check`, on `main` and the +merged validation branch. Another 40 of 40 passed in an earlier measurement on #5328. The +quest stays open on its own until a failure is captured with its stderr. Update +affected test documentation and comments inline. No new public API, wire format, or user guide is expected; a newly required behavior or API change returns to the maintainer for a decision. ## Related From 6bb3dadebc56293b2fc9ea232929249870d5bbef Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 16:14:23 -0700 Subject: [PATCH 17/17] ci(ts): trigger TS validation on its workspace dependencies kio drives the TS follower through kio::wait; moq-sock, moq-pattern, moq-msf and moq-flate are the other workspace crates under moq-net, moq-tokio and moq-mux. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ts.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/ts.yml b/.github/workflows/ts.yml index 442b9a43fb4..85b01c848a6 100644 --- a/.github/workflows/ts.yml +++ b/.github/workflows/ts.yml @@ -19,6 +19,12 @@ on: - "rs/moq-relay/**" - "rs/moq-loc/**" - "rs/moq-json/**" + # Workspace dependencies of moq-net, moq-tokio and moq-mux on the TS path. + - "rs/kio/**" + - "rs/moq-flate/**" + - "rs/moq-msf/**" + - "rs/moq-pattern/**" + - "rs/moq-sock/**" - "Cargo.toml" - "Cargo.lock" - "rust-toolchain.toml"