diff --git a/quest/m1/release-audit/README.md b/quest/m1/release-audit/README.md index 7a5bccd119b..03b44203011 100644 --- a/quest/m1/release-audit/README.md +++ b/quest/m1/release-audit/README.md @@ -88,13 +88,13 @@ work, because obs-moq ships on its own schedule. ## Required -The children comprise reproduced or source-confirmed findings, +The entries comprise reproduced or source-confirmed findings, contract/lifetime investigations, and one security-review pointer. Each child records whether its behavior is new on main or inherited from release. - [Review HLS admission](/quest/m1/release-audit/hls-admission.md) - resolve the security-sensitive resource-boundary review before the cut - [Bound decoded datagram retention](/quest/m1/release-audit/datagram-memory.md) - shared ownership accounts for retained plain and compressed payloads across tracks -- [Bound export reorder storage](/quest/m1/release-audit/export-reorder.md) - flat timestamps cannot bypass MKV/TS backpressure or cause quadratic draining +- [Decode order](/quest/m1/ts-reliability/decode-order.md) - the exporters' decode clock holds at most the declared reorder depth, so flat timestamps cannot bypass MKV/TS backpressure or cause quadratic draining - [Validate CMAF header fields](/quest/m1/release-audit/cmaf-counts.md) - tiny fragments cannot force allocations proportional to an untrusted sample count, and a zero timescale is refused - [Preserve fMP4 sample runs](/quest/m1/release-audit/fmp4-passthrough.md) - passthrough does not substitute gap or interleaved bytes for media samples - [JS honors subscription bounds](/quest/m1/release-audit/js-subscription-bounds.md) - each reader and each datagram stays inside its own subscription's group bounds, not the aggregate's diff --git a/quest/m1/release-audit/export-reorder.md b/quest/m1/release-audit/export-reorder.md deleted file mode 100644 index 5be921c2d62..00000000000 --- a/quest/m1/release-audit/export-reorder.md +++ /dev/null @@ -1,36 +0,0 @@ -# [M] Include decode reordering in exporter backpressure - -## Goal - -MKV and TS export cannot retain unlimited frames or spend quadratic time -draining them when timestamps stop advancing. Preserve valid B-frame timing -while bounding all exporter-owned payloads, including lookahead storage. - -## Plan - -P1, introduced since release by -[moq-dev/moq#5309](https://github.com/moq-dev/moq/pull/5309), with later TS reuse. -At main 6369eca, `rs/moq-mux/src/container/decode.rs:62-93` inserts every frame -in `held` and `slots`, but timestamp-driven `pop` waits for the highest PTS to -advance past lookahead. `MAX_REORDER` bounds media time, not retained bytes. -Compiling this actual source and pushing 100,000 1KiB items at PTS 90,000, -calling `pop(9_000, 0, false)` after each, released zero frames. An explicit -flush recovered all 102,400,000 payload bytes. Flushing also repeatedly scans -remaining slots and removes vector element zero. - -MKV export gates reads on the downstream jitter queue (`mkv/export.rs:351`) -but inserts video into DecodeClock first (`:398-406`); a stuck clock leaves -that queue empty. Catalog jitter or SPS metadata supplies lookahead, and the -normal CLI buffered export defaults to 500ms. MKV ignores group ends. TS has -the same open-group exposure for timestamp-driven clocks (`ts/export.rs:417`, -`:431`, `:798`), although it flushes at later group boundaries and some SPS -paths use a bounded reorder depth. Consumer integration is source-traced; -the reproduction exercised the clock component. - -Tie reorder storage to exporter capacity/ownership, with explicit behavior -for unsupported timestamp progress and without a timer or arbitrary frame -limit. Add exporter-level open-group tests with flat/backward PTS and valid -reordering. Sweep buffered items and payload sizes to measure retained memory -and drain cost. Source-cache accounting cannot solve this downstream queue. -No public API or wire change is expected; report any changed malformed-input -behavior. Current consumers are CLI MKV and TS export. diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index c30fabb339e..c6940230d2b 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -89,7 +89,9 @@ and PAT plans build on, rank ahead of the remaining rows. Rejected: carrying decode time in every frame container, one shared ingest byte budget, and replacing the demux with GStreamer or another parser (it fixes neither cause). Out of scope for now: decoupling catalog publication from mux-rate -measurement. +measurement. Re-planned the same day: decode order covers MKV and FLV too and +absorbs the release audit's export-reorder quest, and the shared parser lands +before the open stream-routes and PAT drafts, which rebase onto it. PMT fragmentation (#5220), initial mux-rate measurement (#5287), and decoder-time admission (#5306) have landed; PSI runs and catalog linger @@ -98,7 +100,7 @@ remain with their current quests. ## Required - [State bound](/quest/m1/ts-reliability/state-bound.md) - SI and revision state retire with what the source stops repeating -- [Decode order](/quest/m1/ts-reliability/decode-order.md) - exporters author DTS from decode order and fixed per-rendition constants +- [Decode order](/quest/m1/ts-reliability/decode-order.md) - TS, MKV and FLV author DTS from decode order and a declared `reorder` depth - [Shared parser](/quest/m1/ts-reliability/shared-parser.md) - `--program all` scans the multiplex once - [PMT export](/quest/m1/ts-reliability/pmt-export.md) - a TS export announces each PMT revision, mux rate included, at the next `version_number` - [Access units](/quest/m1/ts-reliability/pes-access-units.md) - a PES carrying several access units publishes each at its own time diff --git a/quest/m1/ts-reliability/decode-order.md b/quest/m1/ts-reliability/decode-order.md index cc13d29a786..6dc58667284 100644 --- a/quest/m1/ts-reliability/decode-order.md +++ b/quest/m1/ts-reliability/decode-order.md @@ -1,53 +1,116 @@ -# [L] Decode time comes from decode order and fixed constants +# [XL] Decode time comes from decode order and a declared reorder depth ## Goal -An exporter that needs a decode time (TS, FLV) derives it only from the -frames of the current group, in the decode order they arrive, and from -constants fixed for the rendition's life. Two exporters that read the same -group author the same DTS for it, whatever they joined at or lost before. -Media frames stay presentation-time only: no decode time on the wire. +Every exporter that authors a decode time (TS, MKV, FLV) derives it only from +the frames of the current group, in the decode order they arrive, and from the +rendition's declared reorder depth. Two exporters that read the same group +author the same DTS for it, whatever they joined at or lost before, and an +exporter never holds more video frames than that depth. A rendition that +declares no reordering adds no decode delay. Media frames stay +presentation-time only: no decode time on the wire. ## Plan +Why: a DTS that depends on what the exporter read before has been patched +six times ([moq-dev/moq#1843](https://github.com/moq-dev/moq/pull/1843), +[#2967](https://github.com/moq-dev/moq/pull/2967), +[#4500](https://github.com/moq-dev/moq/pull/4500), +[#4574](https://github.com/moq-dev/moq/pull/4574), +[#4645](https://github.com/moq-dev/moq/pull/4645), +[#5949](https://github.com/moq-dev/moq/pull/5949)). Today there are three +authors: `DecodeClock::pop` (`rs/moq-mux/src/container/decode.rs`, TS without +`--sync` and MKV) settles a slot by the highest PTS read and a delay that only +grows; `pop_counted` (TS `--sync`) depends on the held frames alone; FLV +export authors `PTS - jitter` bumped past the last DTS, which passes the PTS +on B-frames. fMP4 export already anchors its decode timeline at each group's +first frame and needs no change. + Decided 2026-10-22 by the maintainer, after the TS root-cause audit: -- Frames carry PTS only, in decode order. A decoder needs order; a TS - output needs *a* DTS that is monotonic, never after its PTS, and fits the - T-STD buffer, not the source's own. Source DTS spacing (field coding, - pulldown) is normalized, not preserved. +- Frames carry PTS only, in decode order. A decoder needs order; an export + needs *a* DTS that is monotonic, never after its PTS, and fits the T-STD + buffer, not the source's own. Source DTS spacing (field coding, pulldown) + is normalized, not preserved. - Rejected: carrying decode time in the frame (CMAF composition offsets, a LOC property, Legacy inference). It adds a wire field to every container to preserve spacing no consumer needs. -- `DecodeClock::pop_counted` (`rs/moq-mux/src/container/decode.rs`), today - the `--sync` path, becomes the only path. Delete the history-dependent - `pop` settling and the delay that only grows; the catalog `jitter` stops - feeding DTS. -- The constants: an optional per-rendition reorder delay in the Hang - catalog, set by an importer that knows it (TS from source `PTS - DTS`, - fMP4 from `ctts`/`trun` offsets, an encoder from its config) and fixed for - the rendition's life. A source that exceeds it publishes a new rendition - under a fresh name, as a PMT revision does. Absent, export falls back to - the SPS: its declared reorder depth, else the level's `MaxDpbFrames` - bound. Propose the field's name with alternatives in the PR. - -Why: [moq-dev/moq#1843](https://github.com/moq-dev/moq/pull/1843), -[#2967](https://github.com/moq-dev/moq/pull/2967), -[#4500](https://github.com/moq-dev/moq/pull/4500), -[#4574](https://github.com/moq-dev/moq/pull/4574), -[#4645](https://github.com/moq-dev/moq/pull/4645) and -[#5949](https://github.com/moq-dev/moq/pull/5949) each patched a DTS that -depended on what an exporter had read before. - -Rust: `rs/hang` catalog, `rs/moq-mux` TS, fMP4 import and TS, FLV export. -JS: `js/hang` catalog. Draft: `drafts/draft-lcurley-moq-hang.md` (the -catalog field), `drafts/draft-lcurley-moq-mpegts.md` (how export authors -DTS). Public API: a new optional catalog field; the wire change is additive. - -Tests: two exporters joining one B-frame source at different groups, and -one losing a group, author identical DTS for every group both read; a source -exceeding its declared delay publishes a new rendition; a source without the -field falls back to the SPS bound. Keep the T-STD controls passing. +- One clock serves TS, MKV and FLV. The counted path becomes the only one, + and it is not called `pop_counted`: the depth is fixed for the + rendition's life, so it moves into the constructor + (`DecodeClock::new(reorder)`) and the release is a plain `pop(flush)`. + A new rendition gets a new clock. Delete the timestamp-settled `pop`, its + growing delay, `reach`, FLV's `author_dts`/`dts_reserve`, and the catalog + `jitter`'s role in DTS. +- The depth is a new optional `reorder` field on the Hang video rendition: + the most frames that can precede a frame in decode order and follow it in + presentation, the unit of H.264 `max_num_reorder_frames` and HEVC + `sps_max_num_reorder_pics`. +- DTS shifts by slots, not by a time: the nth frame of a group decodes at the + group's (n - `reorder`)th presentation slot, which no frame still to come + can precede. The group's first `reorder` frames step one 90 kHz tick apart + up to the keyframe's slot: the clock cannot know the group's spacing until + frames it may not hold have arrived (`references_far_apart_keep_their_slots`), + and one tick needs no spacing, keeps DTS increasing within a group and + across groups without history, holds no frame past `reorder`, and only + relaxes those frames' delivery deadlines (found in review). Rejected: a delay of + `reorder` times a frame period, since the catalog `framerate` may change in + place and two exporters would build different clocks (found in review), and + freezing the rate or requiring an SPS fixed rate, which both add a rule + the slot shift does not need. +- Those first frames can decode before the timeline's zero, so the clock + works in signed ticks and each exporter maps them to its wire: TS wraps + modulo 2^33 as its timestamps already do, MKV writes no DTS and only + schedules by it, and FLV, whose tag time is unsigned, rebases its whole + audio and video timeline at its first DTS. Never saturate or push a DTS + past its PTS. +- Absent means 0: no reordering and no decode delay, so a stream without + B-frames pays no latency. Rejected: absent meaning the level's + `MaxDpbFrames` (hundreds of ms on High profile streams that never reorder), + and inferring 0 from Baseline profile only (still charges High profile). +- Every producer that can reorder sets it before publishing the rendition: + encoders from their config; importers (TS, FLV, fMP4, MKV) from the SPS's + declared depth, else from the source's own decode offsets on its first + keyframe (TS PTS - DTS, FLV composition time, fMP4 `ctts`/`trun`): none + means 0, any means the level's `MaxDpbFrames`. MKV carries no decode + offsets, so it takes `MaxDpbFrames` when the SPS declares nothing. An + importer knows a frame reorders when its PTS is below an earlier frame's in + decode order; one that reorders deeper than declared publishes a new + rendition under a fresh name, as a PMT revision does. Rejected (from + review): sampling the whole first group before publishing, which delays + every start by a group to catch a source the new rendition already covers. + `js/publish` refuses encoder output that reorders, since WebCodecs exposes + no B-frame setting to declare. +- An exporter that reads a frame reordered deeper than its rendition's + `reorder` refuses the export, never authors an invalid DTS. So does a + `reorder` above the codec level's `MaxDpbFrames` (at most 16), which no + conformant stream can reach, and a nonzero `reorder` on a codec whose + presentation order is its decode order. This bounds the + clock's held frames by the declared depth whatever the timestamps do, which + replaces the release audit's export-reorder quest: flat or backward PTS can + no longer hold frames without bound or drain quadratically. + +Rust: `rs/hang` catalog; `rs/moq-mux` decode clock, TS/MKV/FLV export, and +the TS, FLV, fMP4 and MKV importers; `rs/moq-video` encoders. JS: `js/hang` +catalog, `js/publish`. Draft: `drafts/draft-lcurley-moq-hang.md` (the field +and its absent meaning), `drafts/draft-lcurley-moq-mpegts.md` (how export +authors DTS). Public API and wire: an optional catalog field whose absence +now means "never reorders", a breaking change for any producer that reorders +without it; add a `MIGRATE.md` entry for `hang`, `@moq/hang` and `moq-mux`, whose +direct users must now set it. + +Tests: two exporters joining one B-frame source at different groups, and one +losing a group, author identical DTS for every group both read, in TS, MKV +and FLV; a stream without `reorder` exports with DTS equal to PTS; a frame +reordered past the declared depth refuses the export, and so does a `reorder` +above the level's `MaxDpbFrames`; a framerate change mid-rendition leaves every +DTS unchanged; a group starting at timestamp zero with a nonzero `reorder` +exports valid TS, MKV and FLV; a keyframe without a decode offset followed by +a reordered frame publishes a new rendition; a source deepening its +reordering publishes a new rendition; flat and backward PTS through MKV and +TS export retain at most `reorder` frames, swept over item count and payload +size; each importer sets `reorder` from the SPS and from source offsets. +Keep the T-STD controls passing. ## Related diff --git a/quest/m1/ts-reliability/shared-parser.md b/quest/m1/ts-reliability/shared-parser.md index 9dddeb81fa2..f3d56f0fd75 100644 --- a/quest/m1/ts-reliability/shared-parser.md +++ b/quest/m1/ts-reliability/shared-parser.md @@ -18,7 +18,10 @@ Decided 2026-10-22 by the maintainer: this lands before [PAT changes](/quest/m1/ts-reliability/pat-lifecycle.md), whose plans work around per-program reparsing (multiplex-wide PID rows moving between owners, a removed program's `handle_pat` error stopping the whole ingest). Revisit -both plans once it lands. +both plans once it lands. Re-confirmed 2026-10-22 after drafts of both opened +first: they rebase onto this and drop the accounting it replaces, such as the +`max()` merges in `Programs::stats` and PID rows moving between owners, +rather than landing ahead of it. Add a benchmark in `rs/moq-mux/benches` that feeds `Programs` a generated MPTS and sweeps program count and input rate independently; land it first