diff --git a/doc/concept/hang.md b/doc/concept/hang.md index 875e0bc6233..bdff727955b 100644 --- a/doc/concept/hang.md +++ b/doc/concept/hang.md @@ -67,6 +67,7 @@ A few things the catalog can express beyond decoder config: - **Track names.** The map key is a rendition's ID and, by default, its track name. A rendition sets `track` only when the two must differ, such as two renditions of same-named tracks in different broadcasts, or a rendition whose media changed and so needs a new name; readers subscribe to `track` when present. An ID can't name both a video and an audio rendition. - **Jitter.** A rendition can say how much media it writes behind a frame's timestamp, in whole milliseconds rounded up: the reorder delay of B-frames and the span of a fragment or batch written together. It is measured from media timestamps alone, never the time a frame was written, so neither a constant encoder delay nor a capture clock counts. It describes the publisher, never the network, only grows over the life of a stream, and a player sizes its buffer to at least this much. A `0` is read as absent. - **Delay.** A rendition can also say how far its frames' timestamps trail those another rendition already wrote, so a video encoder running 200 ms behind audio advertises `delay: 200` on video. A rendition alone measures none. It follows the same rules as jitter. A player holds the largest `delay + jitter` among the renditions it subscribes to, and never subtracts one rendition's `delay` from another's. +- **Reorder depth.** A video rendition may declare `reorder`, the most frames that can precede one in decode order and follow it in presentation, as H.264 `max_num_reorder_frames` counts them; `0` declares none, which is not the same as absent. The Rust H.264 and H.265 importers copy it from the SPS when the SPS declares one (progressive H.264 only), and `@moq/publish` declares `0`. The MPEG-TS, Matroska, and FLV exporters then derive each group's decode times from that group's frames alone, so exports that joined at different groups agree, and they end the export on a frame reordered deeper than declared or a depth the codec level cannot reach. Without it they infer decode times from the timestamps they have read, as before. A publisher never changes it once declared. - **Colour.** A video rendition may carry `color`: the H.273 primaries, transfer, matrix, and range, plus the HDR10 mastering display and content light levels, so a renderer can set up HDR without parsing the bitstream. The Rust importers fill it from the SPS VUI, AV1 sequence header, and VP9 header, then fill gaps from a container's `colr` box or Matroska `Colour` element; the stream wins a disagreement. The HDR10 values come from H.264 and H.265 SEI or AV1 metadata OBUs in MPEG-TS and raw elementary streams, an AV1 track's av1C, or Matroska, but not yet from H.264 or H.265 in fMP4 or FLV. Nothing in `@moq/watch` reads it yet. - **Spatial media.** A video rendition may carry `projection` (flat `rectilinear`, or `equirectangular` with the degrees of longitude it spans, 180 for VR180) and `stereo` (`mono`, `left-right`, or `top-bottom` frame packing), and an audio rendition may carry `layout: "ambisonic"` for AmbiX channels. Readers treat a value they don't recognize as absent. Nothing in `@moq/watch` renders them yet. - **Disabled renditions.** An audio or video rendition with `enabled: false` has no frames coming, and a player must not select it. Pausing a rendition in `@moq/publish` keeps it in the catalog this way rather than removing it, so a mute is a one-field update; a disabled video rendition encodes one black keyframe first, so a player that predates the field shows black instead of a frozen picture. The field is written only when false. The older video `stalled` field is no longer written, and readers ignore it. diff --git a/drafts/draft-lcurley-moq-hang.md b/drafts/draft-lcurley-moq-hang.md index a482cf8e9a8..d86edea5db1 100644 --- a/drafts/draft-lcurley-moq-hang.md +++ b/drafts/draft-lcurley-moq-hang.md @@ -194,7 +194,7 @@ This draft needs `version` 4 for: - a `track` ({{field-track}}) other than the entry's ID, which an older consumer subscribes to instead; - a video `warmup` ({{field-warmup}}) above zero, since an older consumer starts decoding a group at a picture that is not a keyframe. -Every other catalog change in this draft is safe to ignore: `enabled` ({{field-enabled}}) leaves an older consumer selecting a disabled rendition, which plays nothing; `archive` ({{archive-catalog}}) is ignored when undecodable; `stats` and `echo` ({{media-stats}}) only offer optional tracks; `jitter`, `delay`, audio `warmup`, and the data track estimates are hints. +Every other catalog change in this draft is safe to ignore: `enabled` ({{field-enabled}}) leaves an older consumer selecting a disabled rendition, which plays nothing; `archive` ({{archive-catalog}}) is ignored when undecodable; `stats` and `echo` ({{media-stats}}) only offer optional tracks; video `reorder` leaves an older consumer inferring decode times from timestamps; `jitter`, `delay`, audio `warmup`, and the data track estimates are hints. ## Clock {#clock} The catalog's root `clock` field is the broadcast's one continuous clock: @@ -258,6 +258,7 @@ type VideoDecoderConfigExtensions = { "displayAspectWidth": number | undefined, "displayAspectHeight": number | undefined, "preference": number | undefined, + "reorder": number | undefined, "color": Color | undefined, "projection": Projection | undefined, "stereo": string | undefined, @@ -276,6 +277,16 @@ A consumer selecting a rendition automatically MUST choose among the enabled ({{ A consumer MAY select another rendition at the viewer's explicit request. A consumer that carries every rendition, rather than selecting one, ignores the field. +`reorder` is the most frames that can precede a frame in decode order and follow it in presentation order, the unit of H.264 `max_num_reorder_frames` and HEVC `sps_max_num_reorder_pics`, as an integer from 0 to 255. +`0` declares that presentation order is decode order. +A publisher that does not know the depth MUST omit the field; absent declares nothing. +A publisher that declares it MUST NOT send a frame that more than `reorder` earlier frames of its group follow in presentation order, and MUST NOT change or remove it for the track once declared. + +A consumer that authors decode times, such as a transmuxer to a container that carries them, derives them from a declared `reorder` and the frames of the current group alone, so consumers that joined at different groups agree. +The n-th frame of a group in decode order decodes at the group's (n - `reorder`)-th earliest presentation time, counting from zero, and the group's first `reorder` frames decode at successive instants just before its earliest presentation time. +A consumer MUST refuse a frame reordered deeper than its rendition declares rather than author an invalid decode time, and SHOULD refuse a `reorder` the codec cannot reach: above the level's decoded picture buffer, or nonzero for a codec whose presentation order is its decode order. +Without `reorder`, a consumer infers decode times from the timestamps it has read. + `color` describes the decoded pixels so a renderer can configure its pipeline without parsing the bitstream: ~~~ @@ -1204,6 +1215,7 @@ This document has no IANA actions. - Each archive timeline declares its own fixed nominal record duration, omitted for an untimed track; removed the broadcast-wide duration bound. - Removed the Recording section: the store layout is a moq-archive format, documented with that crate, not part of this draft. - Added the optional video `preference` field: a consumer selects automatically among the highest preference it can decode. +- Added the optional video `reorder` field: the most frames that can precede one in decode order and follow it in presentation, fixed for the track once declared. A consumer that authors decode times derives them from it and the current group alone, refusing deeper reordering; without it, a consumer infers them from timestamps. - Added an optional `epoch` beside every rendition and data track `broadcast` reference, and `replayEpoch` beside the archive's `replay` ({{field-epoch}}). A consumer reads each referenced path from the first broadcast it resolves for a catalog instance; a present `epoch` fails a rendition proven to resolve another instance, and is best effort over a route without an Epoch. - Derived HLS and DASH exports hand disabled references to enabled renditions at the first record numbered past the advertised ones, keeping the record `sequence` as the HLS media sequence and starting a DASH period. - Added optional `stats` and `echo` catalog sections, publisher/viewer snapshots keyed by rendition ID, and the `.echo` feedback broadcast convention. diff --git a/js/hang/src/catalog/video.test.ts b/js/hang/src/catalog/video.test.ts index 8f80a4c0d68..975ba2d4b48 100644 --- a/js/hang/src/catalog/video.test.ts +++ b/js/hang/src/catalog/video.test.ts @@ -53,6 +53,14 @@ test("video config accepts a signed 32-bit preference", () => { expect(parse(2 ** 31).success).toBe(false); }); +test("video config keeps a zero reorder distinct from an absent one", () => { + const parse = (reorder?: number) => + VideoConfigSchema.safeParse({ codec: "vp8", container: { kind: "legacy" }, reorder }); + expect(parse().data?.reorder).toBeUndefined(); + expect(parse(0).data?.reorder).toBe(Catalog.u8(0)); + expect(parse(256).success).toBe(false); +}); + test("ranked puts a known zero bitrate above an unknown one", () => { const renditions = { unknown: rendition({ codedWidth: 1280, codedHeight: 720 }), diff --git a/js/hang/src/catalog/video.ts b/js/hang/src/catalog/video.ts index b0d8f41e413..57abf3523e5 100644 --- a/js/hang/src/catalog/video.ts +++ b/js/hang/src/catalog/video.ts @@ -139,6 +139,12 @@ export const VideoConfigSchema = z.object({ // The maximum frame rate of the video in frames per second. framerate: z.optional(z.number()), + // The most frames that can precede a frame in decode order and follow it in presentation order, + // as H.264 max_num_reorder_frames and HEVC sps_max_num_reorder_pics count them. 0 declares that + // presentation order is decode order. Absent declares nothing: a consumer that authors decode + // times infers them from the timestamps instead. Once declared, it holds for the track's life. + reorder: z.optional(u8Schema), + // The maximum bitrate of the video in bits per second. // TODO: Support up to Number.MAX_SAFE_INTEGER bitrate: z.optional(u53Schema), diff --git a/js/publish/src/video/encoder.test.ts b/js/publish/src/video/encoder.test.ts index a905fc51fee..c5a663372b2 100644 --- a/js/publish/src/video/encoder.test.ts +++ b/js/publish/src/video/encoder.test.ts @@ -1001,6 +1001,19 @@ test("cut forces a keyframe on the next frame, coalescing requests without spaci expect(keys).toEqual([0, 200, 300]); }); +test("a repeated capture timestamp is dropped rather than failing the track", async () => { + let closed: unknown; + const keys = await recordKeyframes({}, async ({ encoder, track, send }) => { + await send(0); + await send(0); + encoder.cut(); + await send(100); + closed = track.closed.peek(); + }); + expect(keys).toEqual([0, 100]); + expect(closed).toBeUndefined(); +}); + test("a join over the refresh ratio forces a keyframe on the next frame", async () => { const keys = await recordKeyframes({ refresh: 2 }, async ({ track, send }) => { const watching = track.subscribe(); diff --git a/js/publish/src/video/encoder.ts b/js/publish/src/video/encoder.ts index d2ccc0d00bf..6828af26c58 100644 --- a/js/publish/src/video/encoder.ts +++ b/js/publish/src/video/encoder.ts @@ -375,6 +375,8 @@ export class Encoder { let lastKeyframe: Micro | undefined; let lastEncoded: Micro | undefined; + // The catalog declares no reordering, so each chunk must present after the one before it. + let lastOutput: number | undefined; // The largest group demand asked for that a keyframe already answers. A join over the refresh // ratio asks for one past the largest, which the next frame opens. let answered: number | undefined; @@ -387,6 +389,14 @@ export class Encoder { effect.spawn(async () => { const encoder = new VideoEncoder({ output: (frame: EncodedVideoChunk) => { + if (lastOutput !== undefined && frame.timestamp <= lastOutput) { + this.#abort( + new Error(`encoder reordered a frame at ${frame.timestamp}us behind ${lastOutput}us`), + track, + ); + return; + } + lastOutput = frame.timestamp; const key = frame.type === "key"; if (key) { lastKeyframe = frame.timestamp as Micro; @@ -467,6 +477,9 @@ export class Encoder { // Allow half an interval of slack so jittery capture timestamps don't drop // a frame we meant to keep. const targetFrameRate = config?.frameRate; + // A repeated timestamp would present with the frame before it, which no reordering + // declared allows; a backward one still fails the track at output. + if (frame.timestamp === lastEncoded) continue; if (targetFrameRate && lastEncoded !== undefined) { const minGap = Micro.fromSecond((1 / targetFrameRate) as Second); if (frame.timestamp - lastEncoded < minGap - minGap / 2) continue; @@ -581,6 +594,8 @@ export class Encoder { codedWidth: Catalog.u53(config.width), codedHeight: Catalog.u53(config.height), optimizeForLatency: true, + // Realtime WebCodecs encoders emit no B-frames; the output callback refuses one that does. + reorder: Catalog.u8(0), container: { kind: "legacy" } as const, ...estimate, }; diff --git a/quest/m1/encoder-reorder.md b/quest/m1/encoder-reorder.md index b58b32546b6..1328ffa787f 100644 --- a/quest/m1/encoder-reorder.md +++ b/quest/m1/encoder-reorder.md @@ -8,13 +8,11 @@ guessing. ## Plan -[Decode order](/quest/m1/ts-reliability/decode-order.md) adds -`VideoHint.reorder` and lists the moq-video encoders as a producer that -declares it, but leaves them out. Set it in `rendition_hint` -(`rs/moq-video/src/encode/producer.rs`) and assert it in an encoder test. +[moq-dev/moq#6453](https://github.com/moq-dev/moq/pull/6453) adds +`VideoHint.reorder`, which `rendition_hint` +(`rs/moq-video/src/encode/producer.rs`) copies from the probed rendition, but +the encoders leave it unset, so their renditions stay on the inferred clock. +Declare `0` there, even when the SPS omits its bitstream restriction, and +assert it in an encoder test. Public API: none. Wire: none. - -## Required - -- [Decode order](/quest/m1/ts-reliability/decode-order.md) - adds the `reorder` field diff --git a/quest/m1/release-audit/README.md b/quest/m1/release-audit/README.md index e3397ceec66..63434a8bf8d 100644 --- a/quest/m1/release-audit/README.md +++ b/quest/m1/release-audit/README.md @@ -85,14 +85,6 @@ The 2026-10-22 binding and OBS audit at main rate recovery child. Its two OBS findings went to m2 next to the other OBS work, because obs-moq ships on its own schedule. -## Required - -The entries comprise reproduced or source-confirmed findings -and contract/lifetime investigations. Each -child records whether its behavior is new on main or inherited from release. - -- [Decode order](/quest/m1/ts-reliability/decode-order.md) - the exporters' decode clock holds at most the declared reorder depth, so flat timestamps cannot bypass MKV/TS backpressure or cause quadratic draining - ## Related - [JS subscription ranges](/quest/m1/subscribe-ranges/js.md) - range lists and frame offsets build on the bounds fixed here diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index 230b6ea2189..9a95475434b 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -62,7 +62,7 @@ refused means an error or a counted drop. | Mechanism | Contract | Owner | |:--|:--|:--| -| H.264 (0x1B), H.265 (0x24) | Preserved: video rendition, PID and ES descriptors in `mpegts.tracks`; DTS normalized from decode order. Today one access unit per PES | [access units](/quest/m1/ts-reliability/pes-access-units.md), [decode order](/quest/m1/ts-reliability/decode-order.md) | +| H.264 (0x1B), H.265 (0x24) | Preserved: video rendition, PID and ES descriptors in `mpegts.tracks`; DTS normalized from decode order. Today one access unit per PES | [access units](/quest/m1/ts-reliability/pes-access-units.md) | | AAC ADTS (0x0F), MP2, AC-3, E-AC-3, Opus (0x06 + `Opus`) | Preserved: audio rendition; Opus export labels mono and stereo, and 3 to 8 channels with an OpusHead in the Vorbis layout. Other mapping families (such as 255) refuse | landed | | Private and unknown PES types, LATM (0x11), metadata (0x15) | Preserved: `pesPacket`, PTS and DTS rebased | landed | | PES clock fields (ESCR, pack header, TREF), PES scrambling | Refused, counted in `refused` | landed | @@ -99,7 +99,6 @@ remain with their current quests. ## Required - [State bound](/quest/m1/ts-reliability/state-bound.md) - SI and revision state retire with what the source stops repeating -- [Decode order](/quest/m1/ts-reliability/decode-order.md) - TS, MKV and FLV author DTS from decode order and a declared `reorder` depth - [Access units](/quest/m1/ts-reliability/pes-access-units.md) - a PES carrying several access units publishes each at its own time - [PAT changes](/quest/m1/ts-reliability/pat-lifecycle.md) - a revised PAT never leaves a broadcast meaning another program - [Stream routes](/quest/m1/ts-reliability/stream-routes.md) - every PID gets a declared route, and what is not carried is counted diff --git a/quest/m1/ts-reliability/decode-order.md b/quest/m1/ts-reliability/decode-order.md deleted file mode 100644 index 56c2197f8cf..00000000000 --- a/quest/m1/ts-reliability/decode-order.md +++ /dev/null @@ -1,116 +0,0 @@ -# [XL] Decode time comes from decode order and a declared reorder depth - -## Goal - -Every exporter that authors a decode time (TS, MKV, FLV) derives it only from -the frames of the current group, in the decode order they arrive, and from the -rendition's declared reorder depth. Two exporters that read the same group -author the same DTS for it, whatever they joined at or lost before, and an -exporter never holds more video frames than that depth. A rendition that -declares no reordering adds no decode delay. Media frames stay -presentation-time only: no decode time on the wire. - -## Plan - -Why: a DTS that depends on what the exporter read before has been patched -six times ([moq-dev/moq#1843](https://github.com/moq-dev/moq/pull/1843), -[#2967](https://github.com/moq-dev/moq/pull/2967), -[#4500](https://github.com/moq-dev/moq/pull/4500), -[#4574](https://github.com/moq-dev/moq/pull/4574), -[#4645](https://github.com/moq-dev/moq/pull/4645), -[#5949](https://github.com/moq-dev/moq/pull/5949)). Today there are two -authors: `DecodeClock::pop` (`rs/moq-mux/src/container/decode.rs`, TS and -MKV) settles a slot by the highest PTS read and a delay that only grows; FLV -export authors `PTS - jitter` bumped past the last DTS, which passes the PTS -on B-frames. #5949's `pop_counted`, which depended on the held frames alone, -left with `export ts --sync`. fMP4 export already anchors its decode timeline at each group's -first frame and needs no change. - -Decided 2026-10-22 by the maintainer, after the TS root-cause audit: - -- Frames carry PTS only, in decode order. A decoder needs order; an export - needs *a* DTS that is monotonic, never after its PTS, and fits the T-STD - buffer, not the source's own. Source DTS spacing (field coding, pulldown) - is normalized, not preserved. -- Rejected: carrying decode time in the frame (CMAF composition offsets, a - LOC property, Legacy inference). It adds a wire field to every container - to preserve spacing no consumer needs. -- One clock serves TS, MKV and FLV, counting frames against the depth. The - depth is fixed for the rendition's life, so it goes into the constructor - (`DecodeClock::new(reorder)`) and the release is a plain `pop(flush)`. - A new rendition gets a new clock. Delete the timestamp-settled `pop`, its - growing delay, `reach`, FLV's `author_dts`/`dts_reserve`, and the catalog - `jitter`'s role in DTS. -- The depth is a new optional `reorder` field on the Hang video rendition: - the most frames that can precede a frame in decode order and follow it in - presentation, the unit of H.264 `max_num_reorder_frames` and HEVC - `sps_max_num_reorder_pics`. -- DTS shifts by slots, not by a time: the nth frame of a group decodes at the - group's (n - `reorder`)th presentation slot, which no frame still to come - can precede. The group's first `reorder` frames step one 90 kHz tick apart - up to the keyframe's slot: the clock cannot know the group's spacing until - frames it may not hold have arrived (`references_far_apart_keep_their_slots` in #5949), - and one tick needs no spacing, keeps DTS increasing within a group and - across groups without history, holds no frame past `reorder`, and only - relaxes those frames' delivery deadlines (found in review). Rejected: a delay of - `reorder` times a frame period, since the catalog `framerate` may change in - place and two exporters would build different clocks (found in review), and - freezing the rate or requiring an SPS fixed rate, which both add a rule - the slot shift does not need. -- Those first frames can decode before the timeline's zero, so the clock - works in signed ticks and each exporter maps them to its wire: TS wraps - modulo 2^33 as its timestamps already do, MKV writes no DTS and only - schedules by it, and FLV, whose tag time is unsigned, rebases its whole - audio and video timeline at its first DTS. Never saturate or push a DTS - past its PTS. -- Absent means 0: no reordering and no decode delay, so a stream without - B-frames pays no latency. Rejected: absent meaning the level's - `MaxDpbFrames` (hundreds of ms on High profile streams that never reorder), - and inferring 0 from Baseline profile only (still charges High profile). -- Every producer that can reorder sets it before publishing the rendition: - encoders from their config; importers (TS, FLV, fMP4, MKV) from the SPS's - declared depth, else from the source's own decode offsets on its first - keyframe (TS PTS - DTS, FLV composition time, fMP4 `ctts`/`trun`): none - means 0, any means the level's `MaxDpbFrames`. MKV carries no decode - offsets, so it takes `MaxDpbFrames` when the SPS declares nothing. An - importer knows a frame reorders when its PTS is below an earlier frame's in - decode order; one that reorders deeper than declared publishes a new - rendition under a fresh name, as a PMT revision does. Rejected (from - review): sampling the whole first group before publishing, which delays - every start by a group to catch a source the new rendition already covers. - `js/publish` refuses encoder output that reorders, since WebCodecs exposes - no B-frame setting to declare. -- An exporter that reads a frame reordered deeper than its rendition's - `reorder` refuses the export, never authors an invalid DTS. So does a - `reorder` above the codec level's `MaxDpbFrames` (at most 16), which no - conformant stream can reach, and a nonzero `reorder` on a codec whose - presentation order is its decode order. This bounds the - clock's held frames by the declared depth whatever the timestamps do, which - replaces the release audit's export-reorder quest: flat or backward PTS can - no longer hold frames without bound or drain quadratically. - -Rust: `rs/hang` catalog; `rs/moq-mux` decode clock, TS/MKV/FLV export, and -the TS, FLV, fMP4 and MKV importers; `rs/moq-video` encoders. JS: `js/hang` -catalog, `js/publish`. Draft: `drafts/draft-lcurley-moq-hang.md` (the field -and its absent meaning), `drafts/draft-lcurley-moq-mpegts.md` (how export -authors DTS). Public API and wire: an optional catalog field whose absence -now means "never reorders", a breaking change for any producer that reorders -without it; the Impact bullet says how to migrate in `hang`, `@moq/hang` and `moq-mux`, whose -direct users must now set it. - -Tests: two exporters joining one B-frame source at different groups, and one -losing a group, author identical DTS for every group both read, in TS, MKV -and FLV; a stream without `reorder` exports with DTS equal to PTS; a frame -reordered past the declared depth refuses the export, and so does a `reorder` -above the level's `MaxDpbFrames`; a framerate change mid-rendition leaves every -DTS unchanged; a group starting at timestamp zero with a nonzero `reorder` -exports valid TS, MKV and FLV; a keyframe without a decode offset followed by -a reordered frame publishes a new rendition; a source deepening its -reordering publishes a new rendition; flat and backward PTS through MKV and -TS export retain at most `reorder` frames, swept over item count and payload -size; each importer sets `reorder` from the SPS and from source offsets. -Keep the T-STD controls passing. - -## Related - -- [Access units across PES](/quest/m1/ts-reliability/pes-access-units.md) - TS import's other video timing gap diff --git a/rs/hang/src/catalog/root.rs b/rs/hang/src/catalog/root.rs index 78798d0078d..1fa1d3a92f9 100644 --- a/rs/hang/src/catalog/root.rs +++ b/rs/hang/src/catalog/root.rs @@ -701,6 +701,7 @@ mod test { enabled: true, preference: 0, framerate: None, + reorder: None, optimize_for_latency: None, container: Container::Legacy, warmup: None, diff --git a/rs/hang/src/catalog/video/mod.rs b/rs/hang/src/catalog/video/mod.rs index 79396dfe633..3de3607c62a 100644 --- a/rs/hang/src/catalog/video/mod.rs +++ b/rs/hang/src/catalog/video/mod.rs @@ -296,6 +296,15 @@ pub struct VideoConfig { #[serde(default)] pub framerate: Option, + /// The most frames that can precede a frame in decode order and follow it in presentation + /// order, as H.264 `max_num_reorder_frames` and HEVC `sps_max_num_reorder_pics` count them. + /// + /// `0` declares that presentation order is decode order. Absent declares nothing: a consumer + /// that authors decode times infers them from the timestamps instead. A publisher that + /// declares it keeps it for the track's life, and never reorders deeper. + #[serde(default)] + pub reorder: Option, + /// If true, the decoder will optimize for latency. /// /// Default: true @@ -371,6 +380,7 @@ impl VideoConfig { enabled: true, preference: 0, framerate: None, + reorder: None, optimize_for_latency: None, container: Container::default(), jitter: None, @@ -403,6 +413,19 @@ mod test { } } + #[test] + fn reorder_keeps_zero_distinct_from_absent() { + let mut config = VideoConfig::new(VideoCodec::VP8); + assert!(serde_json::to_value(&config).unwrap().get("reorder").is_none()); + for depth in [0, 2] { + config.reorder = Some(depth); + let encoded = serde_json::to_value(&config).unwrap(); + assert_eq!(encoded["reorder"], depth); + let decoded: VideoConfig = serde_json::from_value(encoded).unwrap(); + assert_eq!(decoded.reorder, Some(depth)); + } + } + #[test] fn warmup_rounds_up() { let mut config = VideoConfig::new(VideoCodec::VP8); diff --git a/rs/moq-mux/benches/export.rs b/rs/moq-mux/benches/export.rs index 6815de0c639..58383df9172 100644 --- a/rs/moq-mux/benches/export.rs +++ b/rs/moq-mux/benches/export.rs @@ -162,5 +162,123 @@ fn bench_export(c: &mut Criterion) { group.finish(); } -criterion_group!(benches, bench_export); +criterion_group!(benches, bench_export, video_reorder); criterion_main!(benches); + +/// A progressive AVC SPS declaring a reorder depth of two (25 fps), as the catalog does. +const REORDER_SPS: &[u8] = &[ + 0x67, 0x64, 0x00, 0x0a, 0xac, 0xd9, 0x44, 0x26, 0xc0, 0x44, 0x00, 0x00, 0x03, 0x00, 0x04, 0x00, 0x00, 0x03, 0x00, + 0xca, 0x3c, 0x48, 0x96, 0x58, +]; + +async fn prepare_video( + count: usize, + size: usize, + pattern: &str, +) -> (moq_mux::Source, moq_mux::catalog::Consumer, Fixture) { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); + let stream = moq_mux::catalog::Consumer::from(catalog.consume().unwrap()); + let track = broadcast + .create_track("video", catalog.track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let mut config = hang::catalog::VideoConfig::new(hang::catalog::H264 { + profile: 100, + constraints: 0, + level: 10, + inline: false, + }); + config.coded_width = Some(64); + config.coded_height = Some(64); + config.reorder = Some(2); + let mut description = vec![1, 100, 0, 10, 0xff, 0xe1]; + description.extend_from_slice(&(REORDER_SPS.len() as u16).to_be_bytes()); + description.extend_from_slice(REORDER_SPS); + description.extend_from_slice(&[1, 0, 4, 0x68, 0xce, 0x3c, 0x80]); + config.description = Some(description.into()); + config.container = hang::catalog::Container::Legacy; + let mut video = catalog.video(track, Container::Legacy(Kind::Video), config).unwrap(); + for index in 0..count { + let pts = match pattern { + "pyramid" => ((index / 9 * 9 + [0, 4, 2, 1, 3, 8, 6, 5, 7][index % 9]) * 3_600) as u64, + "flat" => 90_000, + "backward" => 90_000 - index as u64, + _ => unreachable!(), + }; + let mut payload = vec![0; size]; + payload[..4].copy_from_slice(&((size - 4) as u32).to_be_bytes()); + payload[4] = if index == 0 { 0x65 } else { 0x41 }; + video + .write(Frame { + timestamp: Timestamp::from_scale(pts, 90_000).unwrap(), + duration: None, + payload: payload.into(), + keyframe: index == 0, + }) + .unwrap(); + } + video.finish().unwrap(); + catalog.finish().unwrap(); + let (origin, driver) = moq_net::origin::Producer::new(Default::default()); + let driver = tokio::spawn(moq_net::time::run(driver)); + let dynamic = origin.dynamic_route("", Default::default()).unwrap(); + let requests = tokio::spawn(async move { + while let Ok(request) = dynamic.requested_broadcast().await { + request.accept(consumer.clone()); + } + }); + let resolved = origin.consume().request_broadcast("bench", None).await.unwrap(); + let source = moq_mux::Source::new(origin.consume(), resolved); + ( + source, + stream, + Fixture { + _published: Box::new((broadcast, catalog, video)), + _origin: origin, + driver, + requests, + }, + ) +} + +fn video_reorder(c: &mut Criterion) { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let mut group = c.benchmark_group("video-reorder"); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(50)); + group.measurement_time(Duration::from_millis(200)); + for pattern in ["pyramid", "flat", "backward"] { + for count in [64, 256, 1024] { + for size in [256, 16_384] { + group.bench_function(BenchmarkId::new(pattern, format!("{count}x{size}")), |b| { + b.iter_batched( + || runtime.block_on(prepare_video(count, size, pattern)), + |(source, stream, fixture)| { + let refused = runtime.block_on(async { + let mut export = moq_mux::container::mkv::Export::new(source, stream) + .with_fragment_duration(Duration::ZERO); + loop { + match export.next().await { + Ok(Some(frame)) => { + std::hint::black_box(frame); + } + Ok(None) => break false, + Err(_) => break true, + } + } + }); + assert_eq!(refused, pattern != "pyramid"); + fixture + }, + BatchSize::PerIteration, + ); + }); + } + } + } + group.finish(); +} diff --git a/rs/moq-mux/benches/jitter.rs b/rs/moq-mux/benches/jitter.rs index 16f49a643f8..07c2fc93d63 100644 --- a/rs/moq-mux/benches/jitter.rs +++ b/rs/moq-mux/benches/jitter.rs @@ -26,7 +26,7 @@ async fn steer(tracks: u16, floors: u64) -> u64 { let arrival = jitter::Arrival { arrived: at, read: at, - decode: Timestamp::from_millis(second * 1_000).unwrap(), + decode: Timestamp::from_millis(second * 1_000).unwrap().into(), restart: 0, skip: 0, cut: 0, @@ -88,7 +88,7 @@ fn acquisition(c: &mut Criterion) { jitter::Arrival { arrived: start, read: start, - decode: Timestamp::from_millis(frame * 20).unwrap(), + decode: Timestamp::from_millis(frame * 20).unwrap().into(), restart: 0, skip: 0, cut: 0, diff --git a/rs/moq-mux/src/catalog/tracks.rs b/rs/moq-mux/src/catalog/tracks.rs index eab366a5381..90299bd11a1 100644 --- a/rs/moq-mux/src/catalog/tracks.rs +++ b/rs/moq-mux/src/catalog/tracks.rs @@ -200,6 +200,8 @@ pub struct VideoHint { pub bitrate: Option, /// The frame rate in frames per second. pub framerate: Option, + /// The rendition's [`reorder`](hang::catalog::VideoConfig::reorder) depth. + pub reorder: Option, /// If true, the decoder optimizes for latency. pub optimize_for_latency: Option, /// The maximum jitter before the next frame is emitted. @@ -243,6 +245,7 @@ impl From for VideoHint { color: config.color, bitrate: config.bitrate, framerate: config.framerate, + reorder: config.reorder, optimize_for_latency: config.optimize_for_latency, jitter: config.jitter, delay: config.delay, @@ -269,6 +272,7 @@ impl VideoHint { crate::codec::color::fill(&mut config.color, self.color.as_ref(), "hint"); fill(&mut config.bitrate, self.bitrate); fill(&mut config.framerate, self.framerate); + fill(&mut config.reorder, self.reorder); fill(&mut config.optimize_for_latency, self.optimize_for_latency); fill(&mut config.jitter, self.jitter); fill(&mut config.delay, self.delay); diff --git a/rs/moq-mux/src/codec/h264/import.rs b/rs/moq-mux/src/codec/h264/import.rs index 91dbf8342e0..786a7299d6c 100644 --- a/rs/moq-mux/src/codec/h264/import.rs +++ b/rs/moq-mux/src/codec/h264/import.rs @@ -251,7 +251,7 @@ pub fn config(buf: &[u8]) -> Result { /// The config an `AVCDecoderConfigurationRecord` describes, stored verbatim as the `description` /// (`inline: false`, the avc1 shape). -fn config_from_avcc(avcc_bytes: &[u8]) -> Result { +pub(crate) fn config_from_avcc(avcc_bytes: &[u8]) -> Result { let avcc = super::Avcc::parse(avcc_bytes)?; let mut config = hang::catalog::VideoConfig::new(hang::catalog::H264 { @@ -392,6 +392,43 @@ mod tests { assert_eq!((color.mastering, color.content_light), (None, None)); } + /// A frame-only SPS declares its reorder depth, which holds for the track's life: a later SPS + /// declaring less or none keeps it, and one declaring more is refused. + #[tokio::test(start_paused = true)] + async fn a_declared_reorder_depth_holds_for_the_track() { + use crate::codec::h264::fixtures::{PPS, SPS_FIELD, SPS_IPB, SPS_PYRAMID}; + + let idr: &[u8] = &[0x65, 0x88, 0x84, 0x21]; + let (deep, catalog) = setup("deep"); + let (shallow, _) = setup("shallow"); + let mut deep = Import::new(deep, catalog.reserve(), Default::default()).unwrap(); + let mut shallow = Import::new(shallow, catalog.reserve(), Default::default()).unwrap(); + let mut split = Split::new(); + let mut keyframe = |import: &mut Import, name: &str, ms: u64, sps: &[u8]| { + let mut annexb = BytesMut::new(); + for nal in [sps, PPS, idr] { + annexb.extend_from_slice(&[0, 0, 0, 1]); + annexb.extend_from_slice(nal); + } + let pts = moq_net::Timestamp::from_millis(ms).unwrap(); + let mut frames = split.decode(&annexb, pts).expect("split keyframe"); + frames.extend(split.flush(pts).expect("flush keyframe")); + import + .decode(frames) + .map(|()| catalog.snapshot().video.renditions[name].reorder) + }; + + assert_eq!(keyframe(&mut deep, "deep", 0, SPS_PYRAMID).unwrap(), Some(2)); + assert_eq!(keyframe(&mut deep, "deep", 40, SPS_IPB).unwrap(), Some(2)); + assert_eq!( + keyframe(&mut deep, "deep", 80, SPS_FIELD).unwrap(), + Some(2), + "fields declare none" + ); + assert_eq!(keyframe(&mut shallow, "shallow", 0, SPS_IPB).unwrap(), Some(1)); + assert!(keyframe(&mut shallow, "shallow", 40, SPS_PYRAMID).is_err()); + } + /// A new out-of-band avcC starts a new coded sequence, so in-band HDR metadata from the last one /// is dropped. #[tokio::test(start_paused = true)] diff --git a/rs/moq-mux/src/codec/h264/mod.rs b/rs/moq-mux/src/codec/h264/mod.rs index 69e7458fb59..0221e89df71 100644 --- a/rs/moq-mux/src/codec/h264/mod.rs +++ b/rs/moq-mux/src/codec/h264/mod.rs @@ -238,6 +238,11 @@ impl Vui { config.display_aspect_width = display_aspect.map(|(width, _)| width); config.display_aspect_height = display_aspect.map(|(_, height)| height); config.color = self.color.clone(); + // A field picture counts as half a frame, so only a frame-only SPS declares in frames. + config.reorder = self + .depth + .filter(|depth| self.frame_only && *depth <= crate::codec::video::max_reorder(config)) + .and_then(|depth| u8::try_from(depth).ok()); } } @@ -859,7 +864,7 @@ mod tests { /// Only a VUI with a video signal type adds colour, and its colour description is read whole. #[test] fn sps_color_reads_the_video_signal_type() { - let sps_color = |nal| sps_vui(nal).and_then(|vui| vui.color); + let sps_color = |nal: &[u8]| sps_vui(nal)?.color; let sdr = sps_color(fixtures::SPS_BT709).unwrap(); assert_eq!( (sdr.primaries, sdr.transfer, sdr.matrix, sdr.range), diff --git a/rs/moq-mux/src/codec/h265/import.rs b/rs/moq-mux/src/codec/h265/import.rs index 655a518b95f..5d1c6100c29 100644 --- a/rs/moq-mux/src/codec/h265/import.rs +++ b/rs/moq-mux/src/codec/h265/import.rs @@ -372,6 +372,14 @@ mod tests { assert_eq!(config(&ceiling).unwrap().framerate, None); } + /// The SPS's reorder depth at its highest sub-layer becomes the catalog's declaration. + #[test] + fn config_declares_the_sps_reorder_depth() { + let mut pyramid = vec![0, 0, 0, 1]; + pyramid.extend_from_slice(fixtures::SPS_PYRAMID); + assert_eq!(config(&pyramid).unwrap().reorder, Some(2)); + } + /// Square samples carry no display aspect, so the picture is shown at its own size. #[test] fn square_samples_carry_no_display_aspect() { diff --git a/rs/moq-mux/src/codec/h265/mod.rs b/rs/moq-mux/src/codec/h265/mod.rs index 7e6218707a1..4b86f65f2d4 100644 --- a/rs/moq-mux/src/codec/h265/mod.rs +++ b/rs/moq-mux/src/codec/h265/mod.rs @@ -237,7 +237,7 @@ pub(crate) fn config_from_hvcc(hvcc: &[u8]) -> Result Result<()> { @@ -255,6 +255,7 @@ pub(crate) fn apply_hvcc_sps(hvcc: &mp4_atom::Hvcc, config: &mut hang::catalog:: config.display_aspect_width = metadata.display_aspect_width; config.display_aspect_height = metadata.display_aspect_height; config.color = metadata.color; + config.reorder = metadata.reorder; Ok(()) } @@ -292,6 +293,14 @@ pub(crate) fn config_from_sps(sps_nal: &[u8], in_band: bool) -> Result) -> (u64, u64) { let period = match self.period { @@ -99,6 +103,72 @@ impl Reorder { } } +/// The deepest catalog `reorder` a rendition's codec and level can reach: what fits the level's +/// decoded picture buffer at the coded size (H.264 A.3.1 MaxDpbFrames, H.265 A.4.2 MaxDpbSize +/// less the current picture), or the codec's largest buffer when the level or size is unknown. +/// VP8, VP9 and AV1 present in decode order. +pub(crate) fn max_reorder(config: &hang::catalog::VideoConfig) -> u32 { + use hang::catalog::VideoCodec; + + match &config.codec { + VideoCodec::H264(h264) => { + // level_idc 11 with constraint_set3_flag is level 1b for Baseline, Main and Extended. + let level_1b = h264.level == 11 && h264.constraints & 0x10 != 0 && matches!(h264.profile, 66 | 77 | 88); + let max_dpb_mbs: Option = match h264.level { + _ if level_1b => Some(396), + 9 | 10 => Some(396), + 11 => Some(900), + 12 | 13 | 20 => Some(2_376), + 21 => Some(4_752), + 22 | 30 => Some(8_100), + 31 => Some(18_000), + 32 => Some(20_480), + 40 | 41 => Some(32_768), + 42 => Some(34_816), + 50 => Some(110_400), + 51 | 52 => Some(184_320), + 60..=62 => Some(696_320), + _ => None, + }; + let frame_mbs = config + .coded_width + .zip(config.coded_height) + .map(|(width, height)| u64::from(width.div_ceil(16)) * u64::from(height.div_ceil(16))); + match max_dpb_mbs.zip(frame_mbs.filter(|mbs| *mbs > 0)) { + Some((dpb, frame)) => (dpb / frame).min(u64::from(Reorder::MAX_DEPTH)) as u32, + None => Reorder::MAX_DEPTH, + } + } + VideoCodec::H265(h265) => { + let max_luma_ps: Option = match h265.level_idc { + 30 => Some(36_864), + 60 => Some(122_880), + 63 => Some(245_760), + 90 => Some(552_960), + 93 => Some(983_040), + 120 | 123 => Some(2_228_224), + 150 | 153 | 156 => Some(8_912_896), + 180 | 183 | 186 => Some(35_651_584), + _ => None, + }; + let samples = config + .coded_width + .zip(config.coded_height) + .map(|(width, height)| u64::from(width) * u64::from(height)); + let max_dpb_size = match max_luma_ps.zip(samples) { + Some((luma, size)) if size <= luma >> 2 => 16, + Some((luma, size)) if size <= luma >> 1 => 12, + Some((luma, size)) if size <= (3 * luma) >> 2 => 8, + Some(_) => 6, + None => 16, + }; + max_dpb_size - 1 + } + VideoCodec::VP8 | VideoCodec::VP9(_) | VideoCodec::AV1(_) => 0, + _ => Reorder::MAX_DEPTH, + } +} + /// The hypothetical reference decoder a video sequence header declares: its NAL HRD's last /// schedule, the one the transport stream's decoder buffers are sized from (ISO 13818-1 /// 2.14.3.1, 2.17.2). @@ -110,6 +180,14 @@ pub(crate) struct Hrd { pub cpb_size: u64, } +/// A sequence header that reorders deeper than the track already declared. +#[derive(Debug, thiserror::Error)] +#[error("video reorder depth {reorder} exceeds the {declared} the track declared")] +struct Deepened { + declared: u8, + reorder: u8, +} + /// The catalog-publishing state a video importer overlays onto every config it resolves. /// /// Holds the caller's hint and the last config published (to dedupe re-publishes). Not generic @@ -152,10 +230,22 @@ impl Catalog { /// Overlay the hint onto `config` and publish it to `rendition`, unless it matches the last /// publish. A changed config just re-mirrors the rendition; there are no fixed tracks to /// reject a reconfiguration. + /// + /// A declared `reorder` is fixed for the track's life, so a later sequence header declaring + /// less or none keeps it, and one declaring more fails: frames reordered that deep would + /// break the bound consumers already clock the track by. pub(crate) fn publish(&mut self, track: &mut Track, mut config: hang::catalog::VideoConfig) -> crate::Result<()> { self.resolved = Some(config.clone()); self.hdr.apply(&mut config.color); self.hint.apply(&mut config); + if let Some(declared) = self.last.as_ref().and_then(|last| last.reorder) { + match config.reorder { + Some(reorder) if reorder > declared => { + return Err(anyhow::Error::new(Deepened { declared, reorder }).into()); + } + _ => config.reorder = Some(declared), + } + } if self.last.as_ref() == Some(&config) { return Ok(()); } diff --git a/rs/moq-mux/src/container/decode.rs b/rs/moq-mux/src/container/decode.rs index 66335f1a465..76c8e785584 100644 --- a/rs/moq-mux/src/container/decode.rs +++ b/rs/moq-mux/src/container/decode.rs @@ -3,6 +3,8 @@ use std::time::Duration; use moq_clock::Instant; +use hang::catalog::VideoConfig; + /// The largest supported reorder span in 90 kHz ticks. pub(crate) const MAX_REORDER: u64 = 2 * 90_000; @@ -42,6 +44,10 @@ pub(crate) enum Settle { /// once ([`Self::settle`]): a live source's PTS keeps pace with its arrival, so the frames /// held stay within what the source sends in that time, like the jitter buffer's. /// +/// That settling, and a delay that only grows, depend on everything read before, so two +/// exporters that joined or lost frames at different points can clock the same frame apart. +/// A rendition that declares its catalog `reorder` gets a [`DeclaredClock`] instead. +/// /// Ticks are continuous (unwrapped) 90 kHz, so the clock never wraps mid-stream; the 33-bit /// wire wrap happens once at emission. pub(crate) struct DecodeClock { @@ -154,3 +160,254 @@ impl DecodeClock { lookahead.max(self.reach).min(MAX_REORDER) } } + +/// The reorder depth `config` declares, or `None` when it declares none. +/// +/// Refuses a depth its codec and level cannot reach, so a declared clock never holds more frames +/// than a conforming decoder's picture buffer. +pub(crate) fn declared(config: &VideoConfig) -> crate::Result> { + let Some(reorder) = config.reorder else { + return Ok(None); + }; + let max = crate::codec::video::max_reorder(config); + if u32::from(reorder) > max { + return Err(ClockError::Depth { + reorder, + max, + codec: config.codec.to_string(), + } + .into()); + } + Ok(Some(usize::from(reorder))) +} + +/// One group's decode clock, at its rendition's declared reorder depth. +/// +/// The n-th frame of a group in decode order decodes at the group's (n - depth)-th presentation +/// slot. Once the group's first depth + 1 frames are read, no frame still to come can be presented +/// below the earliest of them, so from then on each frame is released as it is read: it takes the +/// least slot not yet taken, which a later frame could undercut only by following more than depth +/// frames in presentation order. The first depth frames decode one tick apart up to that earliest +/// slot, which needs no frame spacing, so a DTS can fall before the timeline's zero. A tick is the +/// caller's timescale, the finest its container writes: 90 kHz for MPEG-TS, 1 ms for FLV. +/// +/// A clock lives for one group, so its DTS depends on nothing read before the group, and a frame +/// reordered deeper than declared fails rather than move the timeline. +pub(crate) struct DeclaredClock { + depth: usize, + /// The group's earliest PTS, once enough frames are read that none to come can precede it. + earliest: Option, + /// Frames still to be released that decode before [`Self::earliest`]. + leading: usize, + /// Frames read but not yet released, in decode order, with their PTS. + held: VecDeque<(T, u64)>, + /// Presentation slots not yet taken, ascending. + slots: Vec, + /// The largest depth + 1 PTS read, ascending: all a deeper reorder is checked against. + highest: Vec, + /// The last DTS released. + last: Option, +} + +impl DeclaredClock { + /// A clock for one group of a rendition reordering at most `depth` frames, from [`declared`]. + pub(crate) fn new(depth: usize) -> Self { + Self { + depth, + earliest: None, + leading: depth, + held: VecDeque::new(), + slots: Vec::new(), + highest: Vec::new(), + last: None, + } + } + + /// How many frames are held. + #[cfg(test)] + pub(crate) fn len(&self) -> usize { + self.held.len() + } + + /// Hold the group's next frame in decode order, presented at `pts`. + pub(crate) fn push(&mut self, item: T, pts: u64) -> crate::Result<()> { + if self.held.len() > self.depth { + return Err(ClockError::Undrained.into()); + } + let at = self.highest.partition_point(|&prior| prior <= pts); + if self.highest.len() - at > self.depth { + return Err(ClockError::Reordered(self.depth).into()); + } + self.highest.insert(at, pts); + if self.highest.len() > self.depth + 1 { + self.highest.remove(0); + } + let at = self.slots.partition_point(|&slot| slot <= pts); + self.slots.insert(at, pts); + self.held.push_back((item, pts)); + Ok(()) + } + + /// The next frame and its DTS in signed ticks, once the group's earliest slot is known, + /// or at once when `flush` because the group ended. + pub(crate) fn pop(&mut self, flush: bool) -> crate::Result> { + if self.held.is_empty() || (self.earliest.is_none() && !flush && self.held.len() <= self.depth) { + return Ok(None); + } + let earliest = *self.earliest.get_or_insert(self.slots[0]); + let dts = match self.leading { + 0 => i128::from(self.slots.remove(0)), + leading => { + self.leading -= 1; + i128::from(earliest) - leading as i128 + } + }; + let (item, pts) = self.held.pop_front().expect("the held queue is nonempty"); + if dts > i128::from(pts) || self.last.is_some_and(|last| dts <= last) { + return Err(ClockError::Presentation.into()); + } + self.last = Some(dts); + Ok(Some((item, dts))) + } +} + +#[derive(Debug, thiserror::Error)] +enum ClockError { + #[error("video reorder depth {reorder} exceeds the {max} that {codec} can reach")] + Depth { reorder: u8, max: u32, codec: String }, + #[error("video reorders past its declared depth {0}")] + Reordered(usize), + #[error("video presentation slots cannot author increasing DTS at or before PTS")] + Presentation, + #[error("the decode clock must be drained before reading another frame")] + Undrained, +} + +impl From for crate::Error { + fn from(err: ClockError) -> Self { + anyhow::Error::new(err).into() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A B-pyramid (`max_num_reorder_frames` 2) in decode order, in 40 ms frames. + const PYRAMID: [u64; 9] = [0, 4, 2, 1, 3, 8, 6, 5, 7]; + const FRAME: u64 = 3_600; + + /// The DTS of one group read through a declared clock of `depth`, released as each frame is + /// read, checking it never holds more than `depth` frames between reads. + fn declared_dts(depth: usize, read: &[u64]) -> Vec { + let mut clock = DeclaredClock::new(depth); + let mut out = Vec::new(); + for &pts in read { + clock.push((), pts).unwrap(); + while let Some(((), dts)) = clock.pop(false).unwrap() { + out.push(dts); + } + assert!(clock.len() <= depth); + } + while let Some(((), dts)) = clock.pop(true).unwrap() { + out.push(dts); + } + out + } + + #[test] + fn declared_slots_shift_by_count_and_start_before_zero() { + let ticks = |frames: &[i128]| frames.iter().map(|i| i * FRAME as i128).collect::>(); + let leading = |dts: Vec| [vec![-2, -1], dts].concat(); + assert_eq!( + declared_dts(2, &PYRAMID.map(|i| i * FRAME)), + leading(ticks(&[0, 1, 2, 3, 4, 5, 6])) + ); + // References spaced wider than any frame period still take their slots by count. + let far = [0, 6, 10, 1, 2, 3, 4, 5, 7, 8, 9].map(|i| i * FRAME); + assert_eq!(declared_dts(2, &far), leading(ticks(&[0, 1, 2, 3, 4, 5, 6, 7, 8]))); + } + + /// Open-GOP leading pictures present before the keyframe that starts their group, so the + /// group's first frames decode below its earliest slot, not the keyframe's. + #[test] + fn leading_pictures_decode_before_the_keyframe() { + let read = [10, 8, 9, 14, 12, 11, 13]; + assert_eq!(declared_dts(2, &read), [6, 7, 8, 9, 10, 11, 12]); + // A group that ends before the lookahead fills still takes its earliest slot. + assert_eq!(declared_dts(2, &read[..2]), [6, 7]); + } + + #[test] + fn a_zero_depth_releases_each_frame_at_its_pts() { + let mut clock = DeclaredClock::new(0); + for pts in [0, 1, 90_000, u64::MAX] { + clock.push(pts, pts).unwrap(); + assert_eq!(clock.pop(false).unwrap(), Some((pts, i128::from(pts)))); + assert_eq!(clock.len(), 0); + } + clock.push(0, 0).unwrap_err(); + } + + /// Flat or backward PTS read past the declared depth fail within depth + 1 frames, however + /// many follow and however large they are, rather than accumulate. + #[test] + fn malformed_presentation_is_refused_without_retaining_its_payload() { + for backwards in [false, true] { + for count in [16, 128, 1024] { + for size in [256, 16_384] { + let mut clock = DeclaredClock::new(2); + let refused = (0..count).position(|index| { + let pts = 90_000 - if backwards { index } else { 0 }; + let refused = clock.push(vec![0u8; size], pts).is_err() + || std::iter::from_fn(|| clock.pop(false).transpose()).any(|popped| popped.is_err()); + assert!(clock.len() <= 2); + refused + }); + assert!( + refused.is_some_and(|index| index <= 3), + "backwards={backwards} count={count}" + ); + } + } + } + } + + #[test] + fn a_declared_depth_fits_the_codec_level() { + use hang::catalog::{H264, VideoCodec}; + + let config = |codec: VideoCodec, size: Option<(u32, u32)>, reorder: u8| { + let mut config = VideoConfig::new(codec); + config.coded_width = size.map(|(width, _)| width); + config.coded_height = size.map(|(_, height)| height); + config.reorder = Some(reorder); + config + }; + let h264 = |level| H264 { + profile: 100, + constraints: 0, + level, + inline: false, + }; + let h265: VideoCodec = "hvc1.1.6.L123.B0".parse().unwrap(); + let vp9: VideoCodec = "vp09.00.10.08".parse().unwrap(); + let cases = [ + // Level 4.0's 32768 macroblocks hold four 1920x1088 frames. + (config(h264(40).into(), Some((1920, 1080)), 4), true), + (config(h264(40).into(), Some((1920, 1080)), 5), false), + // Without a size, the deepest buffer any picture of the level gets. + (config(h264(40).into(), None, 16), true), + (config(h264(40).into(), None, 17), false), + // Level 4.1's MaxLumaPs holds six 1920x1080 pictures, one of them the current. + (config(h265.clone(), Some((1920, 1080)), 5), true), + (config(h265, Some((1920, 1080)), 6), false), + (config(vp9.clone(), None, 0), true), + (config(vp9, None, 1), false), + ]; + for (config, valid) in cases { + assert_eq!(declared(&config).is_ok(), valid, "{config:?}"); + } + assert_eq!(declared(&VideoConfig::new(VideoCodec::VP8)).unwrap(), None); + } +} diff --git a/rs/moq-mux/src/container/flv/export.rs b/rs/moq-mux/src/container/flv/export.rs index 797b2656f74..703cb9e2081 100644 --- a/rs/moq-mux/src/container/flv/export.rs +++ b/rs/moq-mux/src/container/flv/export.rs @@ -34,6 +34,8 @@ use super::{ VIDEO_EX_HEADER, VIDEO_PACKET_CODED_FRAMES, VIDEO_PACKET_MULTITRACK, VIDEO_PACKET_SEQUENCE_START, }; use crate::catalog::Stream; +use crate::container::consumer::Event; +use crate::container::decode::DeclaredClock; use crate::container::{ExportSource, Frame}; /// Before the header, unbind every track the catalog no longer lists (removed, or disabled) or @@ -154,6 +156,9 @@ pub struct Export { /// True once the file header and sequence headers have been emitted. header_emitted: bool, + /// Milliseconds added to every tag's timestamps, so the decode times a declared reorder + /// depth puts before a group's earliest presentation still fit FLV's unsigned clock. + shift: u32, } /// A frame waiting to go out as a tag. @@ -161,6 +166,8 @@ struct Tag { frame: Frame, /// The tag timestamp, the authored DTS for a codec with composition time. timestamp: u32, + /// The presentation time on the tag clock. + presentation: u32, /// The codec config record the frame was read under, which a sequence-header tag must /// have announced before it. record: Option, @@ -184,6 +191,11 @@ struct FlvTrack { /// tag, used when the catalog (and thus [`ExportSource::description`]) carries /// none. Only AV1 needs this today (its av1C is optional in the catalog). fallback_description: Option, + /// The video rendition's catalog entry as last listed. + config: Option, + /// The current group's clock, when its rendition declared `reorder` at the group's start, + /// holding each frame with the config record it was read under. + declared: Option, Option)>>, /// How far to run the authored decode clock behind the PTS, in FLV milliseconds. dts_reserve: u32, /// Last authored video DTS, in FLV milliseconds. @@ -200,8 +212,8 @@ impl FlvTrack { self.empty = moq_clock::now(); } - fn tag_timestamp(&self, frame: &Frame) -> anyhow::Result { - let pts = frame_timestamp_ms(frame)?; + /// The tag timestamp of a frame presented at `pts` on the tag clock, on the inferred clock. + fn tag_timestamp(&self, pts: u32) -> anyhow::Result { if self.flavor.has_composition_time() { author_dts(pts, self.dts_reserve, self.last_dts) } else { @@ -209,6 +221,107 @@ impl FlvTrack { } } + /// Start a group: queue the last one's frames, settled by `arrived`, the keyframe's arrival, + /// then clock this one at the depth its rendition declares now, or on the inferred clock. + fn start_group( + &mut self, + key: (u8, usize), + arrived: moq_clock::Instant, + shift: u32, + jitter: &mut crate::jitter::Buffer<(u8, usize), Tag>, + ) -> anyhow::Result<()> { + self.release(key, Some(arrived), shift, true, jitter)?; + let depth = match &self.config { + Some(config) if self.flavor.has_composition_time() => crate::container::decode::declared(config)?, + _ => None, + }; + self.declared = depth.map(DeclaredClock::new); + Ok(()) + } + + /// Queue each frame the declared clock releases, every one when `flush`. A frame held for a + /// later one is judged as arriving with it, at `settled`. + fn release( + &mut self, + key: (u8, usize), + settled: Option, + shift: u32, + flush: bool, + jitter: &mut crate::jitter::Buffer<(u8, usize), Tag>, + ) -> anyhow::Result<()> { + let Some(mut clock) = self.declared.take() else { + return Ok(()); + }; + let released = (|| { + while let Some(((mut arrival, record), dts)) = clock.pop(flush)? { + if let Some(settled) = settled { + arrival.arrived = arrival.arrived.max(settled); + } + let timestamp = u32::try_from(dts + i128::from(shift)) + .context("FLV decode time before the stream's origin or past 32 bits")?; + self.queue(key, arrival, record, timestamp, shift, jitter)?; + } + anyhow::Ok(()) + })(); + self.declared = Some(clock); + released + } + + /// Queue `arrival` as a tag decoding at `timestamp`, under the config `record` it was read with. + fn queue( + &mut self, + key: (u8, usize), + arrival: crate::jitter::Arrival, + record: Option, + timestamp: u32, + shift: u32, + jitter: &mut crate::jitter::Buffer<(u8, usize), Tag>, + ) -> anyhow::Result<()> { + if self.flavor.has_composition_time() { + self.last_dts = Some(timestamp); + } + let presentation = frame_timestamp_ms(&arrival.item, shift)?; + let crate::jitter::Arrival { + arrived, + read, + restart, + skip, + cut, + sync, + leading, + item, + .. + } = arrival; + let pushed = jitter.push( + key, + crate::jitter::Arrival { + arrived, + read, + decode: moq_net::Timestamp::from_millis(u64::from(timestamp))?.into(), + restart, + skip, + cut, + sync, + leading, + item: Tag { + frame: item, + timestamp, + presentation, + record, + }, + }, + )?; + if pushed == crate::jitter::Push::Late { + tracing::warn!( + kind = key.0, + index = key.1, + dropped = jitter.dropped(), + "frame missed its deadline; dropped" + ); + } + Ok(()) + } + /// The out-of-band config record for this track's sequence-header tag, or /// `None` for codecs that carry their config in band. fn config_record(&self) -> anyhow::Result> { @@ -241,6 +354,7 @@ impl Export { video: Vec::new(), audio: Vec::new(), header_emitted: false, + shift: 0, } } @@ -378,8 +492,14 @@ impl Export { if track.finished { break; } - let frame = match track.source.poll_read(waiter)? { - Poll::Ready(Some(frame)) => frame, + let frame = match track.source.poll_event(waiter)? { + Poll::Ready(Some(Event::Frame { frame, .. })) => frame, + // A group that ended or was cut short holds frames no later one can precede. + Poll::Ready(Some(Event::GroupEnd | Event::Cut)) => { + track.release((kind, index), None, self.shift, true, &mut self.jitter)?; + continue; + } + Poll::Ready(Some(Event::FrameEnd { .. })) => continue, Poll::Ready(None) => { track.finished = true; break; @@ -396,7 +516,7 @@ impl Export { crate::jitter::Arrival { arrived: track.empty, read: moq_clock::now(), - decode: frame.timestamp, + decode: frame.timestamp.into(), restart: track.source.markers(), skip: track.source.skips(), cut: track.source.cuts(), @@ -409,49 +529,34 @@ impl Export { track.pending = Some(arrival); break; } - let timestamp = track.tag_timestamp(&arrival.item)?; - if track.flavor.has_composition_time() { - track.last_dts = Some(timestamp); - } + let key = (kind, index); let record = track.config_record()?; - let crate::jitter::Arrival { - arrived, - read, - restart, - skip, - cut, - sync, - leading, - item, - .. - } = arrival; - if self.jitter.push( - (kind, index), - crate::jitter::Arrival { - arrived, - read, - decode: moq_net::Timestamp::from_millis(u64::from(timestamp))?, - restart, - skip, - cut, - sync, - leading, - item: Tag { - frame: item, - timestamp, - record, - }, - }, - )? == crate::jitter::Push::Late - { - tracing::warn!( - kind, - index, - dropped = self.jitter.dropped(), - "frame missed its deadline; dropped" - ); + if kind == 0 && arrival.item.keyframe { + track.start_group(key, arrival.arrived, self.shift, &mut self.jitter)?; + } + match &mut track.declared { + Some(clock) => { + // Clocked in FLV's milliseconds, so the leading frames step a tag tick apart. + let pts = arrival.item.timestamp.as_millis() as u64; + let arrived = arrival.arrived; + clock.push((arrival, record), pts)?; + track.release(key, Some(arrived), self.shift, false, &mut self.jitter)?; + } + None => { + let timestamp = track.tag_timestamp(frame_timestamp_ms(&arrival.item, self.shift)?)?; + track.queue(key, arrival, record, timestamp, self.shift, &mut self.jitter)?; + } } } + let track = if kind == 0 { + &mut self.video[index] + } else { + &mut self.audio[index] + }; + // Nothing settles the frames held when a track ends, so they keep their own arrival. + if track.finished && self.header_emitted { + track.release((kind, index), None, self.shift, true, &mut self.jitter)?; + } } } Ok(()) @@ -468,6 +573,16 @@ impl Export { // single-track stream rebinds to a better-ranked rendition. A bound rendition the // catalog moves to another track follows it, under a new sequence-header tag when // its config record changes. + for track in &mut self.video { + if let Some(config) = catalog.video.renditions.get(&track.name) { + // A depth declared after the header deepens the shift, so its group's first DTS stays + // at or past zero. The shift never shrinks, so tag times keep increasing. + if self.header_emitted && track.flavor.has_composition_time() { + self.shift = self.shift.max(config.reorder.map_or(0, u32::from)); + } + track.config = Some(config.clone()); + } + } if !self.header_emitted { self.bind_video(&catalog)?; self.bind_audio(&catalog)?; @@ -569,6 +684,8 @@ impl Export { finished: false, flavor, fallback_description, + config: Some(config.clone()), + declared: None, dts_reserve: dts_reserve(config), last_dts: None, announced: None, @@ -616,6 +733,8 @@ impl Export { finished: false, flavor, fallback_description: None, + config: None, + declared: None, dts_reserve: 0, last_dts: None, announced: None, @@ -659,6 +778,15 @@ impl Export { for track in self.video.iter_mut().chain(&mut self.audio) { track.announced = track.config_record()?; } + // A group decodes its first `reorder` frames a millisecond apart before its earliest + // presentation, so the deepest declaration bounds how far before zero a DTS can fall. + self.shift = self + .video + .iter() + .filter(|track| track.flavor.has_composition_time()) + .filter_map(|track| track.config.as_ref()?.reorder) + .max() + .map_or(0, u32::from); Ok(out.freeze()) } @@ -669,6 +797,7 @@ impl Export { let Tag { frame, timestamp: timestamp_ms, + presentation: pts_ms, record, } = tag; let mut out = BytesMut::with_capacity(TAG_HEADER_LEN + frame.payload.len() + 8); @@ -690,7 +819,6 @@ impl Export { let track = &self.video[index]; let flavor = track.flavor; let track_id = track.track_id; - let pts_ms = frame_timestamp_ms(&frame)?; let frame_type = if frame.keyframe { FRAME_TYPE_KEY } else { @@ -873,11 +1001,11 @@ fn audio_flavor(config: &hang::catalog::AudioConfig) -> anyhow::Result { } } -fn frame_timestamp_ms(frame: &Frame) -> anyhow::Result { - frame - .timestamp - .as_millis() - .try_into() +/// The frame's presentation time on the tag clock, `shift` milliseconds past its own. +fn frame_timestamp_ms(frame: &Frame, shift: u32) -> anyhow::Result { + u32::try_from(frame.timestamp.as_millis()) + .ok() + .and_then(|pts| pts.checked_add(shift)) .context("FLV timestamp exceeds 32 bits") } diff --git a/rs/moq-mux/src/container/flv/export_test.rs b/rs/moq-mux/src/container/flv/export_test.rs index 00da99ecda0..4a6e495c0d3 100644 --- a/rs/moq-mux/src/container/flv/export_test.rs +++ b/rs/moq-mux/src/container/flv/export_test.rs @@ -1342,6 +1342,210 @@ async fn export_authors_dts_and_composition_time_for_reordered_avc() { ); } +/// The `(DTS, PTS)` of every AVC frame tag, and the audio tag timestamps, an export renders of +/// a broadcast that declares `reorder` and carries a group of `stamps` (ms, decode order) for +/// each entry, plus one audio frame at 20 ms. With `open`, the last group is cut but the video +/// track stays open. +async fn export_declared(reorder: u8, groups: &[&[u64]], open: bool) -> anyhow::Result<(Vec<(u32, i64)>, Vec)> { + use hang::catalog::{AAC, AudioConfig, Container, H264, VideoConfig}; + use moq_net::Timestamp; + + let mut producer = moq_net::broadcast::Info::new().produce(); + let consumer = producer.consume(); + let mut catalog = crate::catalog::Producer::new(&mut producer, crate::catalog::Config::default()).unwrap(); + let video_track = producer.create_track(producer.unique_name(".avc1"), timed()).unwrap(); + let audio_track = producer.create_track(producer.unique_name(".aac"), timed()).unwrap(); + let mut video_config = VideoConfig::new(H264 { + profile: 0x42, + constraints: 0xc0, + level: 0x1f, + inline: false, + }); + video_config.container = Container::Legacy; + video_config.description = Some(Bytes::from(avcc())); + video_config.reorder = Some(reorder); + let mut audio_config = AudioConfig::new(AAC { profile: 2 }, 44100, 2); + audio_config.container = Container::Legacy; + audio_config.description = Some(Bytes::from_static(&ASC)); + { + let mut catalog = catalog.modify().unwrap(); + catalog + .video + .renditions + .insert(video_track.name().to_string(), video_config); + catalog + .audio + .renditions + .insert(audio_track.name().to_string(), audio_config); + } + let frame = |ms: u64, payload: &'static [u8], keyframe| crate::container::Frame { + timestamp: Timestamp::from_millis(ms).unwrap(), + duration: None, + payload: Bytes::from_static(payload), + keyframe, + }; + let mut video = crate::container::Producer::new( + video_track, + crate::catalog::hang::Container::Legacy(crate::container::Kind::Data), + ); + for stamps in groups { + for (index, &ms) in stamps.iter().enumerate() { + let payload: &'static [u8] = if index == 0 { + &[0, 0, 0, 1, 0x65] + } else { + &[0, 0, 0, 1, 0x41] + }; + video.write(frame(ms, payload, index == 0)).unwrap(); + } + } + match open { + true => video.cut(None).unwrap(), + false => video.finish().unwrap(), + } + let mut audio = crate::container::Producer::new( + audio_track, + crate::catalog::hang::Container::Legacy(crate::container::Kind::Data), + ); + audio.write(frame(20, &[0xde, 0xad], true)).unwrap(); + audio.finish().unwrap(); + catalog.finish().unwrap(); + + let mut exporter = open_export(crate::source::announced(&consumer).await) + .await + .with_delay(RECORDING_MAX_DELAY) + .with_replay(); + let mut exported = Vec::new(); + if let Some(Err(err)) = drain_until_idle(&mut exporter, &mut exported).await { + return Err(err); + } + drop(video); + let tags = parse_tags(&exported); + let video = tags + .iter() + .filter(|tag| tag.tag_type == super::TAG_VIDEO && tag.body[1] == super::AVC_NALU) + .map(|tag| { + let pts = i64::from(tag.timestamp) + i64::from(super::read_i24(&tag.body[2..5])); + (tag.timestamp, pts) + }) + .collect(); + let audio = tags + .iter() + .filter(|tag| tag.tag_type == super::TAG_AUDIO && tag.body[1] == super::AAC_RAW) + .map(|tag| tag.timestamp) + .collect(); + Ok((video, audio)) +} + +/// A rendition that declares its reorder depth decodes each group from that group's frames: its +/// first frames a millisecond apart below its earliest presentation, and the rest at its +/// presentation slots. The whole stream shifts by the depth, so those fit FLV's unsigned clock. +#[tokio::test(start_paused = true)] +async fn export_declared_reorder_decodes_each_group_from_its_frames() { + let groups: [&[u64]; 2] = [&[0, 160, 80, 40, 120], &[400, 560, 480, 440, 520]]; + let (video, audio) = export_declared(2, &groups, false).await.unwrap(); + assert_eq!( + video, + [ + (0, 2), + (1, 162), + (2, 82), + (42, 42), + (82, 122), + (400, 402), + (401, 562), + (402, 482), + (442, 442), + (482, 522) + ] + ); + assert_eq!(audio, [22]); + + // A first group cut short leaves the next decoding where it would have. + let cut: [&[u64]; 2] = [&[0, 160, 80], &[400, 560, 480, 440, 520]]; + let (cut, _) = export_declared(2, &cut, false).await.unwrap(); + assert_eq!(cut[3..], video[5..]); + + // A depth of zero decodes at presentation, and refuses a frame that reorders. + let (video, _) = export_declared(0, &[&[0, 40, 80]], false).await.unwrap(); + assert_eq!(video, [(0, 0), (40, 40), (80, 80)]); + let err = export_declared(0, &[&[0, 80, 40]], false).await.unwrap_err(); + assert!(err.to_string().contains("declared depth"), "{err}"); + + // A group that ends goes out whole while the track stays open, though fewer frames than the + // depth settle nothing. + let (video, _) = export_declared(2, &[&[0, 80]], true).await.unwrap(); + assert_eq!(video, [(0, 2), (1, 82)]); +} + +/// A depth the catalog declares only after the header raises the shift for the groups it clocks, +/// rather than put their first DTS before zero. +#[tokio::test(start_paused = true)] +async fn a_reorder_declared_after_the_header_raises_the_shift() { + use hang::catalog::{Container, H264, VideoConfig}; + use moq_net::Timestamp; + + let mut producer = moq_net::broadcast::Info::new().produce(); + let consumer = producer.consume(); + let mut catalog = crate::catalog::Producer::new(&mut producer, crate::catalog::Config::default()).unwrap(); + let video_track = producer.create_track(producer.unique_name(".avc1"), timed()).unwrap(); + let name = video_track.name().to_string(); + let mut config = VideoConfig::new(H264 { + profile: 0x42, + constraints: 0xc0, + level: 0x1f, + inline: false, + }); + config.container = Container::Legacy; + config.description = Some(Bytes::from(avcc())); + catalog.modify().unwrap().video.renditions.insert(name.clone(), config); + let frame = |ms: u64, keyframe| crate::container::Frame { + timestamp: Timestamp::from_millis(ms).unwrap(), + duration: None, + payload: Bytes::from_static(if keyframe { + &[0, 0, 0, 1, 0x65] + } else { + &[0, 0, 0, 1, 0x41] + }), + keyframe, + }; + let mut video = crate::container::Producer::new( + video_track, + crate::catalog::hang::Container::Legacy(crate::container::Kind::Data), + ); + video.write(frame(0, true)).unwrap(); + + let mut exporter = open_export(crate::source::announced(&consumer).await) + .await + .with_delay(RECORDING_MAX_DELAY) + .with_replay(); + let mut exported = exporter.next().await.unwrap().expect("the header").to_vec(); + + catalog + .modify() + .unwrap() + .video + .renditions + .get_mut(&name) + .unwrap() + .reorder = Some(2); + for (ms, keyframe) in [(1, true), (5, false), (3, false)] { + video.write(frame(ms, keyframe)).unwrap(); + } + video.finish().unwrap(); + catalog.finish().unwrap(); + drain_until_idle(&mut exporter, &mut exported).await.unwrap().unwrap(); + + let video: Vec<_> = parse_tags(&exported) + .iter() + .filter(|tag| tag.tag_type == super::TAG_VIDEO && tag.body[1] == super::AVC_NALU) + .map(|tag| { + let pts = i64::from(tag.timestamp) + i64::from(super::read_i24(&tag.body[2..5])); + (tag.timestamp, pts) + }) + .collect(); + assert_eq!(video, [(0, 2), (1, 3), (2, 7), (3, 5)]); +} + /// Poll `exporter` into `out` until it ends or goes idle; `None` if it went idle. async fn drain_until_idle( exporter: &mut Export, diff --git a/rs/moq-mux/src/container/mkv/export.rs b/rs/moq-mux/src/container/mkv/export.rs index 15e63b9b6ec..14be1a9a40d 100644 --- a/rs/moq-mux/src/container/mkv/export.rs +++ b/rs/moq-mux/src/container/mkv/export.rs @@ -73,7 +73,10 @@ pub struct Export { } struct MkvTrack { + /// The inferred decode clock of a rendition that declares no `reorder`. clock: DecodeClock>, + /// The current group's clock, when its rendition declared `reorder` at the group's start. + declared: Option>>, config: Option, restart: u64, reorder: Option<(u64, u64)>, @@ -98,6 +101,7 @@ impl MkvTrack { fn new(source: ExportSource, kind: TrackKind, config: Option, track_number: u64) -> Self { Self { clock: Default::default(), + declared: None, config, restart: 0, reorder: None, @@ -123,6 +127,7 @@ impl MkvTrack { ) -> Result<()> { self.release(name, jitter, true)?; self.clock = Default::default(); + self.declared = None; self.reorder = None; self.source = source.continuing(&self.source); self.finished = false; @@ -154,7 +159,49 @@ impl MkvTrack { } } + /// Start a group: release the last one's frames, then clock this one at the depth its + /// rendition declares now, or on the inferred clock when it declares none. + fn start_group(&mut self, name: &str, jitter: &mut crate::jitter::Buffer) -> Result<()> { + let depth = match &self.config { + Some(config) => crate::container::decode::declared(config)?, + None => None, + }; + // The inferred clock spans groups, so it is only flushed when a declaration replaces it. + if self.declared.is_some() || depth.is_some() { + self.release(name, jitter, true)?; + } + self.declared = depth.map(crate::container::decode::DeclaredClock::new); + Ok(()) + } + + /// Hold a video frame on its clock, then release what it settles. + fn push( + &mut self, + name: &str, + arrival: crate::jitter::Arrival, + jitter: &mut crate::jitter::Buffer, + ) -> Result<()> { + let pts = (arrival.item.timestamp.as_nanos() * 90_000 / 1_000_000_000) as u64; + match &mut self.declared { + Some(clock) => clock.push(arrival, pts)?, + None => { + let arrived = arrival.arrived; + self.clock.push(arrival, pts, arrived) + } + } + self.release(name, jitter, false) + } + fn release(&mut self, name: &str, jitter: &mut crate::jitter::Buffer, flush: bool) -> Result<()> { + if let Some(clock) = &mut self.declared { + while let Some((mut arrival, dts)) = clock.pop(flush)? { + if !self.finished && !flush { + arrival.arrived = arrival.arrived.max(self.latest); + } + arrival.decode = crate::jitter::Decode::from_ticks(dts); + jitter.push(name.to_owned(), arrival)?; + } + } let Some(config) = &self.config else { return Ok(()) }; let (floor, lookahead) = *self.reorder.get_or_insert_with(|| { let declared = self.source.description().and_then(|description| match config.codec { @@ -188,7 +235,7 @@ impl MkvTrack { if !self.finished && !flush { arrival.arrived = arrival.arrived.max(self.latest); } - arrival.decode = moq_net::Timestamp::from_scale(dts, 90_000)?; + arrival.decode = moq_net::Timestamp::from_scale(dts, 90_000)?.into(); jitter.push(name.to_owned(), arrival)?; } Ok(()) @@ -415,6 +462,16 @@ impl Export { self.jitter.len() + self.tracks.values().map(|track| track.clock.len()).sum::() } + /// How many video frames wait on a declared clock for their DTS. + #[cfg(test)] + pub(super) fn held(&self) -> usize { + self.tracks + .values() + .filter_map(|track| track.declared.as_ref()) + .map(|clock| clock.len()) + .sum() + } + #[cfg(test)] pub(super) fn with_replay(mut self) -> Self { self.jitter = self.jitter.replay(); @@ -439,7 +496,14 @@ impl Export { } let (frame, arrived) = match track.source.poll_event(waiter)? { Poll::Ready(Some(Event::Frame { frame, arrived })) => (frame, arrived.max(track.subscribed)), - Poll::Ready(Some(Event::GroupEnd | Event::FrameEnd { .. } | Event::Cut)) => continue, + // A group that ended or was cut short holds frames no later one can precede. + Poll::Ready(Some(Event::GroupEnd | Event::Cut)) => { + if track.declared.is_some() { + track.release(name, &mut self.jitter, true)?; + } + continue; + } + Poll::Ready(Some(Event::FrameEnd { .. })) => continue, Poll::Ready(None) => { track.finished = true; break; @@ -470,7 +534,7 @@ impl Export { crate::jitter::Arrival { arrived, read: moq_clock::now(), - decode: frame.timestamp, + decode: frame.timestamp.into(), restart: track.source.markers(), skip: track.source.skips(), cut: track.source.cuts(), @@ -483,16 +547,19 @@ impl Export { track.pending = Some(arrival); break; } - if track.kind == TrackKind::Video && !self.delay.is_zero() { - if arrival.restart != track.restart { - track.release(name, &mut self.jitter, true)?; - track.clock = Default::default(); - track.restart = arrival.restart; - } - let pts = (arrival.item.timestamp.as_nanos() * 90_000 / 1_000_000_000) as u64; - let arrived = arrival.arrived; - track.clock.push(arrival, pts, arrived); - track.release(name, &mut self.jitter, false)?; + let video = track.kind == TrackKind::Video; + if video && arrival.restart != track.restart { + track.release(name, &mut self.jitter, true)?; + track.clock = Default::default(); + track.declared = None; + track.restart = arrival.restart; + } + if video && arrival.item.keyframe { + track.start_group(name, &mut self.jitter)?; + } + // A declared depth is checked even when nothing is scheduled by decode time. + if video && (track.declared.is_some() || !self.delay.is_zero()) { + track.push(name, arrival, &mut self.jitter)?; } else { self.jitter.push(name.clone(), arrival)?; } diff --git a/rs/moq-mux/src/container/mkv/export_test.rs b/rs/moq-mux/src/container/mkv/export_test.rs index bf44550200c..7c41893d1f2 100644 --- a/rs/moq-mux/src/container/mkv/export_test.rs +++ b/rs/moq-mux/src/container/mkv/export_test.rs @@ -22,6 +22,119 @@ use crate::container::test_util::{IDR, Live, PPS, SPS, raw_frame, video_frame}; /// completeness has to be asked for, exactly as a real recorder does. const RECORDING_MAX_DELAY: std::time::Duration = std::time::Duration::from_secs(30); +/// An AVC rendition that declares a reorder depth of two. +fn declared_avc() -> Live { + Live::new(".avc1", |catalog, name| { + let mut config = hang::catalog::VideoConfig::new(hang::catalog::H264 { + profile: 100, + constraints: 0, + level: 10, + inline: false, + }); + config.coded_width = Some(64); + config.coded_height = Some(64); + config.reorder = Some(2); + config.container = hang::catalog::Container::Legacy; + config.description = Some( + crate::codec::h264::build_avcc( + &[Bytes::from_static(crate::codec::h264::fixtures::SPS_PYRAMID)], + &[Bytes::from_static(PPS)], + ) + .unwrap(), + ); + catalog.modify().unwrap().video.renditions.insert(name, config); + }) +} + +#[tokio::test(start_paused = true)] +async fn a_declared_depth_bounds_malformed_video_by_count_and_payload() { + for backwards in [false, true] { + for count in [16, 128, 1024] { + for size in [256, 16_384] { + let mut live = declared_avc(); + for index in 0..count { + let mut frame = raw_frame(1_000_000 - if backwards { index } else { 0 }, b"", index == 0); + let mut bytes = vec![0; size]; + bytes[..4].copy_from_slice(&((size - 4) as u32).to_be_bytes()); + bytes[4] = if index == 0 { 0x65 } else { 0x41 }; + frame.payload = bytes.into(); + live.track.write(frame).unwrap(); + } + live.track.finish().unwrap(); + live.catalog.finish().unwrap(); + let mut export = super::Export::new(live.source().await, live.catalog_stream().await) + .with_fragment_duration(std::time::Duration::ZERO) + .with_replay(); + export.next().await.unwrap().unwrap(); + let mut refused = false; + loop { + match export.next().await { + Err(err) => { + assert!(err.to_string().contains("video"), "{err}"); + refused = true; + break; + } + Ok(None) => break, + Ok(Some(_)) => {} + } + assert!(export.held() <= 2, "a depth-two clock"); + } + assert!(refused, "backwards={backwards} count={count} size={size}"); + assert!(export.held() <= 2 && export.buffered() <= 4); + } + } + } +} + +/// MKV retains PTS while signed, group-local DTS orders video against audio. +#[tokio::test(start_paused = true)] +async fn declared_reorder_starts_before_zero_and_resets_each_group() { + let mut live = declared_avc(); + let mut audio = live.add_track("audio", |catalog, name| { + let mut config = hang::catalog::AudioConfig::new(hang::catalog::AAC { profile: 2 }, 44100, 2); + config.description = Some(Bytes::from_static(&[0x12, 0x10])); + config.container = hang::catalog::Container::Legacy; + catalog.modify().unwrap().audio.renditions.insert(name, config); + }); + for start in [0, 400_000] { + for (index, offset) in [0, 160_000, 80_000, 40_000, 120_000].into_iter().enumerate() { + let payload = if index == 0 { + &b"\x00\x00\x00\x01\x65"[..] + } else { + &b"\x00\x00\x00\x01\x41"[..] + }; + live.track + .write(raw_frame(start + offset, payload, index == 0)) + .unwrap(); + } + audio.write(raw_frame(start, b"audio", true)).unwrap(); + } + live.track.finish().unwrap(); + audio.finish().unwrap(); + live.catalog.finish().unwrap(); + let mut export = super::Export::new(live.source().await, live.catalog_stream().await) + .with_delay(std::time::Duration::from_secs(5)) + .with_fragment_duration(std::time::Duration::ZERO) + .with_replay(); + export.next().await.unwrap().unwrap(); + let mut timestamps = Vec::new(); + while let Some(chunk) = export.next().await.unwrap() { + let mut cursor = Cursor::new(chunk); + let mut base = 0; + for tag in WebmIterator::new(&mut cursor, &[]) { + match tag.unwrap() { + MatroskaSpec::Timestamp(timestamp) => base = timestamp as i64, + MatroskaSpec::SimpleBlock(bytes) => { + let block = SimpleBlock::try_from(bytes.as_slice()).unwrap(); + timestamps.push(base + i64::from(block.timestamp)); + } + _ => {} + } + } + } + assert_eq!(timestamps, [0, 160, 80, 0, 40, 120, 400, 560, 480, 400, 440, 520]); +} + #[tokio::test(start_paused = true)] async fn export_header_roundtrip_vp9_opus() { // Build a tiny synthetic WebM with one VP9 video track and one Opus audio track. diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index ff9115ac00a..a02d6053c39 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -47,7 +47,7 @@ use crate::codec::annexb::InBandCodec; use crate::codec::video::{Hrd, Reorder}; use crate::codec::{aac, annexb, opus}; use crate::container::consumer::Event; -use crate::container::decode::{DecodeClock, MAX_REORDER, Settle}; +use crate::container::decode::{DeclaredClock, DecodeClock, MAX_REORDER, Settle}; use crate::container::{ExportSource, Frame}; use crate::jitter::{self, Arrival, Push}; @@ -221,6 +221,9 @@ struct Tally { tables: Option<(VersionNumber, psi::Pmt)>, } +/// A video frame waiting on a decode clock, with the codec config it was read under. +type Held = (Pending, Option); + /// A frame read from its source. struct Pending { frame: Frame, @@ -288,8 +291,12 @@ struct Track { /// PMT ES-level descriptors to re-announce, captured verbatim on import (language, /// registration, ...). Empty for non-TS sources; AC-3/E-AC-3 then synthesize one. descriptors: Vec, - /// Authors each video frame's DTS, holding the frames until it can. - clock: DecodeClock<(Pending, Option)>, + /// Authors each video frame's DTS, holding the frames until it can, for a rendition that + /// declares no `reorder`. + clock: DecodeClock, + /// The current group's clock, when its rendition declared `reorder` at the group's start. + /// A trimmed leading picture holds its slot as `None`. + declared: Option>>, /// Loss counter at the last read, excluding declared ends. recovery_gap: Option, /// Leading pictures below the recovery keyframe have unavailable references, and whether @@ -345,6 +352,7 @@ impl Track { kind, descriptors: Vec::new(), clock: DecodeClock::default(), + declared: None, recovery_gap: None, leading_floor: None, held: None, @@ -373,6 +381,7 @@ impl Track { self.release(name, jitter, true)?; self.restart = pending.restart; self.clock = DecodeClock::default(); + self.declared = None; } let Kind::Video(stream_type) = self.kind else { return self.push(name, pending, None, None, jitter); @@ -389,16 +398,34 @@ impl Track { && !radl(stream_type, self.source.description(), &pending.frame) { jitter.lose(name, ahead); - return Ok(()); + // It still takes its slot, so the group's other frames decode where they do on an + // exporter that kept it. + return match &mut self.declared { + Some(clock) => { + clock.push(None, to_ticks(pending.frame.timestamp))?; + self.release(name, jitter, false) + } + None => Ok(()), + }; } self.held = Some(pending.cut); if pending.frame.keyframe { self.timing.describe(stream_type, self.source.description(), name); + // A group decodes on its own clock at the depth its rendition declares now. The + // inferred clock spans groups, so it is only flushed when a declaration replaces it. + let depth = self.timing.reorder; + if self.declared.is_some() || depth.is_some() { + self.release(name, jitter, true)?; + } + self.declared = depth.map(DeclaredClock::new); } let pts = to_ticks(pending.frame.timestamp); let arrived = pending.arrived; - self.clock - .push((pending, self.source.description().cloned()), pts, arrived); + let item = (pending, self.source.description().cloned()); + match &mut self.declared { + Some(clock) => clock.push(Some(item), pts)?, + None => self.clock.push(item, pts, arrived), + } self.release(name, jitter, false) } @@ -608,7 +635,23 @@ impl Track { /// schedule that has already gone past them. Nothing settles the frames still held when the /// track ends, so they keep their own arrival and go out late if they must. fn release(&mut self, name: &str, jitter: &mut jitter::Buffer, flush: bool) -> anyhow::Result<()> { - let (delay, lookahead) = self.timing.reorder(); + if let Some(mut clock) = self.declared.take() { + let released = (|| { + while let Some((item, dts)) = clock.pop(flush)? { + let Some((mut pending, description)) = item else { + continue; + }; + if !self.finished { + pending.arrived = pending.arrived.max(self.latest); + } + self.push(name, pending, Some(dts), description, jitter)?; + } + anyhow::Ok(()) + })(); + self.declared = Some(clock); + released?; + } + let (delay, lookahead) = self.timing.inferred(); let lookahead = lookahead.max(self.timing.jitter); loop { let settle = match flush { @@ -621,17 +664,21 @@ impl Track { if !self.finished { pending.arrived = pending.arrived.max(self.latest); } - self.push(name, pending, Some(dts), description, jitter)?; + self.push(name, pending, Some(dts.into()), description, jitter)?; } Ok(()) } - /// Queue a frame decoding at `dts` (90 kHz ticks), else at its PTS. + /// Queue a frame decoding at `dts` (signed 90 kHz ticks), else at its PTS. + /// + /// A DTS before the timeline's zero, a declared group's leading frames at its start, wraps + /// into the 33-bit wire field like any other, and is scheduled from zero: the few ticks it + /// precedes zero by are below anything the T-STD resolves. fn push( &mut self, name: &str, pending: Pending, - dts: Option, + dts: Option, description: Option, jitter: &mut jitter::Buffer, ) -> anyhow::Result<()> { @@ -643,10 +690,12 @@ impl Track { arrived, read, } = pending; - let dts = dts.filter(|&dts| dts != to_ticks(frame.timestamp)); + let dts = dts.filter(|&dts| dts != i128::from(to_ticks(frame.timestamp))); let decode = dts - .and_then(|ticks| Timestamp::from_scale(ticks, 90_000).ok()) + .and_then(|ticks| Timestamp::from_scale(u64::try_from(ticks.max(0)).ok()?, 90_000).ok()) .unwrap_or(frame.timestamp); + // Two's complement, so the 33-bit mask wraps a DTS before zero as it would a later one. + let dts = dts.map(|ticks| ticks as u64); let (key, pes) = self.release_key(decode, jitter.delay(), &frame)?; let video = matches!(self.kind, Kind::Video(_)); let leading = video @@ -670,7 +719,7 @@ impl Track { let arrival = Arrival { arrived: arrived - ahead, read: read - ahead, - decode: key, + decode: key.into(), restart, skip, cut, @@ -709,6 +758,8 @@ struct Timing { declared: Option, /// The stream's level limits, from the catalog codec. level: Option, + /// The reorder depth the catalog declares, which clocks each group on its own. + reorder: Option, /// The HRD its SPS declares. hrd: Option, } @@ -716,10 +767,12 @@ struct Timing { impl Timing { /// Take the rendition's timing from a catalog snapshot, before or after the program /// tables are written. - fn configure(&mut self, config: &VideoConfig, name: &str) { + fn configure(&mut self, config: &VideoConfig, name: &str) -> crate::Result<()> { self.jitter = config.jitter.map_or(0, |t| (t.as_micros() * 90_000 / 1_000_000) as u64); self.framerate = config.framerate.filter(|fps| fps.is_finite() && *fps > 0.0); self.level = Some(video_level(config, name)); + self.reorder = crate::container::decode::declared(config)?; + Ok(()) } /// Read the reorder depth from the codec config a keyframe is carried with. @@ -730,7 +783,7 @@ impl Timing { self.description = description.cloned(); self.declared = description.and_then(|d| declared_reorder(stream_type, d)); self.hrd = description.and_then(|d| declared_hrd(stream_type, d)); - tracing::debug!(track = %name, reorder = ?self.reorder(), "video reordering declared"); + tracing::debug!(track = %name, reorder = ?self.inferred(), "video reordering declared"); } /// The receiver's buffers (H.222.0 2.14.3.1, 2.17.2): EB is the HRD's CPB, else the @@ -765,7 +818,7 @@ impl Timing { /// `framerate`'s. A frame can be presented up to `2^depth - 1` pictures below one decoded /// before it (a B-pyramid that deep), which bounds the lookahead the catalog `jitter` /// gives when it is not published. - fn reorder(&self) -> (u64, u64) { + fn inferred(&self) -> (u64, u64) { let Some(declared) = self.declared else { return (0, 0); }; @@ -1653,7 +1706,7 @@ where // `framerate` published after the tables still sizes the decode clock. for (name, config) in catalog.video.renditions.iter() { if let Some(track) = self.tracks.get_mut(name).filter(|track| track.listed) { - track.timing.configure(config, name); + track.timing.configure(config, name)?; } } // A track that leaves the catalog is read to its own end. A publisher retires a @@ -1753,7 +1806,7 @@ where }, }; track.kind = kind; - track.timing.configure(config, name); + track.timing.configure(config, name)?; self.refresh(name, track, pid, track_descriptors(&mpegts, name)); } for (name, config) in catalog.audio.renditions.iter() { @@ -1901,6 +1954,7 @@ where ); track.release(name, &mut self.jitter, true)?; track.clock = DecodeClock::default(); + track.declared = None; track.source = source.continuing(&track.source); track.finished = false; track.subscribed = moq_clock::now(); diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index c41b2294b40..765eaae1a29 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1149,6 +1149,154 @@ async fn export_decodes_where_the_source_did() { } } +/// A broadcast of one H.264 rendition declaring `reorder`, with a group of `stamps` (ms, in +/// decode order) for each entry, exported as a recording. +/// The producers stay alive with the export, so it reads the retained tracks. +async fn export_declared(reorder: u8, groups: &[&[u64]]) -> (Export, impl Sized) { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let track = broadcast + .create_track("video", hang::container::track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let mut config = VideoConfig::new(H264 { + profile: 0x64, + constraints: 0, + level: 0x1f, + inline: true, + }); + config.reorder = Some(reorder); + catalog + .modify() + .unwrap() + .video + .renditions + .insert("video".into(), config); + let format = HangContainer::Legacy(Kind::Video); + for (sequence, stamps) in groups.iter().enumerate() { + let mut group = track + .create_group(moq_net::group::Info { + sequence: sequence as u64, + }) + .unwrap(); + let frames: Vec<_> = stamps + .iter() + .enumerate() + .map(|(index, &ms)| Frame { + timestamp: Timestamp::from_millis(ms).unwrap(), + duration: None, + keyframe: index == 0, + payload: if index == 0 { + annexb(&[SPS, PPS, &[0x65, 0xab, 0xcd]]) + } else { + annexb(&[&[0x41, 0xab, 0xcd]]) + }, + }) + .collect(); + format.write(&mut group, &frames).unwrap(); + group.finish().unwrap(); + } + track.finish().unwrap(); + let export = open_export(crate::source::announced(&consumer).await) + .await + .unwrap() + .with_delay(RECORDING_MAX_AGE) + .with_replay(); + (export, (broadcast, catalog, track)) +} + +/// A rendition that declares its reorder depth decodes each group from that group's frames: its +/// first frames a tick apart below its earliest presentation, wrapping the 33-bit DTS below the +/// timeline's zero, and the rest at its presentation slots. +#[tokio::test(start_paused = true)] +async fn export_declared_reorder_decodes_each_group_from_its_frames() { + let (mut export, _keep) = export_declared(2, &[&[0, 160, 80, 40, 120], &[400, 560, 480, 440, 520]]).await; + let bytes: Vec = drain_frames(&mut export) + .await + .iter() + .flat_map(|frame| frame.payload.iter().copied()) + .collect(); + let wrap = 1 << 33; + let ms = |ms: u64| ms * 90; + assert_eq!( + video_pes_timing(&bytes), + [ + (ms(0), wrap - 2), + (ms(160), wrap - 1), + (ms(80), 0), + (ms(40), ms(40)), + (ms(120), ms(80)), + (ms(400), ms(400) - 2), + (ms(560), ms(400) - 1), + (ms(480), ms(400)), + (ms(440), ms(440)), + (ms(520), ms(480)), + ] + ); +} + +/// Open-GOP leading pictures an export trims, because it started at their group, still take +/// their slots: the group's other frames decode where they do on an export that read the group +/// before and kept them. +#[tokio::test(start_paused = true)] +async fn trimmed_leading_pictures_keep_their_slots() { + async fn timing(groups: &[&[u64]]) -> Vec<(u64, u64)> { + let (mut export, _keep) = export_declared(2, groups).await; + let bytes: Vec = drain_frames(&mut export) + .await + .iter() + .flat_map(|frame| frame.payload.iter().copied()) + .collect(); + video_pes_timing(&bytes) + } + let open: &[u64] = &[400, 320, 360, 560, 480, 440, 520]; + let kept = timing(&[&[0, 160, 80, 40, 120], open]).await; + let joined = timing(&[open]).await; + let ms = |ms: u64| ms * 90; + assert_eq!( + kept[5..], + [ + (ms(400), ms(320) - 2), + (ms(320), ms(320) - 1), + (ms(360), ms(320)), + (ms(560), ms(360)), + (ms(480), ms(400)), + (ms(440), ms(440)), + (ms(520), ms(480)), + ] + ); + let shared: Vec<_> = kept[5..] + .iter() + .filter(|(pts, _)| ![ms(320), ms(360)].contains(pts)) + .collect(); + assert_eq!(joined.iter().collect::>(), shared); +} + +/// A declared depth of zero decodes every frame at its presentation, and refuses a frame that +/// reorders, rather than inferring a decode delay. +#[tokio::test(start_paused = true)] +async fn export_declared_zero_reorder_decodes_at_presentation() { + let (mut export, _keep) = export_declared(0, &[&[0, 40, 80, 120]]).await; + let bytes: Vec = drain_frames(&mut export) + .await + .iter() + .flat_map(|frame| frame.payload.iter().copied()) + .collect(); + let timing = video_pes_timing(&bytes); + assert_eq!(timing.len(), 4); + assert!(timing.iter().all(|(pts, dts)| pts == dts), "{timing:?}"); + + let (mut export, _keep) = export_declared(0, &[&[0, 80, 40, 120]]).await; + let err = loop { + match tokio::time::timeout(DRAIN, export.next()).await { + Ok(Ok(Some(_))) => {} + Ok(Ok(None)) | Err(_) => panic!("a frame reordered past the declared depth was exported"), + Ok(Err(err)) => break err, + } + }; + assert!(err.to_string().contains("declared depth"), "{err}"); +} + /// #2937: the PCR must be a uniform bounded-interval ramp, not a sample of the /// per-unit decode clock. On a reordered (B-frame) capture the authored DTS is a /// saw (reference frames leap a reorder span, B-frames nudge one tick), so a PCR diff --git a/rs/moq-mux/src/container/ts/export_timing_test.rs b/rs/moq-mux/src/container/ts/export_timing_test.rs index f3b4e405d31..8c9bf3c0e4e 100644 --- a/rs/moq-mux/src/container/ts/export_timing_test.rs +++ b/rs/moq-mux/src/container/ts/export_timing_test.rs @@ -31,6 +31,11 @@ const VIDEO_US: u64 = 40_000; const AUDIO_US: u64 = 21_333; /// Video frames per group: 600 ms. const GOP: u64 = 15; +/// A B-pyramid group in decode order (`max_num_reorder_frames` 2): each frame's place in +/// presentation order. +const PYRAMID: [u64; GOP as usize] = [0, 4, 2, 1, 3, 8, 6, 5, 7, 12, 10, 9, 11, 14, 13]; +/// How many frames after its tick a B-pyramid frame is presented: its decode delay. +const PYRAMID_DELAY: u64 = 2; /// Audio frames per group, roughly matching the video group duration. const AUDIO_GROUP: u64 = 28; @@ -77,6 +82,8 @@ struct Live { /// negative sends the video later. A TS source sends video well ahead of its decode time /// and audio just in time, so its audio lags. audio_lag: i64, + /// Video is a B-pyramid: see [`Self::reorder`]. + reorder: bool, start: Instant, tick: u64, audio_index: u64, @@ -139,6 +146,7 @@ impl Live { scale, rate: MUX_RATE, audio_lag, + reorder: false, start: moq_clock::now(), tick: 0, audio_index: 0, @@ -170,6 +178,23 @@ impl Live { } } + /// Send video as a B-pyramid in decode order, each frame [`PYRAMID_DELAY`] frames after its + /// tick, under an SPS that declares the depth, and declare that depth in the catalog, so each + /// group decodes on its own. + fn reorder(&mut self) { + use crate::codec::h264::fixtures::{PPS as PYRAMID_PPS, SPS_PYRAMID}; + + let description = + crate::codec::h264::build_avcc(&[Bytes::from_static(SPS_PYRAMID)], &[Bytes::from_static(PYRAMID_PPS)]) + .unwrap(); + let mut catalog = self.catalog.modify().unwrap(); + for config in catalog.video.renditions.values_mut() { + config.description = Some(description.clone()); + config.reorder = Some(2); + } + self.reorder = true; + } + /// Send nothing for `gap`, as a publisher that died, then resume at the live edge on the /// same timeline from the next group, as its replacement or the other half of a 1+1 pair /// would. @@ -231,12 +256,16 @@ impl Live { } else { vec![0x41u8; 400] }; + let presented = match self.reorder { + true => (tick - tick % GOP + PYRAMID[(tick % GOP) as usize] + PYRAMID_DELAY) * VIDEO_US, + false => video, + }; let group = self.group.as_mut().expect("the first tick is a keyframe"); HangContainer::Legacy(crate::container::Kind::Video) .write( group, &[Frame { - timestamp: Timestamp::from_micros(video).unwrap(), + timestamp: Timestamp::from_micros(presented).unwrap(), duration: None, payload: length_prefixed(&[&slice]), keyframe, @@ -945,3 +974,50 @@ async fn live_audio_shares_its_pes() { } } } + +/// Each video frame's DTS (else PTS) by its PTS, in 90 kHz ticks, as a leg rendered it. +fn decode_times(leg: &Leg) -> std::collections::BTreeMap { + let bytes: Vec = leg + .out + .iter() + .flat_map(|(_, frame)| frame.payload.iter().copied()) + .collect(); + let mut reader = TsPacketReader::new(Cursor::new(bytes)); + let mut out = std::collections::BTreeMap::new(); + while let Some(packet) = reader.read_ts_packet().unwrap() { + if let Some(TsPayload::PesStart(pes)) = &packet.payload + && packet.header.pid.as_u16() == VIDEO_PID + && let Some(pts) = pes.header.pts + { + out.insert(pts.as_u64(), pes.header.dts.unwrap_or(pts).as_u64()); + } + } + out +} + +/// Legs that joined at different groups, one after reading a group cut short below a frame it +/// never received, author the same DTS for every frame both render once the rendition declares +/// its reorder depth: each group decodes from its own frames alone. +#[tokio::test(start_paused = true)] +async fn declared_legs_author_the_same_dts() { + let stall = |tick: u64| (2 * GOP + 8..3 * GOP).contains(&tick); + for join in [JOIN, 4 * GOP + 2] { + let mut live = Live::new(1.0, 0).await; + live.reorder(); + let mut a = live.join(Duration::ZERO).await; + live.run(join, &mut [&mut a], stall).await; + let mut b = live.join(Duration::from_millis(100)).await; + live.run(12 * GOP, &mut [&mut a, &mut b], stall).await; + live.finish(&mut [&mut a, &mut b]).await; + let (a, b) = (decode_times(&a), decode_times(&b)); + let shared: Vec<_> = b.iter().filter(|(pts, _)| a.contains_key(pts)).collect(); + assert!( + shared.len() > 6 * GOP as usize, + "join {join}: {} shared frames", + shared.len() + ); + for (pts, dts) in shared { + assert_eq!(a[pts], *dts, "join {join}: the frame presented at {pts}"); + } + } +} diff --git a/rs/moq-mux/src/container/ts/recovery_test.rs b/rs/moq-mux/src/container/ts/recovery_test.rs index 063f1903fe7..706ce7c4dca 100644 --- a/rs/moq-mux/src/container/ts/recovery_test.rs +++ b/rs/moq-mux/src/container/ts/recovery_test.rs @@ -1667,11 +1667,16 @@ async fn a_congested_path_sheds_what_it_made_late() { "the queue made pictures late" ); let lags = lags(&run); + // A group's leading pictures decode just before its keyframe is presented, so they leave + // later than the rest: compare the least lag of a whole group at each end. + let group = 12; + let (first, last) = ( + lags[..group].iter().min().unwrap(), + lags[lags.len() - group..].iter().min().unwrap(), + ); assert!( - lags.last().unwrap().abs_diff(lags[0]) <= FRAME, - "the latency recovers: {:?} at first, {:?} at last", - lags[0], - lags.last().unwrap() + last.abs_diff(*first) <= FRAME, + "the latency recovers: {first:?} at first, {last:?} at last" ); } diff --git a/rs/moq-mux/src/container/ts/release.rs b/rs/moq-mux/src/container/ts/release.rs index d3096e2b0f4..591cbdec939 100644 --- a/rs/moq-mux/src/container/ts/release.rs +++ b/rs/moq-mux/src/container/ts/release.rs @@ -159,7 +159,7 @@ impl Release { let arrival = Arrival { arrived: arrived.max(self.subscribed), read: moq_clock::now(), - decode: frame.timestamp, + decode: frame.timestamp.into(), restart: track.markers(), skip: track.skips(), cut: track.cuts(), diff --git a/rs/moq-mux/src/jitter.rs b/rs/moq-mux/src/jitter.rs index e141691ae1e..55ea0a3972d 100644 --- a/rs/moq-mux/src/jitter.rs +++ b/rs/moq-mux/src/jitter.rs @@ -10,6 +10,31 @@ use std::time::Duration; use moq_clock::{Instant, Sleep}; use moq_net::Timestamp; +/// An authored decode instant in signed nanoseconds, including times before PTS zero. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub(crate) struct Decode(i128); + +impl Decode { + pub(crate) fn from_nanos(nanos: i128) -> Self { + Self(nanos) + } + + /// The instant `ticks` of a 90 kHz clock, rounded down. + pub(crate) fn from_ticks(ticks: i128) -> Self { + Self((ticks * 1_000_000_000).div_euclid(90_000)) + } + + fn as_nanos(self) -> i128 { + self.0 + } +} + +impl From for Decode { + fn from(timestamp: Timestamp) -> Self { + Self::from_nanos(timestamp.as_nanos() as i128) + } +} + /// A frame handed to [`Buffer::push`]. pub(crate) struct Arrival { /// When the frame reached the reader: when it landed in the cache, or when the reader @@ -20,7 +45,7 @@ pub(crate) struct Arrival { /// so frames read together do not look fresher than they are. pub read: Instant, /// When the frame decodes, nondecreasing within a track. - pub decode: Timestamp, + pub decode: Decode, /// How many times the source had restarted its timeline when the frame was read. pub restart: u64, /// How many times the source's playhead had jumped (a skipped group, or a restart) @@ -200,7 +225,7 @@ const FIT: f64 = 1e-9; struct Clock { generation: u64, anchor: Instant, - base: Timestamp, + base: Decode, /// As a fraction of the decode timeline. drift: f64, } @@ -226,7 +251,7 @@ impl Acquire { /// decode time) is that track's slack less its queueing, and of those, the one with the least /// slack, so the track sent latest against its decode time still has the delay. fn fresh(&self) -> (&K, &Arrival) { - let behind = |frame: &Arrival| since(self.since, frame.read) - frame.decode.as_nanos() as i128; + let behind = |frame: &Arrival| since(self.since, frame.read) - frame.decode.as_nanos(); let mut freshest: BTreeMap<&K, &Arrival> = BTreeMap::new(); for (key, frame) in &self.frames { let best = freshest.entry(key).or_insert(frame); @@ -246,7 +271,7 @@ impl Acquire { fn end(&self, delay: Duration) -> Instant { let (_, fresh) = self.fresh(); let due = |frame: &Arrival| { - let ahead = frame.decode.as_nanos() as i128 - fresh.decode.as_nanos() as i128; + let ahead = frame.decode.as_nanos() - fresh.decode.as_nanos(); offset(fresh.read + delay, ahead) }; self.frames @@ -343,7 +368,7 @@ struct Floor { struct Track { /// Frames in arrival order, each with its generation, decode time and deadline. - queue: VecDeque<(u64, Timestamp, Instant, T)>, + queue: VecDeque<(u64, Decode, Instant, T)>, /// The source's restart, skip and cut counters at the last frame. restart: u64, skip: u64, @@ -814,18 +839,18 @@ impl Buffer { /// When a frame of `generation` decoding at `decode` goes out: on the current clock, or on the /// previous one while that generation's tail is still going out. - pub fn at(&self, generation: u64, decode: Timestamp) -> Option { + pub fn at(&self, generation: u64, decode: impl Into) -> Option { let clock = [self.clock, self.previous] .into_iter() .flatten() .find(|clock| clock.generation == generation)?; - self.deadline(&clock, decode) + self.deadline(&clock, decode.into()) } /// When a frame decoding at `decode` goes out on `clock`, if any instant holds it. - fn deadline(&self, clock: &Clock, decode: Timestamp) -> Option { + fn deadline(&self, clock: &Clock, decode: Decode) -> Option { // Signed, since a frame can decode before the one that anchored the clock. - let since = decode.as_nanos() as i128 - clock.base.as_nanos() as i128; + let since = decode.as_nanos() - clock.base.as_nanos(); offset( clock.anchor, since + (since as f64 * clock.drift) as i128 + self.delay.as_nanos() as i128, @@ -1144,6 +1169,61 @@ mod tests { const DELAY: Duration = Duration::from_millis(100); + #[moq_clock::test] + async fn decode_before_zero_keeps_its_deadline_and_media_order() { + let start = moq_clock::now(); + let mut buffer = Buffer::new(DELAY).replay(); + buffer.push(2, arrival(start, 0, "audio")).unwrap(); + let before_zero = Decode::from_nanos(-22_223); + buffer + .push( + 1, + Arrival { + decode: before_zero, + ..arrival(start, 0, "video") + }, + ) + .unwrap(); + assert_eq!( + buffer.at(0, before_zero), + Some(start + DELAY - Duration::from_nanos(22_223)) + ); + assert_eq!(buffer.at(0, ms(0)), Some(start + DELAY)); + moq_clock::sleep(DELAY).await; + assert_eq!(due(&mut buffer), ["video", "audio"]); + } + + #[moq_clock::test] + async fn moving_media_zero_preserves_acquisition_and_deadlines() { + let start = moq_clock::now(); + let mut negative = Buffer::new(DELAY); + let mut positive = Buffer::new(DELAY); + for buffer in [&mut negative, &mut positive] { + buffer.expect([1, 2]); + } + for (key, nanos, item) in [(1, -22_223, "video"), (2, 0, "audio"), (1, -11_112, "next")] { + for (buffer, offset) in [(&mut negative, 0), (&mut positive, 1_000_000_000)] { + buffer + .push( + key, + Arrival { + decode: Decode::from_nanos(nanos + offset), + ..arrival(start, 0, item) + }, + ) + .unwrap(); + } + } + moq_clock::sleep(DELAY).await; + assert_eq!(due(&mut negative), due(&mut positive)); + for nanos in [-22_223, -11_112, 0, 11_111] { + assert_eq!( + negative.at(0, Decode::from_nanos(nanos)), + positive.at(0, Decode::from_nanos(nanos + 1_000_000_000)), + ); + } + } + /// Two floors from the recovery arm's exact audio clock and wandering video lead. /// Both envelopes interpolate their two samples, so rounding in their equal depths /// must not make the changing send-ahead look like source clock drift. @@ -1168,7 +1248,7 @@ mod tests { Arrival { arrived, read: arrived, - decode: ms(decode), + decode: ms(decode).into(), restart: 0, skip: 0, cut: 0, @@ -1782,7 +1862,7 @@ mod tests { margin: None, offset: None, }; - let mut last: Option<(f64, Timestamp)> = None; + let mut last: Option<(f64, Decode)> = None; let mut stepped = false; for k in 0..(hours * 3_600.0 * 2.0) as u64 { let decode = Duration::from_millis(k * 500); @@ -1799,7 +1879,7 @@ mod tests { let frame = Arrival { arrived, read: arrived, - decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap(), + decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap().into(), restart: 0, skip: 0, cut: 0, @@ -1985,7 +2065,7 @@ mod tests { let frame = Arrival { arrived, read: arrived, - decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap(), + decode: Timestamp::from_micros(decode.as_micros() as u64).unwrap().into(), restart: 0, skip: 0, cut: 0, diff --git a/rs/moq-video/src/encode/producer.rs b/rs/moq-video/src/encode/producer.rs index 2f25402417c..84ca57a037a 100644 --- a/rs/moq-video/src/encode/producer.rs +++ b/rs/moq-video/src/encode/producer.rs @@ -59,6 +59,7 @@ fn rendition_hint(rendition: hang::catalog::VideoConfig) -> moq_mux::catalog::Vi hint.display_aspect_height = rendition.display_aspect_height; hint.color = rendition.color; hint.framerate = rendition.framerate; + hint.reorder = rendition.reorder; hint.bitrate = rendition.bitrate; hint.optimize_for_latency = rendition.optimize_for_latency; // Authoritative for both the catalog entry and the wire, so dropping it would silently