Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/concept/hang.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ A few things the catalog can express beyond decoder config:
- **Track names.** The map key is a rendition's ID and, by default, its track name. A rendition sets `track` only when the two must differ, such as two renditions of same-named tracks in different broadcasts, or a rendition whose media changed and so needs a new name; readers subscribe to `track` when present. An ID can't name both a video and an audio rendition.
- **Jitter.** A rendition can say how far its frames fell behind the media clock before the publisher flushed them, in whole milliseconds rounded up. Encoders report the spread of lateness above each rendition's own recent minimum, so a constant encoder delay is not jitter; container imports estimate batch spans without counting ingest delay. It describes the publisher, never the network, only grows over the life of a stream, and a player sizes its buffer to at least this much. A `0` is read as absent.
- **Delay.** A rendition can also say how far its frames reach the transport behind the broadcast's earliest rendition, measured the same way from each rendition's minimum lateness, so a video encoder running 200 ms behind audio advertises `delay: 200` on video. It follows the same rules as jitter. A player holds the largest `delay + jitter` among the renditions it subscribes to, and never subtracts one rendition's `delay` from another's.
- **Colour.** A video rendition may carry `color`: the H.273 primaries, transfer, matrix, and range, plus the HDR10 mastering display and content light levels, so a renderer can set up HDR without parsing the bitstream. The Rust importers fill it from the SPS VUI, AV1 sequence header, and VP9 header, then fill gaps from a container's `colr` box or Matroska `Colour` element; the stream wins a disagreement. The HDR10 values come from H.264 and H.265 SEI or AV1 metadata OBUs in MPEG-TS and raw elementary streams, an AV1 track's av1C, or Matroska, but not yet from H.264 or H.265 in fMP4 or FLV. Nothing in `@moq/watch` reads it yet.
- **Disabled renditions.** An audio or video rendition with `enabled: false` has no frames coming, and a player must not select it. Pausing a rendition in `@moq/publish` keeps it in the catalog this way rather than removing it, so a mute is a one-field update; a disabled video rendition encodes one black keyframe first, so a player that predates the field shows black instead of a frozen picture. The field is written only when false. The older video `stalled` field is no longer written, and readers ignore it.
- **Preference.** A video rendition may carry a signed `preference`, default `0`. A player keeps only the highest preference it can decode, then picks by size and bitrate within it, as do the single-rendition egresses (RTMP play, WHEP, FLV, the transcoder's source). A publisher that adds its own H.264 copy of an H.265 source marks the copy `-1`, so only a viewer that can't decode H.265 pulls it. Picking a rendition by name still overrides it. The field is written only when nonzero.
- **Archive.** A broadcast may advertise an `archive` entry mapping each track, the catalog included, to its own timeline track (a small index of the track's spans: timing plus group and frame positions) and fixed nominal record duration, plus the replay MoQ path, object-store URL, and format version when recorded. Media records end at the first group boundary at or beyond a 4s target; sparse timelines such as the catalog declare zero. Tracks commit and expire independently. An untimed track has no timing to index, so its timeline declares no duration and holds one record per group: it can be recorded and replayed, but not turned into HLS or kept in a DVR. The timelines are what let the [HLS gateway](/bin/hls) build playlists without subscribing to media. A reader that cannot decode the `archive` entry, such as one from a different version, ignores it and loses the timelines and replay, but live media still plays.
Expand Down
55 changes: 55 additions & 0 deletions drafts/draft-lcurley-moq-hang.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,28 @@ normative:
date: false

informative:
h273:
title: "Coding-independent code points for video signal type identification"
target: https://www.itu.int/rec/T-REC-H.273
author:
- org: ITU-T
seriesinfo:
ITU-T Recommendation: H.273
date: false
st2086:
title: "Mastering Display Color Volume Metadata Supporting High Luminance and Wide Color Gamut Images"
author:
- org: SMPTE
seriesinfo:
SMPTE: ST 2086
date: false
cta861:
title: "HDR Static Metadata Extensions"
author:
- org: CTA
seriesinfo:
CTA: 861.3
date: false

--- abstract

Expand Down Expand Up @@ -208,6 +230,7 @@ type VideoDecoderConfigExtensions = {
"displayAspectWidth": number | undefined,
"displayAspectHeight": number | undefined,
"preference": number | undefined,
"color": Color | undefined,
}
~~~

Expand All @@ -222,6 +245,37 @@ A consumer selecting a rendition automatically MUST choose among the enabled ({{
A consumer MAY select another rendition at the viewer's explicit request.
A consumer that carries every rendition, rather than selecting one, ignores the field.

`color` describes the decoded pixels so a renderer can configure its pipeline without parsing the bitstream:

~~~
type Color = {
"primaries": number | undefined,
"transfer": number | undefined,
"matrix": number | undefined,
"range": "limited" | "full" | undefined,
"mastering": {
"red": Chromaticity,
"green": Chromaticity,
"blue": Chromaticity,
"white": Chromaticity,
"maxLuminance": number,
"minLuminance": number,
} | undefined,
"contentLight": {
"maxCll": number,
"maxFall": number,
} | undefined,
}

type Chromaticity = { "x": number, "y": number }
~~~

`primaries`, `transfer`, and `matrix` are the ColourPrimaries, TransferCharacteristics, and MatrixCoefficients code points of {{h273}}, and `range` is its VideoFullRangeFlag: `full` for 1 and `limited` for 0.
`mastering` is the {{st2086}} mastering display: CIE 1931 xy chromaticities and luminance in candelas per square metre.
`contentLight` is the {{cta861}} MaxCLL and MaxFALL in candelas per square metre, where 0 means unknown.
A publisher MUST omit a field its source leaves unspecified rather than write a default, and SHOULD prefer the value signalled in the elementary stream over a container's when they disagree.
`color` is not part of the track's identity ({{identity}}).

For example:

~~~
Expand Down Expand Up @@ -1281,6 +1335,7 @@ This document has no IANA actions.
- Audio `sampleRate` and `numberOfChannels` describe the decoded output, so HE-AAC names its SBR rate and parametric stereo names two channels.
- Added an optional `track` field to video, audio, text, `json`, and `binary` entries naming the track in its broadcast; the entry's key is its ID and names the track when `track` is absent. A consumer MUST reject a catalog using one ID for both a video and an audio rendition.
- Defined video `displayAspectWidth` and `displayAspectHeight` as the whole picture's display aspect, not the sample aspect.
- Added the optional video `color` field: H.273 primaries, transfer, matrix, and range, plus the SMPTE ST 2086 mastering display and CTA-861.3 content light levels.

## moq-hang-03
{:numbered="false"}
Expand Down
4 changes: 2 additions & 2 deletions js/hang/src/catalog/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,5 @@ export * from "./root";
export * from "./stats";
export * from "./text";
export * from "./track";
export type { VideoConfig } from "./video";
export { VideoConfigSchema, VideoSchema } from "./video";
export type { Color, VideoConfig } from "./video";
export { ColorSchema, VideoConfigSchema, VideoSchema } from "./video";
32 changes: 32 additions & 0 deletions js/hang/src/catalog/video.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,35 @@ test("video warmup round trips as optional integer milliseconds", () => {
expect(() => VideoConfigSchema.parse({ ...base, warmup })).toThrow();
}
});

test("color parses the H.273 code points and HDR10 metadata a Rust publisher writes", () => {
const color = {
primaries: 9,
transfer: 16,
matrix: 9,
range: "limited",
mastering: {
red: { x: 0.708, y: 0.292 },
green: { x: 0.17, y: 0.797 },
blue: { x: 0.131, y: 0.046 },
white: { x: 0.3127, y: 0.329 },
maxLuminance: 1000,
minLuminance: 0.005,
},
contentLight: { maxCll: 1000, maxFall: 400 },
};
const config = VideoConfigSchema.parse({ codec: "hvc1.2.4.L93.B0", container: { kind: "legacy" }, color });
expect(config.color).toEqual(color as typeof config.color);
expect(Catalog.ColorSchema.parse(color)).toEqual(config.color as Catalog.Color);
expect(VideoConfigSchema.parse({ codec: "vp8", container: { kind: "legacy" } }).color).toBeUndefined();
});

test("color refuses out-of-range code points and light levels", () => {
const parse = (color: unknown) =>
VideoConfigSchema.safeParse({ codec: "vp8", container: { kind: "legacy" }, color }).success;
expect(parse({ primaries: 256 })).toBe(false);
expect(parse({ contentLight: { maxCll: 65536, maxFall: 0 } })).toBe(false);
expect(parse({ matrix: 0 })).toBe(true);
expect(parse({ range: "full" })).toBe(true);
expect(parse({ range: true })).toBe(false);
});
42 changes: 41 additions & 1 deletion js/hang/src/catalog/video.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,51 @@ import * as z from "@zod/mini";
import { compareName, compareNumber } from "./compare";
import { ContainerSchema } from "./container";
import { hexSchema } from "./hex";
import { u53Schema } from "./integers";
import { u8Schema, u53Schema } from "./integers";
import { RelativeBroadcastSchema } from "./path";

// Backwards compatibility: old track schema
const TrackSchema = z.object({
name: z.string(),
});

// A CIE 1931 xy chromaticity coordinate.
const ChromaticitySchema = z.object({ x: z.number(), y: z.number() });

// A CTA-861.3 light level in candelas per square metre; zero means unknown.
const lightSchema = z.number().check(z.int(), z.nonnegative(), z.lte(0xffff));

/**
* Schema for the colour a rendition's pixels carry. The code points are ITU-T H.273's, which every
* codec and container shares; an absent field was left unspecified by the source.
*/
export const ColorSchema = z.object({
// H.273 ColourPrimaries: 1 is BT.709, 9 is BT.2020.
primaries: z.optional(u8Schema),
// H.273 TransferCharacteristics: 1 is BT.709, 16 is PQ, 18 is HLG.
transfer: z.optional(u8Schema),
// H.273 MatrixCoefficients: 0 is RGB, 1 is BT.709, 9 is BT.2020 non-constant luminance.
matrix: z.optional(u8Schema),
// The numeric range of the samples: studio swing, or every code used.
range: z.optional(z.enum(["limited", "full"])),
// The display the content was mastered on, as SMPTE ST 2086, luminance in cd/m².
mastering: z.optional(
z.object({
red: ChromaticitySchema,
green: ChromaticitySchema,
blue: ChromaticitySchema,
white: ChromaticitySchema,
maxLuminance: z.number(),
minLuminance: z.number(),
}),
),
// The content's light levels, as CTA-861.3.
contentLight: z.optional(z.object({ maxCll: lightSchema, maxFall: lightSchema })),
});

/** The colour a rendition's pixels carry: H.273 code points plus static HDR metadata. */
export type Color = z.infer<typeof ColorSchema>;
Comment thread
kixelated marked this conversation as resolved.
Resolved

/** Schema for a single video rendition's decoder config. Mirrors WebCodecs VideoDecoderConfig. */
export const VideoConfigSchema = z.object({
// Optional reference to another broadcast that publishes this track, expressed
Expand Down Expand Up @@ -44,6 +81,9 @@ export const VideoConfigSchema = z.object({
displayAspectWidth: z.optional(u53Schema),
displayAspectHeight: z.optional(u53Schema),

// The colour the decoded pixels carry, so a renderer need not guess or reparse the bitstream.
color: z.optional(ColorSchema),

// The maximum frame rate of the video in frames per second.
framerate: z.optional(z.number()),

Expand Down
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,6 @@ ranked just below deterministic TS legs, which it requires.
- [Go per-platform lib modules](/quest/m2/go-ffi-platform-modules.md) - a Go build downloads only the host's staticlib, and `go mod vendor` works
- [Dart on iOS](/quest/m2/dart-ios.md) - prove the shipped iOS native asset actually loads on a device, which no CI can
- [Mobile completion](/quest/m2/mobile-completion.md) - verify the Swift and Kotlin bindings over Rust-owned capture before closing #700
- [Catalog colour](/quest/m2/color-catalog.md) - the catalog describes a rendition's colour and HDR properties, which the WebGPU HDR renderer reads
- [WebGPU HDR](/quest/m2/webgpu-hdr.md) - HDR renditions play as HDR where the browser and display can show it, and tone-map to SDR elsewhere
- [Grant stats label](/quest/m2/grant-label.md) - an auth grant's optional `label` names what its session's presence and live bytes count under in stats, apart from the path scope it may access
- [RTSP import](/quest/m2/rtsp-import.md) - `moq import rtsp` publishes an IP camera from its own network, through a `moq-rtsp` crate whose one-session ingest a caller can supervise itself
Expand Down
58 changes: 0 additions & 58 deletions quest/m2/color-catalog.md

This file was deleted.

6 changes: 1 addition & 5 deletions quest/m2/webgpu-hdr.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Facts (2026-10):
has less headroom.
- `VideoFrame.colorSpace` gives primaries, transfer, matrix, and range
(Chrome 94, Firefox 130, Safari 16.4). Mastering display and content light
levels come from [Catalog colour](/quest/m2/color-catalog.md).
levels come from the rendition's catalog `color`.

Settle while building: whether the HDR path gives up zero copy (planes via
`copyTo` and a custom conversion shader), and how the renderer detects
Expand All @@ -40,7 +40,3 @@ Tests: an HDR10 rendition renders extended on Chromium desktop and on Safari
where extended output is unavailable, and an SDR rendition is unchanged.

Public API: none beyond the renderer. Wire: none.

## Required

- [Catalog colour](/quest/m2/color-catalog.md) - the mastering display and content light levels
1 change: 1 addition & 0 deletions rs/hang/src/catalog/root.rs
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,7 @@ mod test {
coded_height: None,
display_aspect_width: None,
display_aspect_height: None,
color: None,
bitrate: None,
enabled: true,
preference: 0,
Expand Down
Loading
Loading