Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/concept/hang.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ A few things the catalog can express beyond decoder config:
- **Jitter.** A rendition can say how far its frames fell behind the media clock before the publisher flushed them, in whole milliseconds rounded up. Encoders report the spread of lateness above each rendition's own recent minimum, so a constant encoder delay is not jitter; container imports estimate batch spans without counting ingest delay. It describes the publisher, never the network, only grows over the life of a stream, and a player sizes its buffer to at least this much. A `0` is read as absent.
- **Delay.** A rendition can also say how far its frames reach the transport behind the broadcast's earliest rendition, measured the same way from each rendition's minimum lateness, so a video encoder running 200 ms behind audio advertises `delay: 200` on video. It follows the same rules as jitter. A player holds the largest `delay + jitter` among the renditions it subscribes to, and never subtracts one rendition's `delay` from another's.
- **Colour.** A video rendition may carry `color`: the H.273 primaries, transfer, matrix, and range, plus the HDR10 mastering display and content light levels, so a renderer can set up HDR without parsing the bitstream. The Rust importers fill it from the SPS VUI, AV1 sequence header, and VP9 header, then fill gaps from a container's `colr` box or Matroska `Colour` element; the stream wins a disagreement. The HDR10 values come from H.264 and H.265 SEI or AV1 metadata OBUs in MPEG-TS and raw elementary streams, an AV1 track's av1C, or Matroska, but not yet from H.264 or H.265 in fMP4 or FLV. Nothing in `@moq/watch` reads it yet.
- **Spatial media.** A video rendition may carry `projection` (flat `rectilinear`, or `equirectangular` with the degrees of longitude it spans, 180 for VR180) and `stereo` (`mono`, `left-right`, or `top-bottom` frame packing), and an audio rendition may carry `layout: "ambisonic"` for AmbiX channels. Readers treat a value they don't recognize as absent. Nothing in `@moq/watch` renders them yet.
- **Disabled renditions.** An audio or video rendition with `enabled: false` has no frames coming, and a player must not select it. Pausing a rendition in `@moq/publish` keeps it in the catalog this way rather than removing it, so a mute is a one-field update; a disabled video rendition encodes one black keyframe first, so a player that predates the field shows black instead of a frozen picture. The field is written only when false. The older video `stalled` field is no longer written, and readers ignore it.
- **Preference.** A video rendition may carry a signed `preference`, default `0`. A player keeps only the highest preference it can decode, then picks by size and bitrate within it, as do the single-rendition egresses (RTMP play, WHEP, FLV, the transcoder's source). A publisher that adds its own H.264 copy of an H.265 source marks the copy `-1`, so only a viewer that can't decode H.265 pulls it. Picking a rendition by name still overrides it. The field is written only when nonzero.
- **Archive.** A broadcast may advertise an `archive` entry mapping each track, the catalog included, to its own timeline track (a small index of the track's spans: timing plus group and frame positions) and fixed nominal record duration, plus the replay MoQ path, object-store URL, and format version when recorded. Media records end at the first group boundary at or beyond a 4s target; sparse timelines such as the catalog declare zero. Tracks commit and expire independently. An untimed track has no timing to index, so its timeline declares no duration and holds one record per group: it can be recorded and replayed, but not turned into HLS or kept in a DVR. The timelines are what let the [HLS gateway](/bin/hls) build playlists without subscribing to media. A reader that cannot decode the `archive` entry, such as one from a different version, ignores it and loses the timelines and replay, but live media still plays.
Expand Down
35 changes: 34 additions & 1 deletion drafts/draft-lcurley-moq-hang.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,8 @@ type VideoDecoderConfigExtensions = {
"displayAspectHeight": number | undefined,
"preference": number | undefined,
"color": Color | undefined,
"projection": Projection | undefined,
"stereo": string | undefined,
}
~~~

Expand Down Expand Up @@ -296,6 +298,25 @@ type Chromaticity = { "x": number, "y": number }
A publisher MUST omit a field its source leaves unspecified rather than write a default, and SHOULD prefer the value signalled in the elementary stream over a container's when they disagree.
`color` is not part of the track's identity ({{identity}}).

`projection` maps the decoded picture onto the viewer's field of view:

~~~
type Projection =
{ "kind": "rectilinear" } |
{ "kind": "equirectangular", "fov": number | undefined }
~~~

`rectilinear`, the default when absent, is a flat picture.
`equirectangular` maps longitude linearly across the width and latitude linearly down the height, spanning all 180 degrees of latitude around a viewer at the centre.
Its `fov` is the degrees of longitude the picture spans, centred straight ahead: greater than 0 and at most 360, defaulting to 360, so VR180 is 180.

`stereo` says how each eye's view is packed into the frame.
`mono`, the default when absent, shows the whole frame to both eyes.
`left-right` carries the left eye's view in the left half and the right eye's in the right half, and `top-bottom` carries the left eye's view in the top half and the right eye's in the bottom half.

A consumer MUST treat an unrecognized `projection` kind or `stereo` value as absent, and MUST reject a recognized `projection` kind with a malformed field, such as a `fov` out of range.
Neither field is part of the track's identity.

For example:

~~~
Expand Down Expand Up @@ -370,7 +391,18 @@ See the [WebCodecs specification](https://www.w3.org/TR/webcodecs/#audio-decoder
Any field carrying raw bytes, notably `description`, is a hex string ({{binary}}).
The `sampleRate` and `numberOfChannels` fields describe the decoded output: for HE-AAC, the SBR rate rather than the core's, and two channels under parametric stereo.

In addition to the WebCodecs fields, each rendition MAY carry the common rendition fields ({{common}}).
In addition to the WebCodecs fields, each rendition MAY carry the common rendition fields ({{common}}) plus:

~~~
type AudioDecoderConfigExtensions = {
"layout": string | undefined,
}
~~~

`layout` says what the decoded channels carry; when absent, they follow the codec's own channel order.
`ambisonic` is full-sphere ambisonics in the AmbiX convention: ACN channel order with SN3D normalization, of the order N for which `numberOfChannels` is (N+1)^2.
A consumer MUST reject `ambisonic` with fewer than 4 channels or a count that is not a square, and MUST treat an unrecognized `layout` as absent.
`layout` is not part of the track's identity.

### PCM {#audio-pcm}

Expand Down Expand Up @@ -1358,6 +1390,7 @@ This document has no IANA actions.
- Added an optional `track` field to video, audio, text, `json`, and `binary` entries naming the track in its broadcast; the entry's key is its ID and names the track when `track` is absent. A consumer MUST reject a catalog using one ID for both a video and an audio rendition.
- Defined video `displayAspectWidth` and `displayAspectHeight` as the whole picture's display aspect, not the sample aspect.
- Added the optional video `color` field: H.273 primaries, transfer, matrix, and range, plus the SMPTE ST 2086 mastering display and CTA-861.3 content light levels.
- Added the optional video `projection` (rectilinear, or equirectangular with a longitude `fov`) and `stereo` (mono, left-right, or top-bottom) fields and the optional audio `layout` field (ambisonic).

## moq-hang-03
{:numbered="false"}
Expand Down
18 changes: 18 additions & 0 deletions js/hang/src/catalog/audio.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,21 @@ test("audio warmup round trips as optional integer milliseconds", () => {
expect(() => AudioConfigSchema.parse({ ...base, warmup })).toThrow();
}
});

test("ambisonic layout needs a square channel count of at least 4", () => {
const base = { codec: "opus", container: { kind: "legacy" }, sampleRate: 48_000, layout: "ambisonic" };
for (const numberOfChannels of [4, 9, 16, 225]) {
expect(AudioConfigSchema.parse({ ...base, numberOfChannels }).layout).toBe("ambisonic");
}
for (const numberOfChannels of [0, 1, 2, 6, 8]) {
expect(() => AudioConfigSchema.parse({ ...base, numberOfChannels })).toThrow();
}
});

test("audio layout is optional and keeps unknown values verbatim", () => {
const base = { codec: "opus", container: { kind: "legacy" }, sampleRate: 48_000, numberOfChannels: 6 };
expect(AudioConfigSchema.parse(base).layout).toBeUndefined();
// An unknown layout puts no constraint on the channel count.
expect(AudioConfigSchema.parse({ ...base, layout: "surround" }).layout).toBe("surround");
expect(() => AudioConfigSchema.parse({ ...base, layout: 1 })).toThrow();
});
28 changes: 23 additions & 5 deletions js/hang/src/catalog/audio.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,7 @@ const TrackSchema = z.object({
name: z.string(),
});

/**
* Schema for a single audio rendition's decoder config.
* Mirrors WebCodecs AudioDecoderConfig (https://w3c.github.io/webcodecs/#audio-decoder-config).
*/
export const AudioConfigSchema = z.object({
const AudioConfigFields = z.object({
// Optional reference to another broadcast that publishes this track, expressed
// relative to the broadcast that served this catalog (e.g. "./source").
// If unset, the track lives in the same broadcast as the catalog.
Expand Down Expand Up @@ -44,6 +40,12 @@ export const AudioConfigSchema = z.object({
// The number of channels in the audio
numberOfChannels: u53Schema,

// What the decoded channels carry. Absent means the codec's own channel order. "ambisonic" is
// full-sphere AmbiX (ACN order, SN3D normalization) whose order follows from numberOfChannels,
// a square of at least 4. Typed as a plain string so an unrecognized layout survives a
// republish; a consumer treats one as absent.
layout: z.optional(z.string()),

// The maximum bitrate of the audio in bits per second.
// TODO: Support up to Number.MAX_SAFE_INTEGER
bitrate: z.optional(u53Schema),
Expand Down Expand Up @@ -84,6 +86,22 @@ export const AudioConfigSchema = z.object({
),
});

/**
* Schema for a single audio rendition's decoder config.
* Mirrors WebCodecs AudioDecoderConfig (https://w3c.github.io/webcodecs/#audio-decoder-config).
*/
export const AudioConfigSchema = AudioConfigFields.check(
z.refine((c) => c.layout !== "ambisonic" || isAmbisonic(c.numberOfChannels), {
message: "ambisonic audio needs a square channel count of at least 4",
}),
);

// (order + 1)² channels for an order of at least 1.
function isAmbisonic(channels: number): boolean {
const order = Math.round(Math.sqrt(channels));
return order >= 2 && order * order === channels;
}

/** Schema for the catalog audio section: a map of track name to rendition config. */
export const AudioSchema = z.union([
z.object({
Expand Down
4 changes: 2 additions & 2 deletions js/hang/src/catalog/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,5 @@ export * from "./root";
export * from "./stats";
export * from "./text";
export * from "./track";
export type { Color, VideoConfig } from "./video";
export { ColorSchema, VideoConfigSchema, VideoSchema } from "./video";
export type { Color, Projection, VideoConfig } from "./video";
export { ColorSchema, ProjectionSchema, VideoConfigSchema, VideoSchema } from "./video";
43 changes: 43 additions & 0 deletions js/hang/src/catalog/video.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -198,3 +198,46 @@ test("color refuses out-of-range code points and light levels", () => {
expect(parse({ range: "full" })).toBe(true);
expect(parse({ range: true })).toBe(false);
});

const vr = (fields: object) => VideoConfigSchema.parse({ codec: "vp8", container: { kind: "legacy" }, ...fields });

test("projection parses the known kinds and is optional", () => {
expect(vr({}).projection).toBeUndefined();
for (const projection of [
{ kind: "rectilinear" },
{ kind: "equirectangular" },
{ kind: "equirectangular", fov: 180 },
] as const) {
expect(vr({ projection }).projection).toEqual(projection);
}
expect(Catalog.ProjectionSchema.parse({ kind: "rectilinear" })).toEqual({ kind: "rectilinear" });
});

test("projection keeps an unknown kind verbatim and ignores unknown fields", () => {
const raw = { face: "front", kind: "cubemap" };
expect(vr({ projection: raw }).projection).toEqual({ kind: "unknown", raw });
// A newer publisher may add a parameter to a known kind; this reader must not refuse it.
expect(vr({ projection: { kind: "rectilinear", fov: 90 } }).projection).toEqual({ kind: "rectilinear" });
});

test("projection refuses a malformed known kind", () => {
expect(() => vr({ projection: { fov: 180 } })).toThrow();
expect(() => vr({ projection: { kind: 5 } })).toThrow();
for (const fov of ["wide", 0, -90, 360.5, Number.NaN, Number.POSITIVE_INFINITY]) {
expect(() => vr({ projection: { kind: "equirectangular", fov } })).toThrow();
}
for (const fov of [0.5, 180, 360]) {
expect(vr({ projection: { kind: "equirectangular", fov } }).projection).toEqual({
kind: "equirectangular",
fov,
});
}
});

test("stereo is optional and keeps unknown packings verbatim", () => {
expect(vr({}).stereo).toBeUndefined();
for (const stereo of ["mono", "left-right", "top-bottom", "multiview"]) {
expect(vr({ stereo }).stereo).toBe(stereo);
}
expect(() => vr({ stereo: 2 })).toThrow();
});
44 changes: 44 additions & 0 deletions js/hang/src/catalog/video.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,42 @@ export const ColorSchema = z.object({
/** The colour a rendition's pixels carry: H.273 code points plus static HDR metadata. */
export type Color = z.infer<typeof ColorSchema>;

/** The projection kinds this build knows. */
const KNOWN_PROJECTIONS = ["rectilinear", "equirectangular"];

/**
* Schema for how a rendition's picture maps onto the viewer's field of view. Absent means rectilinear.
*
* An unrecognized kind parses as `{ kind: "unknown", raw }` instead of throwing, and a consumer treats
* it as absent. A recognized kind only parses through its own schema, so a malformed one throws.
*/
export const ProjectionSchema = z.union([
z.discriminatedUnion("kind", [
// A flat picture.
z.object({ kind: z.literal("rectilinear") }),
// Longitude linear across the width and latitude down the height, the full 180 degrees,
// surrounding a viewer at its centre.
z.object({
kind: z.literal("equirectangular"),
// The degrees of longitude the picture spans, centred straight ahead. Absent means 360; VR180 is 180.
fov: z.optional(z.number().check(z.positive(), z.lte(360))),
}),
]),
z.pipe(
z.looseObject({
kind: z.string().check(
z.refine((kind) => !KNOWN_PROJECTIONS.includes(kind), {
message: "recognized projection kind must match its own schema",
}),
),
}),
z.transform((raw) => ({ kind: "unknown" as const, raw })),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep unknown projections in the wire shape

When ProjectionSchema sees a future projection such as { kind: "cubemap", face: "front" }, this transform returns { kind: "unknown", raw: ... } rather than the original wire object. A JS relay that republishes a parsed catalog passes it through Json.Snapshot.Encoder.update (js/json/src/snapshot/encoder.ts:201-205), which parses and stringifies that wrapper; parsing the republished catalog then wraps it again. Rust's UnknownProjection instead serializes the original object, so the implementations do not share the promised round-trip behavior. Keep the unknown output wire-shaped or unwrap raw during encoding.

AGENTS.md reference: AGENTS.md:L111-L111

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining for this PR. ProjectionSchema deliberately copies UnknownContainerSchema in js/hang/src/catalog/container.ts, which has had the same { kind: "unknown", raw } shape (and the same Rust/JS asymmetry) since unknown containers were added. The literal kind keeps kind === "equirectangular" narrowing, and raw is the documented way for a republisher to write the original back. No JS code reparses and republishes a catalog today: CatalogProducer builds its catalog from scratch and passes no schema to the encoder.

If we want a parsed JS catalog to stringify back to its wire shape, the fix belongs in both schemas at once, so I'll suggest it as a follow-up rather than diverge the two here.

(Written by Claude Opus 5.5)

),
]);

/** How a rendition's picture maps onto the viewer's field of view. */
export type Projection = z.infer<typeof ProjectionSchema>;

/** Schema for a single video rendition's decoder config. Mirrors WebCodecs VideoDecoderConfig. */
export const VideoConfigSchema = z.object({
// Optional reference to another broadcast that publishes this track, expressed
Expand Down Expand Up @@ -84,6 +120,14 @@ export const VideoConfigSchema = z.object({
// The colour the decoded pixels carry, so a renderer need not guess or reparse the bitstream.
color: z.optional(ColorSchema),

// How the picture maps onto the viewer's field of view. Absent means rectilinear.
projection: z.optional(ProjectionSchema),

// How each eye's view is packed into the frame: "mono", "left-right" (left eye in the left
// half), or "top-bottom" (left eye in the top half). Absent means mono. Typed as a plain string
// so an unrecognized packing survives a republish; a consumer treats one as absent.
stereo: z.optional(z.string()),

// The maximum frame rate of the video in frames per second.
framerate: z.optional(z.number()),

Expand Down
84 changes: 84 additions & 0 deletions rs/hang/src/catalog/audio/layout.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
use std::str::FromStr;

use derive_more::Display;

/// What a rendition's decoded channels carry, beyond the codec's own channel order.
///
/// Absent means the codec's channel order. An unrecognized value decodes to
/// [`AudioLayout::Unknown`] instead of failing, and a consumer treats it as absent.
#[derive(Debug, Clone, PartialEq, Eq, Display)]
#[non_exhaustive]
pub enum AudioLayout {
/// Full-sphere ambisonics in AmbiX: ACN channel order with SN3D normalization.
///
/// The order follows from the channel count, `(order + 1)²`, so the count is a square of at least 4.
#[display("ambisonic")]
Ambisonic,

/// A layout this build does not recognize, preserved verbatim.
#[display("{_0}")]
Unknown(String),
}

impl AudioLayout {
/// Refuse a channel count this layout cannot carry.
pub(crate) fn check(&self, channel_count: u32) -> crate::Result<()> {
if *self == Self::Ambisonic {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate Unknown layouts by their emitted name

When a programmatic publisher sets AudioLayout::Unknown("ambisonic".into()) with six channels, this variant equality skips the square-count check even though DisplayFromStr emits the recognized wire value "ambisonic". Outputs::emit in rs/moq-mux/src/catalog/producer.rs:212-214 consequently publishes a catalog that both Rust and JS consumers reject. Since Unknown(String) is publicly constructible, validate by the emitted name or prevent reserved names from entering that variant.

AGENTS.md reference: rs/AGENTS.md:L42-L43

Useful? React with 👍 / 👎.

let order = channel_count.isqrt();
if order < 2 || order * order != channel_count {
return Err(crate::Error::InvalidAmbisonicChannels(channel_count));
}
}
Ok(())
}
}

impl FromStr for AudioLayout {
type Err = crate::Error;

fn from_str(s: &str) -> Result<Self, Self::Err> {
Ok(match s {
"ambisonic" => Self::Ambisonic,
_ => Self::Unknown(s.to_string()),
})
}
}

#[cfg(test)]
mod test {
use crate::catalog::{AudioCodec, AudioConfig};

use super::*;

#[test]
fn round_trips() {
for (layout, json) in [
(AudioLayout::Ambisonic, "ambisonic"),
(AudioLayout::Unknown("surround".to_string()), "surround"),
] {
let mut config = AudioConfig::new(AudioCodec::Opus, 48_000, 4);
config.layout = Some(layout.clone());
let encoded = serde_json::to_value(&config).unwrap();
assert_eq!(encoded["layout"], json);
assert_eq!(
serde_json::from_value::<AudioConfig>(encoded).unwrap().layout,
Some(layout)
);
}
}

#[test]
fn ambisonic_needs_a_square_channel_count() {
for channels in [4, 9, 16, 225] {
AudioLayout::Ambisonic.check(channels).unwrap();
}
for channels in [0, 1, 2, 6, 8] {
assert!(matches!(
AudioLayout::Ambisonic.check(channels),
Err(crate::Error::InvalidAmbisonicChannels(c)) if c == channels
));
}
// An unknown layout puts no constraint on the channel count.
AudioLayout::Unknown("surround".to_string()).check(6).unwrap();
}
}
10 changes: 10 additions & 0 deletions rs/hang/src/catalog/audio/mod.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
mod aac;
mod codec;
mod layout;

pub use aac::*;
pub use codec::*;
pub use layout::*;

use std::collections::{BTreeMap, btree_map};

Expand Down Expand Up @@ -127,6 +129,13 @@ pub struct AudioConfig {
#[serde(rename = "numberOfChannels")]
pub channel_count: u32,

/// What the decoded channels carry, or the codec's own channel order when absent.
///
/// Live state, like video `color`.
#[serde(default)]
#[serde_as(as = "Option<DisplayFromStr>")]
pub layout: Option<AudioLayout>,

/// The maximum bitrate of the audio track in bits per second.
#[serde(default)]
pub bitrate: Option<u64>,
Expand Down Expand Up @@ -199,6 +208,7 @@ impl AudioConfig {
codec: codec.into(),
sample_rate,
channel_count,
layout: None,
bitrate: None,
enabled: true,
description: None,
Expand Down
Loading