Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/lib/rs/moq-mux.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Turns existing container formats into hang broadcasts and back. This is what
| Format | Import | Export | Notes |
| --- | --- | --- | --- |
| fMP4 / CMAF | yes | yes | Passthrough as `cmaf` or repackaged as `legacy`. |
| MPEG-TS | yes | yes | H.264/H.265; AAC, MP2, AC-3, E-AC-3; SCTE-35 and subtitle PIDs carried as tracks; service tables round-trip; signalled timebase discontinuities preserved; paced export. |
| MPEG-TS | yes | yes | H.264/H.265; AAC, MP2, AC-3, E-AC-3, Opus up to 7.1; SCTE-35 and subtitle PIDs carried as tracks; service tables round-trip; signalled timebase discontinuities preserved; paced export. |
| FLV / RTMP | yes | yes | Legacy H.264 + AAC + MP3, plus enhanced-RTMP HEVC, AV1, VP9, Opus, AC-3, E-AC-3, and multitrack. |
| Matroska / WebM | yes | yes | |
| Annex-B (H.264, H.265) | yes | yes | Parameter sets extracted to the catalog or re-injected per keyframe. |
Expand Down
4 changes: 3 additions & 1 deletion quest/m1/audio-codecs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,12 @@ its own decode and encode quest so verification stays per host.

- [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers
- [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core
- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the TS exporter refuses Opus heads its channel code cannot describe instead of mislabeling them
- [TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) - TS Opus with a channel code of 0x81 or above imports with its real head or is refused, never guessed as stereo
- [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC
- [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes
- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - `Config::encode` refuses channel counts it cannot name instead of writing stereo
- [AAC parse truncated SBR](/quest/m1/audio-codecs/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core
- [GStreamer surround Opus](/quest/m1/audio-codecs/gst-opus-surround.md) - the moq-gst sink publishes 3 to 8 channel Opus with the OpusHead its caps describe

## Related

Expand Down
21 changes: 21 additions & 0 deletions quest/m1/audio-codecs/gst-opus-surround.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# [S] Surround Opus from GStreamer

## Goal

`moq-gst`'s sink publishes a 3 to 8 channel `audio/x-opus` stream with the
OpusHead its caps describe, so surround Opus from `opusenc` plays natively like
the same stream from Matroska or MPEG-TS.

## Plan

- The sink refuses more than two channels because it builds the head from
`channels` and `rate` alone. Multichannel Opus caps also carry
`channel-mapping-family`, `stream-count`, `coupled-count`, and
`channel-mapping`; build the mapping with `opus::Mapping::new`, which
validates them, and refuse caps that omit or contradict them.
- Prefer a `streamheader` OpusHead when the caps carry one, if that path is
simpler and covers the same streams.
- Regression: an `opusenc` 5.1 pipeline publishes a family 1 description that
`moq-audio` decodes to six channels.

Public API: none. Wire: none.
27 changes: 27 additions & 0 deletions quest/m1/audio-codecs/ts-opus-channel-codes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# [S] TS Opus import refuses or parses extended channel codes

## Goal

An MPEG-TS Opus stream whose extension descriptor carries a
`channel_config_code` of 0x81 or above imports with its real OpusHead or is
refused. It never decodes as a guessed stereo stream, which misreads every
multistream packet.

## Plan

- The importer falls back to stereo for codes it does not know. 0x81 is
specified as an explicitly coded layout (channel count, mapping family, stream
counts, and table in the descriptor), and ffmpeg's muxer also writes
`0x80 | channels` for an alternate table its own demuxer does not read. Check
the Opus-in-TS spec and ffmpeg before deciding which codes to parse.
- Build parsed layouts with the public `opus::Mapping::new`, which validates
the table; refuse any code that is reserved or does not parse, per stream,
without failing the rest of the program.
- Fixtures from a real muxer where one exists; a hand-built descriptor
otherwise, noted as such.

Public API: none. Wire: none.

## Related

- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the exporter side of the same descriptor
23 changes: 23 additions & 0 deletions quest/m1/audio-codecs/ts-opus-export-refusals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# [S] TS Opus export refuses what it cannot label

## Goal

`moq-mux`'s MPEG-TS exporter never labels an Opus track with a channel
configuration its packets do not have. A track whose OpusHead the Opus
extension descriptor can describe exports as today; anything else is refused,
not written with a guessed `channel_config_code`.

## Plan

- The exporter derives `channel_config_code` from the catalog channel count
alone, clamped to 1..=8, so a family 255 or ambisonics head, a family 1 head
with a non-Vorbis table, or more than eight channels exports mislabeled.
Decide from the parsed description: family 0, and family 1 with the Vorbis
default mapping, keep the plain code; refuse the rest, or write the explicit
layout if the import side of
[TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) settles
one both sides agree on.
- A track with no description stays mono or stereo only.
- Tests with real heads, checked against ffprobe where ffmpeg reads the result.

Public API: none. Wire: none; TS output changes only for inputs it mislabeled.
20 changes: 0 additions & 20 deletions quest/m1/audio-codecs/ts-opus-surround.md

This file was deleted.

64 changes: 64 additions & 0 deletions rs/moq-audio/tests/import.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
//! Container imports that must decode end to end: a stream `moq-mux` imports
//! plays through [`decode::Consumer`] like the same stream from any container.

use std::time::Duration;

use moq_audio::{Format, Layout, decode};

/// MPEG-TS names surround Opus only by channel count, so the importer has to
/// synthesize the OpusHead mapping table or the decoder refuses the track.
#[tokio::test]
async fn ts_surround_opus_decodes() {
// A 440 Hz center channel in 5.1; see `import_opus_surround_catalog` in moq-mux.
let data = include_bytes!("../../moq-mux/src/container/ts/test_data/opus_5_1.ts");

let mut broadcast = moq_net::broadcast::Info::new().produce();
let broadcast_consumer = broadcast.consume();
let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap();
let mut import = moq_mux::container::ts::Import::new(broadcast, catalog.reserve());
import.decode(&data[..]).unwrap();
import.finish().unwrap();

let snapshot = catalog.snapshot();
let (name, config) = snapshot.audio.renditions.iter().next().expect("an Opus track");

let mut options = decode::Options::default();
options.output.format = Format::F32;
// The whole file is imported before anything decodes it.
options.max_age = Duration::from_secs(30);
let mut consumer = decode::Consumer::new(&broadcast_consumer, config, name, options)
.await
.unwrap();
assert_eq!(consumer.layout(), Layout::FivePointOne);

// Canonical 5.1 order: FL, FR, FC, LFE, SL, SR.
let mut energy = [0.0f64; 6];
let mut frames = 0;
while let Some(frame) = tokio::time::timeout(Duration::from_secs(5), consumer.read())
.await
.expect("decoded frame timed out")
.unwrap()
{
let (samples, rest) = frame.data.as_chunks::<4>();
assert!(rest.is_empty());
for chunk in samples.as_chunks::<6>().0 {
for (channel, sample) in energy.iter_mut().zip(chunk) {
*channel += f32::from_le_bytes(*sample).powi(2) as f64;
}
frames += 1;
}
}

assert!(frames > 12_000, "expected most of 0.5 s at 48 kHz, got {frames} frames");
let center = energy[2];
assert!(
center / frames as f64 > 0.001,
"center should carry the tone: {energy:?}"
);
for (channel, &other) in energy.iter().enumerate().filter(|&(channel, _)| channel != 2) {
assert!(
other < center / 100.0,
"channel {channel} should be near silent: {energy:?}"
);
}
}
2 changes: 1 addition & 1 deletion rs/moq-mux/src/codec/opus/import.rs
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ impl Import {
pub fn config(init: &[u8]) -> crate::Result<hang::catalog::AudioConfig> {
let mut buf = init;
let mut config: hang::catalog::AudioConfig = Config::parse(&mut buf)?.into();
// Publish the head as given: re-encoding it would drop a channel mapping table.
// Publish the head as given: re-encoding it would drop the output gain.
config.description = Some(bytes::Bytes::copy_from_slice(init));
Ok(config)
}
Expand Down
179 changes: 179 additions & 0 deletions rs/moq-mux/src/codec/opus/mapping.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
//! The OpusHead channel mapping table (RFC 7845 §5.1.1).

use bytes::Buf;

use super::{Error, Result};

/// An OpusHead channel mapping table (RFC 7845 §5.1.1), present for every family but 0.
#[derive(Clone, Copy, PartialEq, Eq)]
pub struct Mapping {
family: u8,
streams: u8,
coupled: u8,
channels: u8,
// Sized for the most channels any family allows, so `opus::Config` stays `Copy`.
table: [u8; 255],
}

/// The fields of a channel mapping table, checked into a [`Mapping`] by [`Mapping::new`].
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Config<'a> {
/// The channel mapping family; 0 has no table and is refused.
pub family: u8,
/// Number of Opus streams in each packet.
pub streams: u8,
/// How many of those streams are coupled (stereo); they come first.
pub coupled: u8,
/// The decoded channel feeding each output channel, 255 for silence.
pub table: &'a [u8],
}

/// The family 1 stream counts and table for 1 to 8 channels, as libopus and
/// RFC 7845 §5.1.1.2 lay out the Vorbis channel orders.
const VORBIS: [(u8, u8, &[u8]); 8] = [
(1, 0, &[0]),
(1, 1, &[0, 1]),
(2, 1, &[0, 2, 1]),
(2, 2, &[0, 1, 2, 3]),
(3, 2, &[0, 4, 1, 2, 3]),
(4, 2, &[0, 4, 1, 2, 3, 5]),
(4, 3, &[0, 4, 1, 2, 3, 5, 6]),
(5, 3, &[0, 6, 1, 2, 3, 4, 5, 7]),
];

impl Mapping {
/// The family 1 mapping a surround encoder uses for `channels` in Vorbis
/// order, for sources that name only a channel count.
pub(crate) fn vorbis(channels: u8) -> Result<Self> {
let (streams, coupled, table) = *channels
.checked_sub(1)
.and_then(|index| VORBIS.get(index as usize))
.ok_or(Error::UnsupportedChannelCount(channels as u32))?;
Self::new(Config {
family: 1,
streams,
coupled,
table,
})
}

/// Check `config` into a mapping with one output channel per table entry.
///
/// Refuses family 0, a channel count the family does not allow, and a table
/// that names no streams, more coupled streams than streams, or a channel
/// past the decoded ones.
pub fn new(config: Config<'_>) -> Result<Self> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep Mapping::new private until a consumer lands

At this commit, repo-wide callers of this constructor are limited to internal parsing, Mapping::vorbis, and this module's tests; the only external use described is a future GStreamer quest. Publishing mapping::Config and Mapping::new now unnecessarily freezes this shape into moq-mux's API even though the TS change can remain crate-private, so keep them private until that consumer lands.

AGENTS.md reference: AGENTS.md:L55-L60

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping it public: this was a deliberate call recorded under Decisions in the PR description. opus::Config::mapping is already a public field and encode now emits any mapping, so without a public constructor an outside crate could only obtain a Mapping by parsing a head it already has. Mapping::new validates every field (the struct fields stay private), and the GStreamer surround quest in this PR (quest/m1/audio-codecs/gst-opus-surround.md) is the planned consumer. Mapping::vorbis stays crate-private since nothing outside moq-mux needs it yet.

(Written by Claude Opus 5.5)

let Config {
family,
streams,
coupled,
table,
} = config;
let max_channels = match family {
0 => return Err(Error::UnsupportedMappingFamily(0)),
1 => 8,
_ => 255,
};
if table.is_empty() || table.len() > max_channels {
return Err(Error::UnsupportedChannelCount(table.len() as u32));
}

let decoded = streams as u32 + coupled as u32;
if streams == 0 || coupled > streams || decoded > 255 {
return Err(Error::InvalidMappingTable);
}
if table.iter().any(|&entry| entry != 255 && entry as u32 >= decoded) {
return Err(Error::InvalidMappingTable);
}

let mut padded = [0u8; 255];
padded[..table.len()].copy_from_slice(table);
Ok(Self {
family,
streams,
coupled,
channels: table.len() as u8,
table: padded,
})
}

pub(super) fn parse<T: Buf>(buf: &mut T, family: u8, channels: u8) -> Result<Self> {
if buf.remaining() < 2 + channels as usize {
return Err(Error::MappingTableTooShort);
}
let streams = buf.get_u8();
let coupled = buf.get_u8();
let mut table = [0u8; 255];
buf.copy_to_slice(&mut table[..channels as usize]);
Self::new(Config {
family,
streams,
coupled,
table: &table[..channels as usize],
})
}

/// The channel mapping family: 1 is the Vorbis speaker order, 2 and 3 are
/// ambisonics, and 255 is channels with no declared position.
pub fn family(&self) -> u8 {
self.family
}

/// Number of Opus streams in each packet.
pub fn streams(&self) -> u8 {
self.streams
}

/// How many of those streams are coupled (stereo); they come first.
pub fn coupled(&self) -> u8 {
self.coupled
}

/// The decoded channel feeding each output channel, 255 for silence.
pub fn table(&self) -> &[u8] {
&self.table[..self.channels as usize]
}
}

impl std::fmt::Debug for Mapping {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Mapping")
.field("family", &self.family)
.field("streams", &self.streams)
.field("coupled", &self.coupled)
.field("table", &self.table())
.finish()
}
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn new_validates_the_table() {
fn new(family: u8, streams: u8, coupled: u8, table: &[u8]) -> Result<Mapping> {
Mapping::new(Config {
family,
streams,
coupled,
table,
})
}

let mapping = new(255, 2, 0, &[0, 1, 255]).unwrap();
assert_eq!(mapping.family(), 255);
assert_eq!((mapping.streams(), mapping.coupled()), (2, 0));
assert_eq!(mapping.table(), &[0, 1, 255]);

assert!(matches!(new(0, 1, 1, &[0, 1]), Err(Error::UnsupportedMappingFamily(0))));
assert!(matches!(new(1, 1, 0, &[]), Err(Error::UnsupportedChannelCount(0))));
assert!(matches!(new(1, 9, 0, &[0; 9]), Err(Error::UnsupportedChannelCount(9))));
for (streams, coupled, table) in [(0, 0, &[0][..]), (1, 2, &[0]), (1, 1, &[2]), (200, 100, &[0])] {
assert!(
matches!(new(1, streams, coupled, table), Err(Error::InvalidMappingTable)),
"{streams} streams, {coupled} coupled, {table:?}"
);
}
}
}
Loading
Loading