Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,6 @@ lite-07; bitrate enforcement is not part of this release.
- [CMake qlog option for OBS](/quest/m2/obs-qlog.md) - an OBS build can opt into qlog trace capture through moq-ffi's `qlog` feature
- [AudioToolbox decode](/quest/m2/audio-decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers
- [AudioToolbox encode](/quest/m2/audio-encode-audiotoolbox.md) - macOS and iOS encode AAC-LC
- [CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) - a released mp4-atom carries the `dOps` channel mapping table, and fMP4 import and export carry it through the OpusHead
- [moqsrc surround Opus](/quest/m2/moqsrc-opus-surround.md) - moqsrc sets the Opus mapping caps, so opusdec plays 3 to 8 channels
- [GPU CI](/quest/m2/gpu-ci.md) - NVIDIA tests gate the merge queue on a self-hosted GPU runner VM, and `just rs nvidia` runs them locally instead of skipping
- [Vendor-free CARLA bridge](/quest/m2/carla-bridge.md) - the CARLA demo's GPU bridge lets moq-video pick the encoder, and its stand-in test runs on AMD as well as NVIDIA
Expand Down
57 changes: 0 additions & 57 deletions quest/m2/cmaf-opus-surround.md

This file was deleted.

4 changes: 0 additions & 4 deletions quest/m2/moqsrc-opus-surround.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,3 @@ channel count from 3 to 8 in the caps unit test, since every one maps
through the same family-1 table.

Public API: none. Wire: none.

## Related

- [CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) - the same mapping table through fMP4 `dOps`, independent of this caps path
15 changes: 6 additions & 9 deletions quest/m2/mux-vr-ambisonic-fmp4.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [S] fMP4 import fills the ambisonic layout
# [XS] fMP4 import fills the ambisonic layout

## Goal

Expand All @@ -12,13 +12,10 @@ catalog cannot carry, and the OpusHead's `AudioConfig` sets the layout).
Split from the VR fields quest on 2026-10-22 (maintainer), which landed the
Matroska and raw Opus side: only this part needs the `dOps` channel mapping
table.
Read the mapping from the `dOps` that
[CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) carries through the
OpusHead, and reuse that family 2 validation. Test with an fMP4
first-order ambisonic fixture and a refused non-diegetic stereo one.
The fMP4 importer carries the `dOps` mapping table into the OpusHead since
CMAF surround Opus (`fmp4::opus_mapping`), so a family 2 `dOps` should
already import as `ambisonic` through that family 2 validation. What remains
is proving it: test with an fMP4 first-order ambisonic fixture and a refused
non-diegetic stereo one.

Public API: none. Wire: none.

## Required

- [CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) - the `dOps` mapping table an ambisonic fMP4 track needs
3 changes: 2 additions & 1 deletion rs/moq-mux/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- MP4 and Matroska import fill a video rendition's `projection` and `stereo` from Spherical Video V2's `sv3d`/`st3d` boxes and Matroska's `Projection`/`StereoMode`, and Opus channel mapping family 2 (raw Opus, Matroska, and other OpusHead sources) imports with `layout: "ambisonic"`. What the catalog cannot describe is refused with `fmp4::Error::UnsupportedSpatial`, `mkv::Error::UnsupportedSpatial`, or `opus::Error::UnsupportedChannelCount`: a cubemap or mesh projection, a nonzero pose, uncentred or vertically cropped equirectangular bounds, a stereo packing other than mono, left-right, or top-bottom with the left eye first, and family 2 at zeroth order or with RFC 8486's non-diegetic stereo pair.
- fMP4 carries surround Opus: a `dOps` channel mapping table (family 1, 2, or 255) imports into the OpusHead description, and an exported `dOps` writes the description's table back instead of refusing it with `opus::Error::UnsupportedMappingFamily`. Family 3, whose OpusHead carries a demixing matrix rather than a table, is still refused both ways.

### Changed

- [**breaking**] Declare per-track archive durations from GOP hints or the first completed record.
- [**breaking**] Update the re-exported `mp4_atom` to 0.17, whose `Trun.entries` is now `Trun.samples`: a run without per-sample fields decodes as a count, so a moof of field-less runs no longer materializes an entry per sample before import refuses it. Read entries with `trun.samples.iter()` and build them with `samples: entries.into()`. `TrunEntry::cts` widens to `i64` (a version 0 offset is unsigned), `EsDescriptor` gains its optional fields, and `Dref.urls` becomes typed `entries`. `Dops` gains `channel_mapping: Option<OpusChannelMapping>` (`None` for family 0); the fMP4 importer still refuses a nonzero family.
- [**breaking**] Update the re-exported `mp4_atom` to 0.17, whose `Trun.entries` is now `Trun.samples`: a run without per-sample fields decodes as a count, so a moof of field-less runs no longer materializes an entry per sample before import refuses it. Read entries with `trun.samples.iter()` and build them with `samples: entries.into()`. `TrunEntry::cts` widens to `i64` (a version 0 offset is unsigned), `EsDescriptor` gains its optional fields, and `Dref.urls` becomes typed `entries`. `Dops` gains `channel_mapping: Option<OpusChannelMapping>` (`None` for family 0).

### Fixed

Expand Down
2 changes: 1 addition & 1 deletion rs/moq-mux/src/codec/opus/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ pub enum Error {

/// A channel mapping family the operation cannot carry: family 0 given to
/// [`Mapping::new`], which has no table, or a container with no room for
/// the table.
/// the mapping, such as family 3's demixing matrix in an fMP4 `dOps`.
#[error("unsupported channel mapping family {0}")]
UnsupportedMappingFamily(u8),
}
Expand Down
12 changes: 6 additions & 6 deletions rs/moq-mux/src/container/fmp4/export_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -900,22 +900,22 @@ fn synthesize_opus_trak_refuses_a_contradicting_head() {
assert!(matches!(err, super::Error::OpusChannelCount { catalog: 2, head: 1 }));
}

/// dOps has nowhere to put a channel mapping table, so a family 1 head is refused rather
/// than written as family 0 without it.
/// A family 3 head carries a demixing matrix that dOps has no room for, so it is refused
/// rather than written as a channel table.
#[test]
fn synthesize_opus_trak_refuses_mapping_family() {
fn synthesize_opus_trak_refuses_mapping_family_three() {
use hang::catalog::{AudioCodec, AudioConfig};

let mut head = crate::codec::opus::Config::new(48_000, 2).encode().unwrap().to_vec();
head[18] = 1; // channel mapping family
head.extend_from_slice(&[1, 1, 0, 1]); // one coupled stream feeding both channels
head[18] = 3; // channel mapping family
head.extend_from_slice(&[1, 1, 0, 1]); // read as a table: one coupled stream feeding both channels
let mut config = AudioConfig::new(AudioCodec::Opus, 48_000, 2);
config.description = Some(head.into());

let err = super::synthesize_audio_trak(1, 48_000, &config).unwrap_err();
assert!(matches!(
err,
super::Error::Opus(crate::codec::opus::Error::UnsupportedMappingFamily(1))
super::Error::Opus(crate::codec::opus::Error::UnsupportedMappingFamily(3))
));
}

Expand Down
9 changes: 2 additions & 7 deletions rs/moq-mux/src/container/fmp4/import.rs
Original file line number Diff line number Diff line change
Expand Up @@ -841,18 +841,13 @@ impl<E: crate::catalog::hang::CatalogExt> Import<E> {
}
mp4_atom::Codec::Opus(opus) => {
// dOps carries the OpusHead fields in ISOBMFF byte order; republish them as the
// OpusHead description so decoders apply its pre-skip and gain. The head built here
// is family 0, so refuse a mapping table rather than drop it.
// OpusHead description so decoders apply its pre-skip, gain, and channel mapping.
let dops = &opus.dops;
if let Some(mapping) = &dops.channel_mapping {
return Err(
Error::Opus(crate::codec::opus::Error::UnsupportedMappingFamily(mapping.family)).into(),
);
}
let mut head =
crate::codec::opus::Config::new(dops.input_sample_rate, dops.output_channel_count as u32)
.with_pre_skip(dops.pre_skip);
head.output_gain = dops.output_gain;
head.mapping = dops.channel_mapping.as_ref().map(super::opus_mapping).transpose()?;

// The shared conversion sets the decoder's 48 kHz output rate and Opus warmup,
// whatever the sample entry or the informational input rate claims.
Expand Down
108 changes: 87 additions & 21 deletions rs/moq-mux/src/container/fmp4/import_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1875,34 +1875,100 @@ fn opus_catalog_uses_the_decode_rate() {
assert_eq!(a.warmup, Some(std::time::Duration::from_millis(80)));
}

/// A dOps with a channel mapping table is refused rather than imported without it.
#[test]
fn opus_dops_mapping_family_is_refused() {
let (_, data) = opus_init(48_000);
let (ftyp, mut moov) = decode_init(&data);
/// `init` with its Opus sample entry declaring `channels` channels and `mapping`.
fn opus_init_with_mapping(init: &[u8], channels: u8, mapping: mp4_atom::OpusChannelMapping) -> Vec<u8> {
let (ftyp, mut moov) = decode_init(init);
let mp4_atom::Codec::Opus(opus) = &mut moov.trak[0].mdia.minf.stbl.stsd.codecs[0] else {
panic!("expected Opus sample entry");
};
opus.dops.channel_mapping = Some(mp4_atom::OpusChannelMapping {
family: 1,
stream_count: 1,
coupled_count: 1,
channel_mapping: vec![0, 1],
});
opus.audio.channel_count = channels.into();
opus.dops.output_channel_count = channels;
opus.dops.channel_mapping = Some(mapping);
let mut data = Vec::new();
ftyp.encode(&mut data).unwrap();
moov.encode(&mut data).unwrap();
data
}

let mut broadcast = moq_net::broadcast::Info::new().produce();
let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap();
let mut fmp4 = crate::container::fmp4::Import::new(broadcast, catalog.reserve());
assert!(matches!(
fmp4.decode(&bytes::BytesMut::from(data.as_slice())),
Err(crate::Error::Cmaf(super::Error::Opus(
crate::codec::opus::Error::UnsupportedMappingFamily(1)
)))
));
assert!(catalog.snapshot().audio.renditions.is_empty());
/// A family 1 5.1 dOps imports with its mapping table in the OpusHead description, and
/// re-exporting that description writes the same dOps, table included.
#[test]
fn opus_dops_surround_round_trips() {
let (_, data) = opus_init(48_000);
let mapping = mp4_atom::OpusChannelMapping {
family: 1,
stream_count: 4,
coupled_count: 2,
channel_mapping: vec![0, 4, 1, 2, 3, 5],
};
let data = opus_init_with_mapping(&data, 6, mapping.clone());
let (_, moov) = decode_init(&data);
let mp4_atom::Codec::Opus(source) = &moov.trak[0].mdia.minf.stbl.stsd.codecs[0] else {
panic!("expected Opus sample entry");
};

let catalog = run_fmp4(&data);
let a = catalog.audio.renditions.values().next().expect("opus rendition");
assert_eq!(a.channel_count, 6);

let desc = a.description.as_ref().expect("opus description");
let head = crate::codec::opus::Config::parse(&mut desc.as_ref()).unwrap();
assert_eq!(head.channel_count, 6);
let parsed = head.mapping.expect("mapping table");
assert_eq!(parsed.family(), 1);
assert_eq!((parsed.streams(), parsed.coupled()), (4, 2));
assert_eq!(parsed.table(), &[0, 4, 1, 2, 3, 5]);

let trak = super::synthesize_audio_trak(1, 48_000, a).expect("synthesize Opus trak");
match &trak.mdia.minf.stbl.stsd.codecs[0] {
mp4_atom::Codec::Opus(opus) => {
assert_eq!(opus.dops, source.dops);
assert_eq!(opus.dops.channel_mapping, Some(mapping));
assert_eq!(opus.audio.channel_count, 6);
}
other => panic!("expected Opus sample entry, got {other:?}"),
}
}

/// A dOps mapping the OpusHead cannot describe is refused rather than imported: family 3,
/// whose OpusHead carries a demixing matrix, and a family 1 table past its 8 channels.
#[test]
fn opus_dops_mapping_the_head_cannot_describe_is_refused() {
let (_, init) = opus_init(48_000);
let cases = [
(
mp4_atom::OpusChannelMapping {
family: 3,
stream_count: 2,
coupled_count: 2,
channel_mapping: vec![0, 1, 2, 3],
},
crate::codec::opus::Error::UnsupportedMappingFamily(3),
),
(
mp4_atom::OpusChannelMapping {
family: 1,
stream_count: 9,
coupled_count: 0,
channel_mapping: (0..9).collect(),
},
crate::codec::opus::Error::UnsupportedChannelCount(9),
),
];
for (mapping, expected) in cases {
let channels = mapping.channel_mapping.len() as u8;
let data = opus_init_with_mapping(&init, channels, mapping);

let mut broadcast = moq_net::broadcast::Info::new().produce();
let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap();
let mut fmp4 = crate::container::fmp4::Import::new(broadcast, catalog.reserve());
let err = fmp4.decode(&bytes::BytesMut::from(data.as_slice())).unwrap_err();
assert!(
matches!(&err, crate::Error::Cmaf(super::Error::Opus(opus)) if opus.to_string() == expected.to_string()),
"{err:?}"
);
assert!(catalog.snapshot().audio.renditions.is_empty());
}
}

// ---- Segment-driven grouping ----
Expand Down
36 changes: 31 additions & 5 deletions rs/moq-mux/src/container/fmp4/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -993,6 +993,36 @@ pub(crate) fn synthesize_video_trak(
))
}

/// The OpusHead mapping for a `dOps` channel mapping table, checked by [`crate::codec::opus::Mapping::new`].
///
/// Family 3 is refused: its OpusHead carries a demixing matrix (RFC 8486 §3.2) where
/// `dOps` has a table, so neither side can describe the other.
pub(crate) fn opus_mapping(mapping: &mp4_atom::OpusChannelMapping) -> Result<crate::codec::opus::Mapping> {
if mapping.family == 3 {
return Err(crate::codec::opus::Error::UnsupportedMappingFamily(3).into());
}
Ok(crate::codec::opus::Mapping::new(crate::codec::opus::mapping::Config {
family: mapping.family,
streams: mapping.stream_count,
coupled: mapping.coupled_count,
table: &mapping.channel_mapping,
})?)
}

/// The `dOps` channel mapping table for an OpusHead mapping, refusing family 3 as
/// [`opus_mapping`] does.
fn dops_mapping(mapping: &crate::codec::opus::Mapping) -> Result<mp4_atom::OpusChannelMapping> {
if mapping.family() == 3 {
return Err(crate::codec::opus::Error::UnsupportedMappingFamily(3).into());
}
Ok(mp4_atom::OpusChannelMapping {
family: mapping.family(),
stream_count: mapping.streams(),
coupled_count: mapping.coupled(),
channel_mapping: mapping.table().to_vec(),
})
}

/// Synthesize a CMAF `Trak` for an audio rendition that has no init segment.
pub(crate) fn synthesize_audio_trak(track_id: u32, timescale: u64, config: &AudioConfig) -> Result<mp4_atom::Trak> {
use mp4_atom::Decode;
Expand All @@ -1009,10 +1039,6 @@ pub(crate) fn synthesize_audio_trak(track_id: u32, timescale: u64, config: &Audi
let head = match &config.description {
Some(description) => {
let head = crate::codec::opus::Config::parse(&mut description.as_ref())?;
// dOps shares OpusHead's family 0 layout; a mapping table would need writing too.
if let Some(mapping) = head.mapping {
return Err(crate::codec::opus::Error::UnsupportedMappingFamily(mapping.family()).into());
}
if head.channel_count != config.channel_count {
return Err(Error::OpusChannelCount {
catalog: config.channel_count,
Expand All @@ -1030,7 +1056,7 @@ pub(crate) fn synthesize_audio_trak(track_id: u32, timescale: u64, config: &Audi
pre_skip: head.pre_skip,
input_sample_rate: head.sample_rate,
output_gain: head.output_gain,
channel_mapping: None,
channel_mapping: head.mapping.as_ref().map(dops_mapping).transpose()?,
},
btrt: None,
})
Expand Down