diff --git a/quest/m2/README.md b/quest/m2/README.md index fddffded403..8ae0ce6a480 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -71,7 +71,6 @@ lite-07; bitrate enforcement is not part of this release. - [CMake qlog option for OBS](/quest/m2/obs-qlog.md) - an OBS build can opt into qlog trace capture through moq-ffi's `qlog` feature - [AudioToolbox decode](/quest/m2/audio-decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [AudioToolbox encode](/quest/m2/audio-encode-audiotoolbox.md) - macOS and iOS encode AAC-LC -- [CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) - a released mp4-atom carries the `dOps` channel mapping table, and fMP4 import and export carry it through the OpusHead - [moqsrc surround Opus](/quest/m2/moqsrc-opus-surround.md) - moqsrc sets the Opus mapping caps, so opusdec plays 3 to 8 channels - [GPU CI](/quest/m2/gpu-ci.md) - NVIDIA tests gate the merge queue on a self-hosted GPU runner VM, and `just rs nvidia` runs them locally instead of skipping - [Vendor-free CARLA bridge](/quest/m2/carla-bridge.md) - the CARLA demo's GPU bridge lets moq-video pick the encoder, and its stand-in test runs on AMD as well as NVIDIA diff --git a/quest/m2/cmaf-opus-surround.md b/quest/m2/cmaf-opus-surround.md deleted file mode 100644 index 80303afef44..00000000000 --- a/quest/m2/cmaf-opus-surround.md +++ /dev/null @@ -1,57 +0,0 @@ -# [M] Carry surround Opus through CMAF - -## Goal - -An fMP4 Opus track whose `dOps` declares channel mapping family 1 imports with -its mapping table in the OpusHead description, and a family 1 description -exports to a `dOps` that keeps the table. Today both directions refuse it. - -This needs a released `mp4-atom` whose `Dops` decodes and encodes any channel -mapping family as `channel_mapping: Option` (`None` is -family 0), the shape drafted in kixelated/mp4-atom#246. That is a breaking -`mp4-atom` 0.17. - -## Plan - -Decided 2026-10-17: the `mp4-atom` change is not worth landing on its own. -Alone it only swaps `mp4-atom`'s generic decode error for -`UnsupportedMappingFamily`; the payoff is surround Opus. So the former -`mp4-atom-dops-mapping` quest (draft moq-dev/moq#5819) folded in here. - -Decided 2026-10-28 (maintainer): kixelated/mp4-atom#246 ships in 0.17.0 -beside kixelated/mp4-atom#254, so downstream takes one breaking bump. -0.17.0 is released (2026-10-30), and moq-mux takes it for the counted -trun (moq-dev/moq#7043, carried by moq-dev/moq#7308), still refusing a -nonzero family explicitly, since `mp4-atom` no longer does. -This quest then only does the import and export. -A git patch to the branch cannot stand in for the release, since `cargo deny` -refuses unknown git sources. - -Settled in #246 and #5819: - -- `Dops` decode and encode both check the table against the output channel - count and the RFC 7845 §5.1.1 stream rules, failing with - `Error::InvalidCombination`. Family 0 encodes byte-identically. -- `family` stays a raw `u8`; family-specific limits (family 1 allows at most 8 - channels) are the consumer's, which `opus::Mapping` already enforces. -- `mp4-atom` does not check that family 0 declares one or two channels; moq's - Opus config checks its own head, so that is left alone. - -Open for the implementing PR (maintainer's call, in #246): - -1. `Dops` shape. Recommended: `Option` beside - `output_channel_count`. Alternatives: a flat `channel_mapping_family: u8` - plus an optional table, which mirrors the spec but allows family 0 with a - table; or an enum replacing `output_channel_count`, which drops the - redundant count but breaks every caller harder. -2. Table field name. Recommended: `channel_mapping`, mirroring the spec. - Alternative: `table`, which reads better at call sites. - -In moq-mux, build the head from the `dOps` family and table with -`opus::Mapping` on import, where the importer today relies on `mp4-atom` -refusing a nonzero family. On export, write the table from the parsed head in -place of the `UnsupportedMappingFamily` refusal in `synthesize_audio_trak`. -Keep refusing families the head cannot describe. - -Regression: a family 1 5.1 `dOps` built with `mp4-atom` round-trips through -import and export with its table intact. diff --git a/quest/m2/moqsrc-opus-surround.md b/quest/m2/moqsrc-opus-surround.md index 197e1e7d127..c422cc49f59 100644 --- a/quest/m2/moqsrc-opus-surround.md +++ b/quest/m2/moqsrc-opus-surround.md @@ -27,7 +27,3 @@ channel count from 3 to 8 in the caps unit test, since every one maps through the same family-1 table. Public API: none. Wire: none. - -## Related - -- [CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) - the same mapping table through fMP4 `dOps`, independent of this caps path diff --git a/quest/m2/mux-vr-ambisonic-fmp4.md b/quest/m2/mux-vr-ambisonic-fmp4.md index 8fdb89c6eaa..4ea1c891536 100644 --- a/quest/m2/mux-vr-ambisonic-fmp4.md +++ b/quest/m2/mux-vr-ambisonic-fmp4.md @@ -1,4 +1,4 @@ -# [S] fMP4 import fills the ambisonic layout +# [XS] fMP4 import fills the ambisonic layout ## Goal @@ -12,13 +12,10 @@ catalog cannot carry, and the OpusHead's `AudioConfig` sets the layout). Split from the VR fields quest on 2026-10-22 (maintainer), which landed the Matroska and raw Opus side: only this part needs the `dOps` channel mapping table. -Read the mapping from the `dOps` that -[CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) carries through the -OpusHead, and reuse that family 2 validation. Test with an fMP4 -first-order ambisonic fixture and a refused non-diegetic stereo one. +The fMP4 importer carries the `dOps` mapping table into the OpusHead since +CMAF surround Opus (`fmp4::opus_mapping`), so a family 2 `dOps` should +already import as `ambisonic` through that family 2 validation. What remains +is proving it: test with an fMP4 first-order ambisonic fixture and a refused +non-diegetic stereo one. Public API: none. Wire: none. - -## Required - -- [CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) - the `dOps` mapping table an ambisonic fMP4 track needs diff --git a/rs/moq-mux/CHANGELOG.md b/rs/moq-mux/CHANGELOG.md index 304f9752987..a850cefe1a5 100644 --- a/rs/moq-mux/CHANGELOG.md +++ b/rs/moq-mux/CHANGELOG.md @@ -10,11 +10,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - MP4 and Matroska import fill a video rendition's `projection` and `stereo` from Spherical Video V2's `sv3d`/`st3d` boxes and Matroska's `Projection`/`StereoMode`, and Opus channel mapping family 2 (raw Opus, Matroska, and other OpusHead sources) imports with `layout: "ambisonic"`. What the catalog cannot describe is refused with `fmp4::Error::UnsupportedSpatial`, `mkv::Error::UnsupportedSpatial`, or `opus::Error::UnsupportedChannelCount`: a cubemap or mesh projection, a nonzero pose, uncentred or vertically cropped equirectangular bounds, a stereo packing other than mono, left-right, or top-bottom with the left eye first, and family 2 at zeroth order or with RFC 8486's non-diegetic stereo pair. +- fMP4 carries surround Opus: a `dOps` channel mapping table (family 1, 2, or 255) imports into the OpusHead description, and an exported `dOps` writes the description's table back instead of refusing it with `opus::Error::UnsupportedMappingFamily`. Family 3, whose OpusHead carries a demixing matrix rather than a table, is still refused both ways. ### Changed - [**breaking**] Declare per-track archive durations from GOP hints or the first completed record. -- [**breaking**] Update the re-exported `mp4_atom` to 0.17, whose `Trun.entries` is now `Trun.samples`: a run without per-sample fields decodes as a count, so a moof of field-less runs no longer materializes an entry per sample before import refuses it. Read entries with `trun.samples.iter()` and build them with `samples: entries.into()`. `TrunEntry::cts` widens to `i64` (a version 0 offset is unsigned), `EsDescriptor` gains its optional fields, and `Dref.urls` becomes typed `entries`. `Dops` gains `channel_mapping: Option` (`None` for family 0); the fMP4 importer still refuses a nonzero family. +- [**breaking**] Update the re-exported `mp4_atom` to 0.17, whose `Trun.entries` is now `Trun.samples`: a run without per-sample fields decodes as a count, so a moof of field-less runs no longer materializes an entry per sample before import refuses it. Read entries with `trun.samples.iter()` and build them with `samples: entries.into()`. `TrunEntry::cts` widens to `i64` (a version 0 offset is unsigned), `EsDescriptor` gains its optional fields, and `Dref.urls` becomes typed `entries`. `Dops` gains `channel_mapping: Option` (`None` for family 0). ### Fixed diff --git a/rs/moq-mux/src/codec/opus/mod.rs b/rs/moq-mux/src/codec/opus/mod.rs index bc57f408e80..fccf59bebbc 100644 --- a/rs/moq-mux/src/codec/opus/mod.rs +++ b/rs/moq-mux/src/codec/opus/mod.rs @@ -50,7 +50,7 @@ pub enum Error { /// A channel mapping family the operation cannot carry: family 0 given to /// [`Mapping::new`], which has no table, or a container with no room for - /// the table. + /// the mapping, such as family 3's demixing matrix in an fMP4 `dOps`. #[error("unsupported channel mapping family {0}")] UnsupportedMappingFamily(u8), } diff --git a/rs/moq-mux/src/container/fmp4/export_test.rs b/rs/moq-mux/src/container/fmp4/export_test.rs index 3506eae4296..0ed3c34d653 100644 --- a/rs/moq-mux/src/container/fmp4/export_test.rs +++ b/rs/moq-mux/src/container/fmp4/export_test.rs @@ -900,22 +900,22 @@ fn synthesize_opus_trak_refuses_a_contradicting_head() { assert!(matches!(err, super::Error::OpusChannelCount { catalog: 2, head: 1 })); } -/// dOps has nowhere to put a channel mapping table, so a family 1 head is refused rather -/// than written as family 0 without it. +/// A family 3 head carries a demixing matrix that dOps has no room for, so it is refused +/// rather than written as a channel table. #[test] -fn synthesize_opus_trak_refuses_mapping_family() { +fn synthesize_opus_trak_refuses_mapping_family_three() { use hang::catalog::{AudioCodec, AudioConfig}; let mut head = crate::codec::opus::Config::new(48_000, 2).encode().unwrap().to_vec(); - head[18] = 1; // channel mapping family - head.extend_from_slice(&[1, 1, 0, 1]); // one coupled stream feeding both channels + head[18] = 3; // channel mapping family + head.extend_from_slice(&[1, 1, 0, 1]); // read as a table: one coupled stream feeding both channels let mut config = AudioConfig::new(AudioCodec::Opus, 48_000, 2); config.description = Some(head.into()); let err = super::synthesize_audio_trak(1, 48_000, &config).unwrap_err(); assert!(matches!( err, - super::Error::Opus(crate::codec::opus::Error::UnsupportedMappingFamily(1)) + super::Error::Opus(crate::codec::opus::Error::UnsupportedMappingFamily(3)) )); } diff --git a/rs/moq-mux/src/container/fmp4/import.rs b/rs/moq-mux/src/container/fmp4/import.rs index 7a4011ecbbd..211bc0705ca 100644 --- a/rs/moq-mux/src/container/fmp4/import.rs +++ b/rs/moq-mux/src/container/fmp4/import.rs @@ -841,18 +841,13 @@ impl Import { } mp4_atom::Codec::Opus(opus) => { // dOps carries the OpusHead fields in ISOBMFF byte order; republish them as the - // OpusHead description so decoders apply its pre-skip and gain. The head built here - // is family 0, so refuse a mapping table rather than drop it. + // OpusHead description so decoders apply its pre-skip, gain, and channel mapping. let dops = &opus.dops; - if let Some(mapping) = &dops.channel_mapping { - return Err( - Error::Opus(crate::codec::opus::Error::UnsupportedMappingFamily(mapping.family)).into(), - ); - } let mut head = crate::codec::opus::Config::new(dops.input_sample_rate, dops.output_channel_count as u32) .with_pre_skip(dops.pre_skip); head.output_gain = dops.output_gain; + head.mapping = dops.channel_mapping.as_ref().map(super::opus_mapping).transpose()?; // The shared conversion sets the decoder's 48 kHz output rate and Opus warmup, // whatever the sample entry or the informational input rate claims. diff --git a/rs/moq-mux/src/container/fmp4/import_test.rs b/rs/moq-mux/src/container/fmp4/import_test.rs index d722276ab0e..55222d4191d 100644 --- a/rs/moq-mux/src/container/fmp4/import_test.rs +++ b/rs/moq-mux/src/container/fmp4/import_test.rs @@ -1875,34 +1875,100 @@ fn opus_catalog_uses_the_decode_rate() { assert_eq!(a.warmup, Some(std::time::Duration::from_millis(80))); } -/// A dOps with a channel mapping table is refused rather than imported without it. -#[test] -fn opus_dops_mapping_family_is_refused() { - let (_, data) = opus_init(48_000); - let (ftyp, mut moov) = decode_init(&data); +/// `init` with its Opus sample entry declaring `channels` channels and `mapping`. +fn opus_init_with_mapping(init: &[u8], channels: u8, mapping: mp4_atom::OpusChannelMapping) -> Vec { + let (ftyp, mut moov) = decode_init(init); let mp4_atom::Codec::Opus(opus) = &mut moov.trak[0].mdia.minf.stbl.stsd.codecs[0] else { panic!("expected Opus sample entry"); }; - opus.dops.channel_mapping = Some(mp4_atom::OpusChannelMapping { - family: 1, - stream_count: 1, - coupled_count: 1, - channel_mapping: vec![0, 1], - }); + opus.audio.channel_count = channels.into(); + opus.dops.output_channel_count = channels; + opus.dops.channel_mapping = Some(mapping); let mut data = Vec::new(); ftyp.encode(&mut data).unwrap(); moov.encode(&mut data).unwrap(); + data +} - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); - let mut fmp4 = crate::container::fmp4::Import::new(broadcast, catalog.reserve()); - assert!(matches!( - fmp4.decode(&bytes::BytesMut::from(data.as_slice())), - Err(crate::Error::Cmaf(super::Error::Opus( - crate::codec::opus::Error::UnsupportedMappingFamily(1) - ))) - )); - assert!(catalog.snapshot().audio.renditions.is_empty()); +/// A family 1 5.1 dOps imports with its mapping table in the OpusHead description, and +/// re-exporting that description writes the same dOps, table included. +#[test] +fn opus_dops_surround_round_trips() { + let (_, data) = opus_init(48_000); + let mapping = mp4_atom::OpusChannelMapping { + family: 1, + stream_count: 4, + coupled_count: 2, + channel_mapping: vec![0, 4, 1, 2, 3, 5], + }; + let data = opus_init_with_mapping(&data, 6, mapping.clone()); + let (_, moov) = decode_init(&data); + let mp4_atom::Codec::Opus(source) = &moov.trak[0].mdia.minf.stbl.stsd.codecs[0] else { + panic!("expected Opus sample entry"); + }; + + let catalog = run_fmp4(&data); + let a = catalog.audio.renditions.values().next().expect("opus rendition"); + assert_eq!(a.channel_count, 6); + + let desc = a.description.as_ref().expect("opus description"); + let head = crate::codec::opus::Config::parse(&mut desc.as_ref()).unwrap(); + assert_eq!(head.channel_count, 6); + let parsed = head.mapping.expect("mapping table"); + assert_eq!(parsed.family(), 1); + assert_eq!((parsed.streams(), parsed.coupled()), (4, 2)); + assert_eq!(parsed.table(), &[0, 4, 1, 2, 3, 5]); + + let trak = super::synthesize_audio_trak(1, 48_000, a).expect("synthesize Opus trak"); + match &trak.mdia.minf.stbl.stsd.codecs[0] { + mp4_atom::Codec::Opus(opus) => { + assert_eq!(opus.dops, source.dops); + assert_eq!(opus.dops.channel_mapping, Some(mapping)); + assert_eq!(opus.audio.channel_count, 6); + } + other => panic!("expected Opus sample entry, got {other:?}"), + } +} + +/// A dOps mapping the OpusHead cannot describe is refused rather than imported: family 3, +/// whose OpusHead carries a demixing matrix, and a family 1 table past its 8 channels. +#[test] +fn opus_dops_mapping_the_head_cannot_describe_is_refused() { + let (_, init) = opus_init(48_000); + let cases = [ + ( + mp4_atom::OpusChannelMapping { + family: 3, + stream_count: 2, + coupled_count: 2, + channel_mapping: vec![0, 1, 2, 3], + }, + crate::codec::opus::Error::UnsupportedMappingFamily(3), + ), + ( + mp4_atom::OpusChannelMapping { + family: 1, + stream_count: 9, + coupled_count: 0, + channel_mapping: (0..9).collect(), + }, + crate::codec::opus::Error::UnsupportedChannelCount(9), + ), + ]; + for (mapping, expected) in cases { + let channels = mapping.channel_mapping.len() as u8; + let data = opus_init_with_mapping(&init, channels, mapping); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut fmp4 = crate::container::fmp4::Import::new(broadcast, catalog.reserve()); + let err = fmp4.decode(&bytes::BytesMut::from(data.as_slice())).unwrap_err(); + assert!( + matches!(&err, crate::Error::Cmaf(super::Error::Opus(opus)) if opus.to_string() == expected.to_string()), + "{err:?}" + ); + assert!(catalog.snapshot().audio.renditions.is_empty()); + } } // ---- Segment-driven grouping ---- diff --git a/rs/moq-mux/src/container/fmp4/mod.rs b/rs/moq-mux/src/container/fmp4/mod.rs index ef5d8730fb2..9cf0fc13a34 100644 --- a/rs/moq-mux/src/container/fmp4/mod.rs +++ b/rs/moq-mux/src/container/fmp4/mod.rs @@ -993,6 +993,36 @@ pub(crate) fn synthesize_video_trak( )) } +/// The OpusHead mapping for a `dOps` channel mapping table, checked by [`crate::codec::opus::Mapping::new`]. +/// +/// Family 3 is refused: its OpusHead carries a demixing matrix (RFC 8486 §3.2) where +/// `dOps` has a table, so neither side can describe the other. +pub(crate) fn opus_mapping(mapping: &mp4_atom::OpusChannelMapping) -> Result { + if mapping.family == 3 { + return Err(crate::codec::opus::Error::UnsupportedMappingFamily(3).into()); + } + Ok(crate::codec::opus::Mapping::new(crate::codec::opus::mapping::Config { + family: mapping.family, + streams: mapping.stream_count, + coupled: mapping.coupled_count, + table: &mapping.channel_mapping, + })?) +} + +/// The `dOps` channel mapping table for an OpusHead mapping, refusing family 3 as +/// [`opus_mapping`] does. +fn dops_mapping(mapping: &crate::codec::opus::Mapping) -> Result { + if mapping.family() == 3 { + return Err(crate::codec::opus::Error::UnsupportedMappingFamily(3).into()); + } + Ok(mp4_atom::OpusChannelMapping { + family: mapping.family(), + stream_count: mapping.streams(), + coupled_count: mapping.coupled(), + channel_mapping: mapping.table().to_vec(), + }) +} + /// Synthesize a CMAF `Trak` for an audio rendition that has no init segment. pub(crate) fn synthesize_audio_trak(track_id: u32, timescale: u64, config: &AudioConfig) -> Result { use mp4_atom::Decode; @@ -1009,10 +1039,6 @@ pub(crate) fn synthesize_audio_trak(track_id: u32, timescale: u64, config: &Audi let head = match &config.description { Some(description) => { let head = crate::codec::opus::Config::parse(&mut description.as_ref())?; - // dOps shares OpusHead's family 0 layout; a mapping table would need writing too. - if let Some(mapping) = head.mapping { - return Err(crate::codec::opus::Error::UnsupportedMappingFamily(mapping.family()).into()); - } if head.channel_count != config.channel_count { return Err(Error::OpusChannelCount { catalog: config.channel_count, @@ -1030,7 +1056,7 @@ pub(crate) fn synthesize_audio_trak(track_id: u32, timescale: u64, config: &Audi pre_skip: head.pre_skip, input_sample_rate: head.sample_rate, output_gain: head.output_gain, - channel_mapping: None, + channel_mapping: head.mapping.as_ref().map(dops_mapping).transpose()?, }, btrt: None, })