From 576285af1f9fc8ef5817deee73d5c59c3c5f4f82 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 24 Sep 2026 18:13:32 -0700 Subject: [PATCH 01/20] quest: open the audio-codecs line Co-Authored-By: Claude Opus 5.5 From c31063b052ebfebb5728d8312436c322bac6b8dd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 00:17:26 -0700 Subject: [PATCH 02/20] fix(mux): parse the AAC program config element instead of guessing stereo (#4093) Co-authored-by: Claude Opus 5.5 --- quest/m1/audio-codecs/README.md | 6 +- quest/m1/audio-codecs/aac-pce.md | 34 -- quest/m1/audio-codecs/he-aac-refusal.md | 1 - quest/m1/audio-codecs/layout.md | 5 +- quest/m1/audio-codecs/ts-export-pce.md | 17 + rs/moq-mux/src/codec/aac/mod.rs | 396 ++++++++++++++++-- rs/moq-mux/src/container/ts/adts.rs | 20 +- rs/moq-mux/src/container/ts/import.rs | 20 +- rs/moq-mux/src/container/ts/import_test.rs | 26 ++ .../src/container/ts/test_data/aac_quad.ts | Bin 0 -> 2632 bytes 10 files changed, 436 insertions(+), 89 deletions(-) delete mode 100644 quest/m1/audio-codecs/aac-pce.md create mode 100644 quest/m1/audio-codecs/ts-export-pce.md create mode 100644 rs/moq-mux/src/container/ts/test_data/aac_quad.ts diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 372157036a7..c1366d831b5 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -37,14 +37,12 @@ mono/stereo. The core configuration and layout contracts land in main. These quests implement surround and backend dispatch on that contract; each platform then lands as -its own decode and encode quest so verification stays per host. The -HE-AAC refusal and the PCE parse are defects in what ships today and are -ready now. +its own decode and encode quest so verification stays per host. ## Quests - [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - implicit-SBR HE-AAC over TS is refused instead of half-decoded as the LC core -- [AAC PCE](/quest/m1/audio-codecs/aac-pce.md) - a channel_config of 0 parses the program config element instead of guessing stereo +- [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config - [Layout](/quest/m1/audio-codecs/layout.md) - the settled `Layout` carries up to 7.1 through decode, resample, playback, and the FFI - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers diff --git a/quest/m1/audio-codecs/aac-pce.md b/quest/m1/audio-codecs/aac-pce.md deleted file mode 100644 index db99ec93f95..00000000000 --- a/quest/m1/audio-codecs/aac-pce.md +++ /dev/null @@ -1,34 +0,0 @@ -# [S] moq-mux: an AAC channel_config of 0 parses the PCE or refuses the track - -## Goal - -The catalog never invents an AAC channel count. A stream whose -channelConfiguration is 0 gets its count from the program config element it -carries, and a reserved value refuses the track with a clear error. Today -both map to stereo with a warning. - -## Plan - -The ADTS and ASC importers in `rs/moq-mux/src/codec/aac` map -`channel_config == 0` and every value from 8 to 15 to stereo and warn. -Warn-then-continue is banned: supported or refused. - -- ASC: with `channel_config == 0` the GASpecificConfig carries a - `program_config_element`; parse its front, side, back, and LFE element - counts into the channel count (and, once [Layout](/quest/m1/audio-codecs/layout.md) - lands, into a layout). -- ADTS: channel_config 0 means the PCE is in the first raw data block. Parse - it there, once per track, the same way the HE-AAC sniff reads the first - block's fill elements. -- Give every ASC configuration from 8 to 15 an explicit disposition: implement - any supported channel mapping and return `Error::Unsupported` for every - remaining value, including reserved configurations. -- Regression: an ASC fixture with a PCE reports its real count; an ADTS - fixture with an in-band PCE does too; each value from 8 to 15 has its - supported count asserted or is refused; the - common configurations 1 to 7 are unchanged. - -## Related - -- [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - reads the same first block -- [Layout](/quest/m1/audio-codecs/layout.md) - what the parsed PCE eventually maps to diff --git a/quest/m1/audio-codecs/he-aac-refusal.md b/quest/m1/audio-codecs/he-aac-refusal.md index f7995cc0f71..c3edf29e218 100644 --- a/quest/m1/audio-codecs/he-aac-refusal.md +++ b/quest/m1/audio-codecs/he-aac-refusal.md @@ -31,4 +31,3 @@ does too. ## Related - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the sniff becomes the software backend's contract -- [AAC PCE](/quest/m1/audio-codecs/aac-pce.md) - the other place the catalog lies about an AAC stream diff --git a/quest/m1/audio-codecs/layout.md b/quest/m1/audio-codecs/layout.md index 2df431c95cd..e92dfbbe381 100644 --- a/quest/m1/audio-codecs/layout.md +++ b/quest/m1/audio-codecs/layout.md @@ -16,7 +16,10 @@ Keep the representation extensible. Each codec module maps its native order into it: AAC's `C L R Ls Rs LFE` and Opus's Vorbis `L C R Ls Rs LFE` both become `L R C LFE Ls Rs`. An unspecified discrete PCM layout remains valid for passthrough but is refused for spatial remixing; never invent speaker -positions from an arbitrary channel count. +positions from an arbitrary channel count. An AAC stream with +channelConfiguration 0 takes its layout from the program config element's +front, side, back, and LFE elements, which `moq_mux::codec::aac` already parses +for the channel count. - Reuse the settled PCM descriptors and codec/consumer settings. This quest adds supported layout behavior rather than replacing public field types. diff --git a/quest/m1/audio-codecs/ts-export-pce.md b/quest/m1/audio-codecs/ts-export-pce.md new file mode 100644 index 00000000000..31e9b76b251 --- /dev/null +++ b/quest/m1/audio-codecs/ts-export-pce.md @@ -0,0 +1,17 @@ +# [S] TS export carries an AAC track's program config element + +## Goal + +`moq-mux`'s MPEG-TS export writes an AAC track described by a program config +element (channelConfiguration 0) as ADTS with channel_config 0 and the PCE at +the start of the first raw data block, so a TS round trip keeps the layout. +Today the export derives the ADTS channel_config from the channel count, which +mislabels such a track. + +## Plan + +Take the PCE from the track's AudioSpecificConfig description with the parser +that reads it on import, and write it as ffmpeg does: once, leading the first +raw data block. Tracks with a nonzero channelConfiguration are unchanged. Test +a round trip of the quad fixture from the PCE import (`aac_quad.ts`): import, +export, and import again, asserting the same description and channel count. diff --git a/rs/moq-mux/src/codec/aac/mod.rs b/rs/moq-mux/src/codec/aac/mod.rs index e2477911fc8..6d7f7cb67a5 100644 --- a/rs/moq-mux/src/codec/aac/mod.rs +++ b/rs/moq-mux/src/codec/aac/mod.rs @@ -27,6 +27,26 @@ pub enum Error { #[error("unsupported sample rate index: {0}")] UnsupportedSampleRateIndex(u8), + + /// The channelConfiguration is a reserved value (8, 9, 10, or 15). + #[error("reserved channelConfiguration: {0}")] + ReservedChannelConfig(u8), + + /// A program config element is used with an object type that has no GASpecificConfig. + #[error("channelConfiguration 0 is unsupported for audioObjectType {0}")] + ProgramConfigUnsupported(u8), + + /// ADTS signals a program config element, but none leads the first raw data block. + #[error("channelConfiguration 0 without a program config element leading the first raw data block")] + ProgramConfigMissing, + + /// The program config element ends before its declared fields. + #[error("program config element truncated")] + ProgramConfigTruncated, + + /// The program config element declares zero channels. + #[error("program config element declares no channels")] + ProgramConfigEmpty, } pub type Result = std::result::Result; @@ -46,19 +66,16 @@ impl Config { /// (object_type == 31), and explicit sample rates (freq_index == 15). The /// fields are bit-packed and not byte-aligned, so a bit reader is required: /// with an explicit 24-bit rate the channelConfiguration lands mid-byte after - /// it. Any SBR/PS extension bits after the core fields are consumed. + /// it. A channelConfiguration of 0 takes the count from the program config element + /// that follows, and a reserved one is refused. Any SBR/PS extension bits after the + /// core fields are consumed. pub fn parse(buf: &mut T) -> Result { if buf.remaining() < 2 { return Err(Error::ConfigTooShort); } let mut reader = BitReader::new(buf); - - // audioObjectType: 5 bits, escaped to 6 more when it reads 31. - let mut object_type = reader.read(5, Error::ConfigTooShort)? as u8; - if object_type == 31 { - object_type = 32 + reader.read(6, Error::ExtendedConfigTooShort)? as u8; - } + let object_type = read_object_type(&mut reader)?; // samplingFrequencyIndex: 4 bits; index 15 means an explicit 24-bit rate follows. let freq_index = reader.read(4, Error::IncompleteConfig)? as u8; @@ -72,7 +89,36 @@ impl Config { // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; - let channel_count = channel_count_from_config(channel_config); + let channel_count = match channel_config { + 0 => { + // Explicit SBR and PS name their core object type after an extension rate; the + // GASpecificConfig carrying the program config element follows that core type. + let mut core = object_type; + if matches!(object_type, 5 | 29) { + if reader.read(4, Error::IncompleteConfig)? == 15 { + reader.read(24, Error::IncompleteConfig)?; + } + core = read_object_type(&mut reader)?; + if core == 22 { + // extensionChannelConfiguration, only for ER BSAC. + reader.read(4, Error::IncompleteConfig)?; + } + } + if !GENERAL_AUDIO.contains(&core) { + return Err(Error::ProgramConfigUnsupported(core)); + } + + // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit + // coreCoderDelay), and extensionFlag precede the element. + reader.read(1, Error::IncompleteConfig)?; + if reader.read(1, Error::IncompleteConfig)? == 1 { + reader.read(14, Error::IncompleteConfig)?; + } + reader.read(1, Error::IncompleteConfig)?; + program_config(&mut reader)? + } + _ => channel_count_from_config(channel_config)?, + }; // AudioSpecificConfig can carry variable-length extensions (SBR, PS, etc.). // We've extracted the essential fields; drain the rest so the buffer is advanced. @@ -132,6 +178,101 @@ impl Config { } } +/// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does. +/// +/// A `channel_config` of 0 means a program config element leads `block`, the stream's first raw +/// data block; it moves into the config, so the config describes the channels on its own. Any +/// other value leaves `block` unread. An element placed anywhere else is refused: reaching past +/// the channel data needs a full Huffman decode. +pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, block: &[u8]) -> Result { + let mut out = BitWriter::default(); + out.write(5, u32::from(profile & 0x1F)); + match SAMPLE_RATES.iter().position(|&rate| rate == sample_rate) { + Some(index) => out.write(4, index as u32), + None => { + out.write(4, 15); + out.write(24, sample_rate); + } + } + out.write(4, u32::from(channel_config)); + + if channel_config == 0 { + // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder, and extensionFlag, all clear. + out.write(3, 0); + + let mut block = block; + let mut reader = BitReader::new(&mut block); + if reader.read(3, Error::ProgramConfigMissing)? != ID_PCE { + return Err(Error::ProgramConfigMissing); + } + reader.record = Some(out); + program_config(&mut reader)?; + out = reader.record.take().expect("recording set above"); + } + + Ok(Bytes::from(out.bytes)) +} + +/// The raw data block element ID of a program config element (ISO 14496-3 Table 4.85). +const ID_PCE: u32 = 5; + +/// The audioObjectTypes whose specific config is a GASpecificConfig (ISO 14496-3 §1.6.2.1), the +/// only ones where channelConfiguration 0 means a program config element follows. +const GENERAL_AUDIO: [u8; 12] = [1, 2, 3, 4, 6, 7, 17, 19, 20, 21, 22, 23]; + +/// Read an audioObjectType: 5 bits, escaped to 6 more when it reads 31. +fn read_object_type(reader: &mut BitReader) -> Result { + let object_type = reader.read(5, Error::ConfigTooShort)? as u8; + if object_type == 31 { + return Ok(32 + reader.read(6, Error::ExtendedConfigTooShort)? as u8); + } + Ok(object_type) +} + +/// Walk a program_config_element (ISO 14496-3 Table 4.2) to the number of channels it outputs. +/// +/// Its byte alignment is relative to where the reader started: the AudioSpecificConfig, or the +/// raw data block. +fn program_config(reader: &mut BitReader) -> Result { + let mut read = |n| reader.read(n, Error::ProgramConfigTruncated); + + // element_instance_tag, object_type, sampling_frequency_index. + read(10)?; + let (front, side, back) = (read(4)?, read(4)?, read(4)?); + let (lfe, assoc, cc) = (read(2)?, read(3)?, read(4)?); + // Mono and stereo mixdowns each name an element; a matrix mixdown an index and a flag. + for bits in [4, 4, 3] { + if read(1)? == 1 { + read(bits)?; + } + } + + let mut channels = lfe; + for _ in 0..front + side + back { + // A channel pair element carries two channels, a single channel element one. + channels += 1 + read(1)?; + read(4)?; + } + // Tags of the LFE and data elements, then each coupling channel element's switch flag and tag. + for _ in 0..lfe + assoc { + read(4)?; + } + for _ in 0..cc { + read(5)?; + } + + reader.align(); + let comment = reader.read(8, Error::ProgramConfigTruncated)?; + for _ in 0..comment { + reader.read(8, Error::ProgramConfigTruncated)?; + } + + if channels == 0 { + return Err(Error::ProgramConfigEmpty); + } + Ok(channels) +} + /// The 13 standard AAC sampling frequencies, indexed by samplingFrequencyIndex /// (ISO 14496-3 Table 1.18). Index 15 is the escape for an explicit 24-bit rate. const SAMPLE_RATES: [u32; 13] = [ @@ -147,6 +288,8 @@ struct BitReader<'a, T: Buf> { buf: &'a mut T, current: u8, bits_left: u8, + /// Every bit read is copied here when set, with alignment redone on the writer's own bytes. + record: Option, } impl<'a, T: Buf> BitReader<'a, T> { @@ -155,6 +298,15 @@ impl<'a, T: Buf> BitReader<'a, T> { buf, current: 0, bits_left: 0, + record: None, + } + } + + /// Skip to the next byte boundary. + fn align(&mut self) { + self.bits_left = 0; + if let Some(record) = &mut self.record { + record.align(); } } @@ -172,27 +324,52 @@ impl<'a, T: Buf> BitReader<'a, T> { self.bits_left -= 1; value = (value << 1) | u32::from((self.current >> self.bits_left) & 1); } + if let Some(record) = &mut self.record { + record.write(n, value); + } Ok(value) } } -/// Map an AAC `channel_config` (ISO 14496-3 Table 1.19) to its real channel count. -/// Configs 1..=6 happen to be identity (5.1 has config=6 and 6 channels). Config -/// 7 is 7.1 = 8 channels. Config 0 means "described elsewhere" — we default to -/// stereo. -fn channel_count_from_config(channel_config: u8) -> u32 { - match channel_config { - 1..=6 => channel_config as u32, - 7 => 8, - 0 => { - tracing::warn!("channel_config=0 (program config element) unsupported, defaulting to stereo"); - 2 - } - _ => { - tracing::warn!(channel_config, "unsupported channel config, defaulting to stereo"); - 2 +/// MSB-first bit writer, the inverse of [`BitReader`]. +#[derive(Default)] +struct BitWriter { + bytes: Vec, + /// Bits used in the last byte; 0 when the next write starts a new one. + used: u8, +} + +impl BitWriter { + /// Write the low `n` bits (n <= 32) of `value`, MSB-first. + fn write(&mut self, n: u8, value: u32) { + for i in (0..n).rev() { + if self.used == 0 { + self.bytes.push(0); + } + let bit = ((value >> i) & 1) as u8; + *self.bytes.last_mut().expect("pushed above") |= bit << (7 - self.used); + self.used = (self.used + 1) % 8; } } + + /// Pad the last byte with zeros. + fn align(&mut self) { + self.used = 0; + } +} + +/// Map an AAC `channel_config` (ISO 14496-3 Table 1.19) to its channel count, for every value but +/// 0, which a program config element describes instead. Configs 1..=6 are identity. 7, 12, and 14 +/// are 8 channels (7.1 with wide fronts, rear surrounds, or front heights), 11 is 6.1, and 13 is +/// 22.2. The rest are reserved. +fn channel_count_from_config(channel_config: u8) -> Result { + match channel_config { + 1..=6 => Ok(channel_config as u32), + 7 | 12 | 14 => Ok(8), + 11 => Ok(7), + 13 => Ok(24), + _ => Err(Error::ReservedChannelConfig(channel_config)), + } } /// Inverse of [`channel_count_from_config`]. Defaults to stereo for unsupported @@ -281,10 +458,177 @@ mod tests { assert_eq!(parsed.channel_count, 8, "7.1 surround should round-trip as 8 channels"); } + /// The AudioSpecificConfig ffmpeg 9.0.1 writes for quad (`-af pan=quad|...`, native `aac` + /// encoder, `.m4a`): channelConfiguration 0 and a program config element with two channel + /// pair elements, front and back, then a "Lavc63.1.101" comment and an SBR sync extension. + const FFMPEG_QUAD_ASC: [u8; 24] = [ + 0x11, 0x80, 0x04, 0xC4, 0x04, 0x00, 0x21, 0x10, 0x0C, 0x4C, 0x61, 0x76, 0x63, 0x36, 0x33, 0x2E, 0x31, 0x2E, + 0x31, 0x30, 0x31, 0x56, 0xE5, 0x00, + ]; + + /// Write a program config element with `front`, `side`, and `back` elements (true for a + /// channel pair) and `lfe` LFE elements, starting at the writer's position. + fn write_pce(out: &mut BitWriter, front: &[bool], side: &[bool], back: &[bool], lfe: u32) { + // element_instance_tag, object_type (LC), sampling_frequency_index (48 kHz). + out.write(4, 0); + out.write(2, 1); + out.write(4, 3); + out.write(4, front.len() as u32); + out.write(4, side.len() as u32); + out.write(4, back.len() as u32); + out.write(2, lfe); + // No data or coupling elements, and no mixdowns. + out.write(3, 0); + out.write(4, 0); + out.write(3, 0); + for (tag, &cpe) in front.iter().chain(side).chain(back).enumerate() { + out.write(1, cpe.into()); + out.write(4, tag as u32); + } + for tag in 0..lfe { + out.write(4, tag); + } + out.align(); + // A one-byte comment. + out.write(8, 1); + out.write(8, b'x'.into()); + } + + /// An AudioSpecificConfig with channelConfiguration 0 around [`write_pce`]'s element. + fn pce_asc(object_type: u8, front: &[bool], side: &[bool], back: &[bool], lfe: u32) -> Vec { + let mut out = BitWriter::default(); + out.write(5, object_type.into()); + out.write(4, 3); + out.write(4, 0); + out.write(3, 0); + write_pce(&mut out, front, side, back, lfe); + out.bytes + } + + #[test] + fn parses_ffmpeg_program_config_element() { + let cfg = Config::parse(&mut FFMPEG_QUAD_ASC.as_slice()).unwrap(); + assert_eq!(cfg.profile, 2); + assert_eq!(cfg.sample_rate, 48_000); + assert_eq!(cfg.channel_count, 4, "two channel pair elements"); + } + + #[test] + fn parses_program_config_element_counts() { + // 5.1: a single and a pair in front, a pair in back, and an LFE. + let asc = pce_asc(2, &[false, true], &[], &[true], 1); + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, 6); + + // Side elements count too: 3/2/2 with no LFE. + let asc = pce_asc(2, &[false, true], &[true], &[true], 0); + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, 7); + } + + #[test] + fn parses_program_config_element_behind_explicit_sbr() { + // audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension + // rate, then the core type (LC) whose GASpecificConfig carries the element. + let mut out = BitWriter::default(); + out.write(5, 5); + out.write(4, 6); + out.write(4, 0); + out.write(4, 3); + out.write(5, 2); + out.write(3, 0); + write_pce(&mut out, &[true], &[], &[true], 0); + let cfg = Config::parse(&mut out.bytes.as_slice()).unwrap(); + assert_eq!(cfg.profile, 5); + assert_eq!(cfg.channel_count, 4); + } + + #[test] + fn refuses_bad_program_config_elements() { + let asc = pce_asc(2, &[], &[], &[], 0); + assert!(matches!( + Config::parse(&mut asc.as_slice()), + Err(Error::ProgramConfigEmpty) + )); + + let asc = pce_asc(2, &[true], &[], &[], 0); + assert!(matches!( + Config::parse(&mut &asc[..asc.len() - 1]), + Err(Error::ProgramConfigTruncated) + )); + + // ALS (36) describes its channels in its own specific config, not a PCE. + let mut out = BitWriter::default(); + out.write(5, 31); + out.write(6, 4); + out.write(4, 3); + out.write(4, 0); + out.write(8, 0); + assert!(matches!( + Config::parse(&mut out.bytes.as_slice()), + Err(Error::ProgramConfigUnsupported(36)) + )); + } + + #[test] + fn channel_config_dispositions() { + let expected = [1, 2, 3, 4, 5, 6, 8]; + for (config, count) in (1..=7).zip(expected) { + let asc = [0x11, 0x80 | (config << 3)]; + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, count); + } + + for (config, count) in [(11, 7), (12, 8), (13, 24), (14, 8)] { + let asc = [0x11, 0x80 | (config << 3)]; + assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, count); + } + + for config in [8, 9, 10, 15] { + let asc = [0x11, 0x80 | (config << 3)]; + assert!(matches!( + Config::parse(&mut asc.as_slice()), + Err(Error::ReservedChannelConfig(c)) if c == config + )); + } + } + + #[test] + fn in_band_config_moves_the_program_config_element() { + // A raw data block leading with ID_PCE, which puts the element 3 bits off the byte grid + // the AudioSpecificConfig puts it on, so the copy has to redo the alignment. + let mut block = BitWriter::default(); + block.write(3, ID_PCE); + write_pce(&mut block, &[false, true], &[], &[true], 1); + // The channel elements that follow are never read. + block.write(8, 0xFF); + + let asc = in_band_config(2, 48_000, 0, &block.bytes).unwrap(); + assert_eq!(asc, pce_asc(2, &[false, true], &[], &[true], 1)); + assert_eq!(Config::parse(&mut asc.as_ref()).unwrap().channel_count, 6); + } + #[test] - fn channel_config_zero_falls_back_to_stereo() { - // Config 0 means "described in PCE" which we don't implement. - assert_eq!(channel_count_from_config(0), 2); + fn in_band_config_refuses_a_block_without_a_leading_pce() { + // ID_CPE first: any element past the channel data is out of reach. + assert!(matches!( + in_band_config(2, 48_000, 0, &[0x20, 0x00]), + Err(Error::ProgramConfigMissing) + )); + assert!(matches!( + in_band_config(2, 48_000, 0, &[]), + Err(Error::ProgramConfigMissing) + )); + } + + #[test] + fn in_band_config_matches_encode() { + // Without a PCE the block is unread and the config is the plain two-byte form. + let asc = in_band_config(2, 44_100, 2, &[]).unwrap(); + let encoded = Config { + profile: 2, + sample_rate: 44_100, + channel_count: 2, + } + .encode(); + assert_eq!(asc, encoded); } #[test] diff --git a/rs/moq-mux/src/container/ts/adts.rs b/rs/moq-mux/src/container/ts/adts.rs index b5a986a2f30..0166c059372 100644 --- a/rs/moq-mux/src/container/ts/adts.rs +++ b/rs/moq-mux/src/container/ts/adts.rs @@ -21,7 +21,8 @@ pub(super) struct Header { /// audioObjectType (ADTS `profile` + 1). AAC-LC is 2. pub object_type: u8, pub sample_rate: u32, - pub channel_count: u32, + /// channelConfiguration; 0 means a program config element in the raw data block describes the channels. + pub channel_config: u8, /// Total access-unit length, header included. pub frame_len: usize, /// Header length: 7 without CRC, 9 with. @@ -51,7 +52,7 @@ impl Header { Ok(Self { object_type: profile + 1, sample_rate, - channel_count: channel_count_from_config(channel_config), + channel_config, frame_len, header_len, }) @@ -96,16 +97,7 @@ fn freq_index_from_rate(sample_rate: u32) -> anyhow::Result { .with_context(|| format!("sample rate {sample_rate} not representable in ADTS")) } -/// Map an AAC `channel_config` (ISO 14496-3 Table 1.19) to a channel count. -fn channel_count_from_config(channel_config: u8) -> u32 { - match channel_config { - 1..=6 => channel_config as u32, - 7 => 8, - _ => 2, - } -} - -/// Inverse of [`channel_count_from_config`]. +/// Map a channel count to an AAC `channel_config` (ISO 14496-3 Table 1.19). fn channel_config_from_count(channel_count: u32) -> u8 { match channel_count { 1..=6 => channel_count as u8, @@ -126,7 +118,7 @@ mod tests { assert_eq!(parsed.object_type, 2); assert_eq!(parsed.sample_rate, 48_000); - assert_eq!(parsed.channel_count, 2); + assert_eq!(parsed.channel_config, 2); assert_eq!(parsed.header_len, 7); assert_eq!(parsed.frame_len, 107, "frame_len includes the 7-byte header"); } @@ -142,7 +134,7 @@ mod tests { fn frame_len_for_5_1() { let header = write_header(2, 44_100, 6, 512).unwrap(); let parsed = Header::parse(&header).unwrap(); - assert_eq!(parsed.channel_count, 6); + assert_eq!(parsed.channel_config, 6); assert_eq!(parsed.sample_rate, 44_100); assert_eq!(parsed.frame_len, 519); } diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index d811e24cc12..643069c77dd 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -2135,21 +2135,23 @@ impl AacStream { let import = match &mut self.import { Some(import) => import, None => { - let config = aac::Config { - profile: header.object_type, - sample_rate: header.sample_rate, - channel_count: header.channel_count, - }; + // Synthesize the AudioSpecificConfig `description` so out-of-band consumers + // (fMP4/MKV export, WebCodecs) can configure the decoder. A channel_config of 0 + // moves the program config element out of this first frame into it. + let asc = aac::in_band_config( + header.object_type, + header.sample_rate, + header.channel_config, + &data[offset + header.header_len..end], + )?; + let mut config = aac::config(&asc)?; + config.container = self.container.clone(); // Consume the reservation held since the PMT: this resolves the gated rendition, // and carries the catalog's declared media retention onto the track. - // The importer synthesizes the AudioSpecificConfig `description` from the config so - // out-of-band consumers (fMP4/MKV export, WebCodecs) can configure the decoder. let reserved = self.reserved.take().expect("aac reservation already consumed"); let track = self .broadcast .unique_track(".aac", reserved.track_info(hang::catalog::PRIORITY.audio))?; - let mut config: hang::catalog::AudioConfig = config.into(); - config.container = self.container.clone(); let aac = aac::Import::new(track, reserved, config)?; self.import.insert(aac) } diff --git a/rs/moq-mux/src/container/ts/import_test.rs b/rs/moq-mux/src/container/ts/import_test.rs index 348f2cf926c..d0c48534ac6 100644 --- a/rs/moq-mux/src/container/ts/import_test.rs +++ b/rs/moq-mux/src/container/ts/import_test.rs @@ -156,6 +156,32 @@ fn import_ac3_catalog() { assert!(audio.description.is_none(), "verbatim AC-3 needs no description"); } +/// `aac_quad.ts` is an ffmpeg-authored audio-only AAC program in quad, which has no +/// channelConfiguration, so its ADTS headers carry 0 and the first raw data block leads with a +/// program config element. Regenerated with (ffmpeg 9.0.1): +/// `ffmpeg -f lavfi -i sine=frequency=440:sample_rate=48000:duration=0.1 +/// -af "pan=quad|FL=c0|FR=c0|BL=c0|BR=c0" -c:a aac -b:a 128k -f mpegts aac_quad.ts`. +#[test] +fn import_aac_program_config_catalog() { + let data = include_bytes!("test_data/aac_quad.ts"); + let catalog = import_ts(data); + + assert_eq!(catalog.audio.renditions.len(), 1, "expected one AAC track"); + let audio = catalog.audio.renditions.values().next().unwrap(); + assert_eq!(audio.codec.to_string(), "mp4a.40.2"); + assert_eq!(audio.sample_rate, 48_000); + assert_eq!( + audio.channel_count, 4, + "two channel pair elements, not a guessed stereo" + ); + + // The element moved into the description is byte-for-byte what ffmpeg itself writes as the + // AudioSpecificConfig for the same stream in MP4, minus the trailing SBR sync extension. + let mut expected = vec![0x11, 0x80, 0x04, 0xC4, 0x04, 0x00, 0x21, 0x10, 0x0C]; + expected.extend_from_slice(b"Lavc63.1.101"); + assert_eq!(audio.description.as_deref(), Some(expected.as_slice())); +} + /// `opus.ts` is an ffmpeg-authored audio-only Opus program (private stream_type 0x06 /// plus the 'Opus' registration and DVB extension descriptors), generated with: /// `ffmpeg -f lavfi -i sine=frequency=440:sample_rate=48000:duration=0.5 diff --git a/rs/moq-mux/src/container/ts/test_data/aac_quad.ts b/rs/moq-mux/src/container/ts/test_data/aac_quad.ts new file mode 100644 index 0000000000000000000000000000000000000000..17aff4786200068fa22e9b89662f19d2ce14ce7f GIT binary patch literal 2632 zcmd6oc~nzZ8o*x&5@etu7#5`(Ng_s&%_VhE*-QwKxUk4naAXTh6l4I|B#;EqAuL4< z5-JL5hDE0Aj0<5qKn@^P>0Xgo^1D+$nimvDz#4H10Fty* z004lA5yfowCKCWL8~`e4fVp%{P{?MhW3_1kaY4lWmNu3));6QS#wWn(l$NpEVVdK! zLtT>|J-0*HOm*J0JUoLUa8M9t8%V{u?Ge*ho-{ zTWH&EwqJz(p)$HUESAKRMLW>Tc;186b@kH1#g_${^a|ghs{`T<@r1+4DI$h%)x$gY z77@96dxH6UAvt%lqYGu`k*!x~3VFn1%BTM=7yE|YMK!Ii@C^tO*~so%pu0}ouhuE! zY=}HK+?W_XybtdJF(dRKcRjpEkCkedk7CwL+`^BaF>mOKr8lSE{%m#m^@tB2%*BXq zU&uKpQd;?a=(H=3%=!#fMV#nuo-R(fU{R2C#ZV-1Xl+IeTuCNiPL?;q7FLo8@R@~` z3+qWBMpg{YRQ0kC_WA7fDOE-^lT5fJgfHj0C06;Q*7@ZqZrarRYZ}Ii-%ZGX%`Y<_ zwCk$%6gP5AHpFhzs{9`tp7h;wOHSp@vLN>@qGFwDDp-*#ead;CoT9vhT=koL$|hEX zjK%c*;^k8Jj%C4D`fi)9Hn8wu5y!B)ZGHK)Wm1pj<)`ZDd1XY?sv8n|bHkqY3|MgA8S%9Z!Q*+m zB}(X_BfIQcU4Ifu`!t5iz`;)0w1cum#Ii-auT@Mt7<)EeY?{9N!K?K1IQxPg&>|;h zD9fPIzUX3}W?*QyDN`8CKN_m$l%P8A9e!&j32hR;&%4%GCtuVY`myaTqr8?HIa=_w z8wb2Puv&uIS`;rG#Y~QOEiS?*$C0cAN@qs^wmW z@K0s;=cuR49Kgqnk6ZoKb)vsXrnFaDt%;=Xr>M`ZY$cSG zhYv=bHHF*@{XweF)bNr!iHvq}5+-X_YMR+r-O}u^QTNnikZc`4xzfi{fC|=>5|o*p zt4ti~KK#>I_o!Pd+TG%^c9?4MxlBTakgMr+@+-*QP~T(|0b#2qyf3=y+{SMpMtFZ6iR4hr&!OLFlBv<$JP=qx*CEgjjX@yIIg4v}^f! zJMOoiQI;fI$~@j&cm33zfj=q6G98v?Gn^$Hxiym%oQ-626{rGhJFVSq6c_Olp#-%Z z2OqDQW#WzLLh4)16qIO%RWZbmuZ*T%r)c}QF0fk_@W~nz;}FDfs{(b;p9#4eX=6}Z zgW%&rCh1Z(G8IA&SlhwJMCzTafH^6f_^Nwud1;sLd7M$($aY)>R=;lr%*~*F*R-n( zA5)EAa`?8P_~{j1cymneuiW2au4^tvw%SOClkSh}+;4c;lCs(RWo~TCW@CC$P-*P> zTVUt}ubICPzqwy!(M*3dIH@^<5^5o`)({8X2f9tEG}s;f3H6bMr@Z)#1Ielg?a%Mg^7 zevv9P3){2D>Yl3gI`BzOW)b!)LH}MaiN0Im^BXOn9U8jX6isc*>%tC@fJb-k5Cp>V z)FuY5`Lw}HXd+H<2jaO=e^9mhI`F>*mOr@@cHj^CL^3KnFsGdSE90fhzw#JZE%~;P zz4#-LyX|4%|NqT1Is(Ov4(n?QHx*CgmPZp3UAw7VHE6CHJe#<{JWPy?AAcr)C;3L2 zMrp^kI1)zBIokF~8;$DE6TzBGEODD$-5! Qn0E6##`H6}Y$K`j-^lX>rT_o{ literal 0 HcmV?d00001 From a963040c8883c0dd1c5472d4f2ba03bde4572706 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 05:25:47 -0700 Subject: [PATCH 03/20] chore(quest): abandon he-aac-refusal (#4090) Co-authored-by: Claude Opus 5.5 --- doc/bin/cli.md | 2 ++ doc/lib/rs/moq-audio.md | 4 +++ quest/m1/audio-codecs/README.md | 7 +++--- quest/m1/audio-codecs/decode-backend.md | 6 +++-- quest/m1/audio-codecs/he-aac-refusal.md | 33 ------------------------- 5 files changed, 14 insertions(+), 38 deletions(-) delete mode 100644 quest/m1/audio-codecs/he-aac-refusal.md diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 1376686f9dd..cbd4d9d03a4 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -90,6 +90,8 @@ moq ... play --delay 500ms # trade latency for a jittery link Decodes H.264, H.265, and AV1 video and Opus, PCM, and AAC-LC audio using the platform hardware decoder where available. `--video-name` and `--audio-name` pick a rendition. +HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) plays as its +half-rate AAC-LC core. Playback runs on a clock it owns. `--delay` (default 100 ms) is how far it trails the live edge, which is both the jitter a late frame may absorb and the diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 82f65a3306a..5fee2602e77 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -27,6 +27,10 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | +AAC decoding refuses HE-AAC its config declares. HE-AAC signaled only in band +(implicit SBR, as over MPEG-TS) goes undetected and plays as its half-rate +AAC-LC core. + Highlights: - **`encode::Publication`** advertises the track and opens the microphone only while someone listens. Stop, swap devices, and restart without changing the track subscribers know; read a level meter for the UI. diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index c1366d831b5..6682cbc9c25 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -16,8 +16,10 @@ publisher can produce AAC. Platform first, exactly like video: AudioToolbox on macOS and iOS, Media Foundation on Windows, MediaCodec on Android, and symphonia (AAC-LC mono/stereo) as the software fallback that openh264 is for H.264. Linux has no -OS audio decoder, so HE-AAC and multichannel AAC stay refused there, stated in -the docs and rejected at construction. A platform backend claims every catalog +OS audio decoder, so multichannel AAC stays refused there, stated in the docs +and rejected at construction. HE-AAC signaled only in band (implicit SBR, as +over MPEG-TS) plays as its half-rate LC core on symphonia; detecting it needs a +full element walk, so the docs state it instead of refusing it. A platform backend claims every catalog codec its framework opens, so AC-3, E-AC-3, MP3, and FLAC ride along on the hosts that have them; each still needs a fixture before the backend advertises it. @@ -41,7 +43,6 @@ its own decode and encode quest so verification stays per host. ## Quests -- [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - implicit-SBR HE-AAC over TS is refused instead of half-decoded as the LC core - [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config - [Layout](/quest/m1/audio-codecs/layout.md) - the settled `Layout` carries up to 7.1 through decode, resample, playback, and the FFI - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video diff --git a/quest/m1/audio-codecs/decode-backend.md b/quest/m1/audio-codecs/decode-backend.md index ecfb6f64ba9..490f5eac844 100644 --- a/quest/m1/audio-codecs/decode-backend.md +++ b/quest/m1/audio-codecs/decode-backend.md @@ -22,8 +22,10 @@ stats and `moq play` surface. AAC-LC mono/stereo only; the platform backends that follow advertise what their framework opens and has a fixture for. - Move today's Opus, PCM, and symphonia code behind the trait without changing - behavior; the HE-AAC sniff from [HE-AAC refusal](/quest/m1/audio-codecs/he-aac-refusal.md) - lands in the symphonia backend. + behavior. Document per host that symphonia plays implicit-SBR HE-AAC as its + half-rate LC core: finding the in-band SBR element needs a full Huffman walk + of the channel elements, and symphonia detects it internally without + exposing or refusing it. - A backend's output rate and layout are what it produced, not what the catalog said (HE-AAC doubles the rate); `Consumer` already resamples and remixes to the requested output, so that stays the seam's contract. diff --git a/quest/m1/audio-codecs/he-aac-refusal.md b/quest/m1/audio-codecs/he-aac-refusal.md deleted file mode 100644 index c3edf29e218..00000000000 --- a/quest/m1/audio-codecs/he-aac-refusal.md +++ /dev/null @@ -1,33 +0,0 @@ -# [S] Refuse implicit-SBR HE-AAC instead of half-decoding it - -## Goal - -An HE-AAC stream that signals AAC-LC and carries SBR only in band is refused -with a clear error, the same way explicitly signaled and backward-compatible -HE-AAC already are. Today it decodes as the LC core at half the sample rate -with no indication anything is wrong. - -## Plan - -ADTS carries a 2-bit profile, so HE-AAC over MPEG-TS (SRT, `moq import ts`) -always arrives as `mp4a.40.2` with a synthesized LC AudioSpecificConfig. The -config-level checks in `rs/moq-audio/src/aac.rs` cannot see it, and the code -says so. The stream itself can: the first raw data block carries an `ID_FIL` -element with `EXT_SBR_DATA` (or `EXT_SBR_DATA_CRC`), and every later frame -does too. - -- Sniff the fill elements of the first packet before handing it to symphonia. - An SBR extension makes the track `Error::Unsupported` with the same wording - as the config-level refusal, naming the host's decoder as the reason. -- Symphonia already walks the element tree, so the sniff is a small parser - over the same syntax, not a decode. It runs once per track. -- Regression: an ADTS fixture with implicit SBR is refused; the existing - explicit and backward-compatible fixtures keep their errors; an LC fixture - with an unrelated fill element still decodes. -- When a platform decoder handles the track ([Decode seam](/quest/m1/audio-codecs/decode-backend.md)), - the sniff belongs to the symphonia backend only; the OS decoders read SBR in - band themselves. - -## Related - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the sniff becomes the software backend's contract From c0c78568c13f59bdd88eba6295adf7f654df04b9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 07:22:21 -0700 Subject: [PATCH 04/20] feat(audio): carry surround layouts through decode, remix, and playback (#4119) Co-authored-by: Claude Opus 5.5 --- doc/lib/c/index.md | 1 + doc/lib/go/index.md | 6 + doc/lib/kt/index.md | 6 + doc/lib/py/index.md | 6 + doc/lib/rs/moq-audio.md | 10 +- doc/lib/swift/index.md | 6 + js/hang/src/util/aac.test.ts | 8 +- js/hang/src/util/aac.ts | 10 +- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/decode-audiotoolbox.md | 1 - quest/m1/audio-codecs/decode-backend.md | 4 - quest/m1/audio-codecs/encode-audiotoolbox.md | 1 - quest/m1/audio-codecs/layout.md | 54 ---- quest/m1/audio-codecs/opus-surround.md | 4 - quest/m2/aac-encode-refusal.md | 1 - quest/m2/audio-decode-mediacodec.md | 1 - quest/m2/audio-decode-mediafoundation.md | 1 - quest/m2/audio-encode-mediacodec.md | 1 - quest/m2/audio-encode-mediafoundation.md | 1 - rs/libmoq/src/audio.rs | 7 +- rs/moq-audio/src/aec.rs | 28 +- rs/moq-audio/src/decode/consumer.rs | 16 +- rs/moq-audio/src/encode/capture.rs | 35 +-- rs/moq-audio/src/encode/encoder.rs | 6 +- rs/moq-audio/src/encode/producer.rs | 16 +- rs/moq-audio/src/layout.rs | 132 ++++++++- rs/moq-audio/src/playback.rs | 7 +- rs/moq-audio/src/playback/device.rs | 120 +++++--- rs/moq-audio/src/playback/driver.rs | 30 +- rs/moq-audio/src/playback/mixer.rs | 205 ++++++++------ rs/moq-audio/src/playback/sink.rs | 170 ++++++++---- rs/moq-audio/src/resample.rs | 271 ++++++++++++++++--- rs/moq-audio/tests/roundtrip.rs | 68 +++++ rs/moq-ffi/src/audio.rs | 7 +- 34 files changed, 901 insertions(+), 340 deletions(-) delete mode 100644 quest/m1/audio-codecs/layout.md diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index 15de2348946..69fec60fbda 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -36,6 +36,7 @@ and `target/include/moq.h`. - **Connection health.** `moq_session_stats()` reports available metrics with per-field validity flags. `moq_session_snapshot()` samples those metrics and the negotiated draft name together from the same connection. Its protocol string is backed by static storage. Both return an offline error between reconnects and leave the destination untouched. `moq_session_bandwidth()` mints an allocator over the send estimate; `moq_bandwidth_reserve` claims a share for an app-owned track, and `moq_encode_video` / `moq_encode_audio` take the same handle so the built-in video encoder follows the grant. - **Raw playback.** Raw audio and video consumers start at the newest cached group when opened, so rebuilding a live decoder skips the retained backlog. - **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec (only `"opus"` today), and `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the 20 ms default. +- **Audio channel layouts.** A `channels` count also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. `moq_decode_audio` remixes to the count you ask for; past 8 channels the samples pass through but can't be remixed. - **Raw decode output.** `moq_video_decoder_output` selects the decoded CPU pixel format (`MOQ_VIDEO_PIXEL_FORMAT_I420` or `_RGBA`) and target size (`width`/`height`, both zero for native; otherwise even and non-zero). Unknown formats and invalid sizes fail `moq_decode_video` before subscribing; accepted requests deliver exactly that layout or fail on the terminal callback. - **Encoded video metadata.** `moq_video_init.hint` is a zero-initialized `moq_video_hint` with `has_*` flags for coded dimensions, bitrate (bits per second), frame rate, and latency preference. Hints seed a video codec track's catalog; detected dimensions take precedence. - **Client config.** A zeroed `moq_client_config` means the defaults for every knob, which is what lets a new one be appended without disturbing callers. Fields cover protocol (`versions`), TLS (`tls_fingerprints`, `tls_roots`, `tls_cert`/`_key`, `tls_host_name`), transport (`bind`, `connect_timeout_us`, the Happy Eyeballs delays, `websocket_enabled`/`_delay_us`), and tuning (reconnect backoff, `quic_*`). Every duration is in microseconds. A knob whose default isn't zero carries a `has_*` flag, so setting `backoff_timeout_us = 0` needs `has_backoff_timeout = true` to mean "retry forever" rather than "use the default". `moq_client_defaults()` reports what a NULL config dials with. diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index 001f5a50e4a..0540cafb905 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -115,6 +115,12 @@ one: `FetchGroup`/`FetchMediaGroup`, `Dynamic()` with `Requests(ctx)`, `AppendDatagram`/`Datagrams(ctx)`, `SetCatalogSection`, `Demand()` for `Used`/`Unused`, `Session().Stats()`. `moq.IsAuthError` and `moq.IsShutdown` classify errors. `moq.ProtocolError(err)` is the structured protocol failure (scope, verbatim code, kind) when the peer sent one. +Audio `Channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `DecodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.Format` is I420 when nil, or `VideoPixelFormatRgba` for four bytes a pixel, and every `VideoDecodedFrame` repeats the layout it was decoded to. `Resize` is best diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index a241ad6b0cb..8f863b942b1 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -94,6 +94,12 @@ native side. `AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +Audio `channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `decodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.format` is `VideoPixelFormat.I420` when null, or `VideoPixelFormat.RGBA` for four bytes a pixel, and every frame repeats the layout it was decoded to. `resize` is best diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index a41cb7255bd..5fe4e12de39 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -110,6 +110,12 @@ WebSocket, TCP, and Unix sockets. `moq.AudioCodec.opus()`, and `AudioEncoderOutput.frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +Audio `channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `decode_video` picks the decoded CPU pixel layout: `VideoDecoderOutput.format` is `VideoPixelFormat.I420` when unset, or `VideoPixelFormat.RGBA` for four bytes a pixel, and every frame repeats the layout it was decoded to. `resize` diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 5fee2602e77..2d57b4a5bc5 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -13,8 +13,14 @@ the far end. Everything is Rust, so there is no C toolchain, CMake step, or codec to install. `Layout` names speaker meaning separately from a channel count. `Mono` is center, -`Stereo` is left then right, and `Discrete(n)` preserves unnamed channels without -inventing speaker positions. Encoding keeps source PCM in `encode::Input` and +`Stereo` is left then right, and the surround layouts up to `SevenPointOne` +interleave in the SMPTE/WAVE order (front left, front right, center, LFE, back, +side). A catalog carries only a count, which reads as that count's WAVE default +(`Layout::from_channels`: 6 is 5.1, 8 is 7.1). `Discrete(n)` preserves unnamed +channels without inventing speaker positions, so it passes through but never +remixes. Decoding and playback downmix with the ITU-R BS.775 coefficients and +upmix by leaving the extra speakers silent; the playback mix runs in whatever +layout the output device opened. Encoding keeps source PCM in `encode::Input` and codec requirements in `encode::Settings`; `encode::Options` adds publication policy. Decoding likewise separates low-level `decode::Config`, PCM `decode::Output`, and subscription `decode::Options`. diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index a9aa8b3c440..dc02120fe7a 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -98,6 +98,12 @@ divides the connection's send estimate; pass it to `encodeVideo` / `AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +Audio `channels` also names the speaker layout, by the WAVE convention: 1 is +mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved +front left, front right, center, LFE, back, then side. Decoding remixes to the +count you ask for; past 8 channels the samples pass through but can't be +remixed. + `decodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.format` is `.i420` when unset, or `.rgba` for four bytes a pixel, and every frame repeats the layout it was decoded to. `resize` is best effort: only NVDEC has a diff --git a/js/hang/src/util/aac.test.ts b/js/hang/src/util/aac.test.ts index c31059ec80f..50dfe903ad9 100644 --- a/js/hang/src/util/aac.test.ts +++ b/js/hang/src/util/aac.test.ts @@ -48,9 +48,11 @@ describe("audioSpecificConfig", () => { expect(audioSpecificConfig(48000, 8)).toEqual(new Uint8Array([0x11, 0xb8])); }); - // Unsupported channel counts fall back to stereo (config 2). - it("unsupported channel count falls back to stereo", () => { - expect(audioSpecificConfig(48000, 7)).toEqual(audioSpecificConfig(48000, 2)); + // A count with no channel configuration is refused rather than described as stereo. + it("refuses a channel count with no configuration", () => { + for (const count of [0, 7, 9]) { + expect(() => audioSpecificConfig(48000, count)).toThrow(); + } }); // Non-table sample rates use the 5-byte explicit-frequency form (freqIndex 0xF). diff --git a/js/hang/src/util/aac.ts b/js/hang/src/util/aac.ts index f417e4c0bdb..ca0cc2fd45b 100644 --- a/js/hang/src/util/aac.ts +++ b/js/hang/src/util/aac.ts @@ -45,19 +45,19 @@ export function pickRate(rate: number): number { const AAC_LC = 2; // audioObjectType for AAC-LC // Map a channel count to its AAC channelConfiguration (ISO 14496-3 Table 1.19). Configs 1..=6 are -// identity (5.1 is config 6 / 6 channels); 8 channels is config 7 (7.1). Anything else has no valid -// config, so fall back to stereo (matching the Rust muxer in rs/moq-mux/src/codec/aac). +// identity (5.1 is config 6 / 6 channels); 8 channels is config 7 (7.1). Anything else has no +// config, and describing it as stereo would mislabel every channel past the first two. function channelConfig(channelCount: number): number { - if (channelCount >= 1 && channelCount <= 6) return channelCount; + if (Number.isInteger(channelCount) && channelCount >= 1 && channelCount <= 6) return channelCount; if (channelCount === 8) return 7; - return 2; + throw new Error(`no AAC channel configuration for ${channelCount} channels`); } /** * Build the AAC-LC AudioSpecificConfig that decoders need when frames are raw (no ADTS header). * * Standard sample rates produce the 2-byte form; non-table rates fall back to the 5-byte form - * with an explicit 24-bit frequency. Mirrors the Rust muxer so JS and Rust agree on the bytes. + * with an explicit 24-bit frequency. Throws for a channel count no configuration names. */ export function audioSpecificConfig(sampleRate: number, channelCount: number): Uint8Array { const config = channelConfig(channelCount); diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 6682cbc9c25..78912f392c9 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,7 +44,6 @@ its own decode and encode quest so verification stays per host. ## Quests - [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config -- [Layout](/quest/m1/audio-codecs/layout.md) - the settled `Layout` carries up to 7.1 through decode, resample, playback, and the FFI - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder diff --git a/quest/m1/audio-codecs/decode-audiotoolbox.md b/quest/m1/audio-codecs/decode-audiotoolbox.md index effbebb8098..e81db0d5543 100644 --- a/quest/m1/audio-codecs/decode-audiotoolbox.md +++ b/quest/m1/audio-codecs/decode-audiotoolbox.md @@ -32,7 +32,6 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. ## Required - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - what a multichannel frame is delivered as ## Related diff --git a/quest/m1/audio-codecs/decode-backend.md b/quest/m1/audio-codecs/decode-backend.md index 490f5eac844..271ef84b526 100644 --- a/quest/m1/audio-codecs/decode-backend.md +++ b/quest/m1/audio-codecs/decode-backend.md @@ -39,7 +39,3 @@ stats and `moq play` surface. backend like the video seam's `probe`. The FFI does not expose `Kind` until a consumer asks. - -## Related - -- [Layout](/quest/m1/audio-codecs/layout.md) - independent; the platform backends need both diff --git a/quest/m1/audio-codecs/encode-audiotoolbox.md b/quest/m1/audio-codecs/encode-audiotoolbox.md index 0f869538358..2ceda88280e 100644 --- a/quest/m1/audio-codecs/encode-audiotoolbox.md +++ b/quest/m1/audio-codecs/encode-audiotoolbox.md @@ -22,5 +22,4 @@ the encode seam as the platform candidate on macOS and iOS. ## Required - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - the input layout the encoder accepts - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the round-trip regression decodes through it diff --git a/quest/m1/audio-codecs/layout.md b/quest/m1/audio-codecs/layout.md deleted file mode 100644 index e92dfbbe381..00000000000 --- a/quest/m1/audio-codecs/layout.md +++ /dev/null @@ -1,54 +0,0 @@ -# [M] Channel layouts through decode, playback, and the FFI - -## Goal - -`moq-audio` carries up to 7.1 end to end. A decoded frame says which layout -it is in, the playback mixer downmixes it to whatever the device opened, a raw -PCM consumer can ask for a layout, and the FFI and libmoq expose the same. -Proven with multichannel PCM, the one codec that needs no new decoder. - -## Plan - -Extend the Layout contract settled in main with supported surround layouts -(2.1, quad, 5.0, 5.1, 6.1, 7.1, and the other AAC channelConfiguration and -Opus mapping family 1 entries) in one canonical order, the SMPTE/WAVE order. -Keep the representation extensible. Each codec module maps its native -order into it: AAC's `C L R Ls Rs LFE` and Opus's Vorbis `L C R Ls Rs LFE` -both become `L R C LFE Ls Rs`. An unspecified discrete PCM layout remains -valid for passthrough but is refused for spatial remixing; never invent speaker -positions from an arbitrary channel count. An AAC stream with -channelConfiguration 0 takes its layout from the program config element's -front, side, back, and LFE elements, which `moq_mux::codec::aac` already parses -for the channel count. - -- Reuse the settled PCM descriptors and codec/consumer settings. This quest - adds supported layout behavior rather than replacing public field types. - Frame stays layout-free since the consumer fixed it at construction. -- `resample::remix` becomes a generic remix over layouts: ITU-R BS.775 - coefficients for downmix, silence in the extra speakers for upmix, and the - existing mono/stereo paths as the two-channel special cases. The resampler is - already channel-generic. -- Playback: the mix bus takes the device's layout instead of fixed stereo, the - device chooser prefers the widest well-known layout the device offers (a - 5.1 HDMI sink opens at six channels, a headset at two), and each `Sink` - remixes into the bus. Today's "silence past the front pair" fan-out goes. -- moq-ffi and libmoq keep `channels` as a count, and the count means the - default layout for that count (the WAVE convention: 3 is 2.1, 4 is quad, 6 - is 5.1, 8 is 7.1), delivered or accepted in the canonical order. No record, - `repr(C)` struct, or binding changes, so this stays on `main`; only the - Rust API names the layout, and a binding that needs 5.0 rather than 5.1 is - a later additive field. Document the mapping in every binding's audio doc. -- The catalog does not change: `channel_count` already carries what the - description implies, and `js/hang`'s `aac.ts` stops falling back to stereo - for a count it cannot map. -- Regressions: a 5.1 PCM broadcast published through `encode::Producer` and - read back through `decode::Consumer` at 5.1, at stereo, and at mono; the - mixer fed a 5.1 sink into a stereo bus and a stereo sink into a 5.1 bus; the - device chooser picking six channels when offered. - -Capture stays mono/stereo, and Opus encode stays mapping family 0. - -## Related - -- [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - the first coded multichannel consumer of the layout -- [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the first platform decoder producing more than stereo diff --git a/quest/m1/audio-codecs/opus-surround.md b/quest/m1/audio-codecs/opus-surround.md index de6247ae0c0..059552dddef 100644 --- a/quest/m1/audio-codecs/opus-surround.md +++ b/quest/m1/audio-codecs/opus-surround.md @@ -20,7 +20,3 @@ so this is the one multichannel path Linux gets. channels. - Regression: a family-1 5.1 fixture decodes to six channels in canonical order; family 255 is refused at construction. - -## Required - -- [Layout](/quest/m1/audio-codecs/layout.md) - the type the mapping resolves to diff --git a/quest/m2/aac-encode-refusal.md b/quest/m2/aac-encode-refusal.md index 3008bf0702b..316e7f72e29 100644 --- a/quest/m2/aac-encode-refusal.md +++ b/quest/m2/aac-encode-refusal.md @@ -19,4 +19,3 @@ every count from 1 to 8 and one beyond. ## Related - [AAC PCE](https://github.com/moq-dev/moq/pull/4093) - the parse half -- [Layout](/quest/m1/audio-codecs/layout.md) - the layout a PCE would be derived from diff --git a/quest/m2/audio-decode-mediacodec.md b/quest/m2/audio-decode-mediacodec.md index 936bc13a325..5b7df87c71c 100644 --- a/quest/m2/audio-decode-mediacodec.md +++ b/quest/m2/audio-decode-mediacodec.md @@ -26,7 +26,6 @@ behind a new optional audio `mediacodec` feature and the decode seam, on `target ## Required - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - what a multichannel frame is delivered as ## Related diff --git a/quest/m2/audio-decode-mediafoundation.md b/quest/m2/audio-decode-mediafoundation.md index 20a25184589..1da6a8337c0 100644 --- a/quest/m2/audio-decode-mediafoundation.md +++ b/quest/m2/audio-decode-mediafoundation.md @@ -25,7 +25,6 @@ ones. Behind the decode seam as the first candidate on `target_os = ## Required - [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - what a multichannel frame is delivered as ## Related diff --git a/quest/m2/audio-encode-mediacodec.md b/quest/m2/audio-encode-mediacodec.md index 7bb26674eaf..d488ec184ba 100644 --- a/quest/m2/audio-encode-mediacodec.md +++ b/quest/m2/audio-encode-mediacodec.md @@ -21,5 +21,4 @@ behind the `mediacodec` feature and the encode seam. ## Required - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - the input layout the encoder accepts - [MediaCodec decode](/quest/m2/audio-decode-mediacodec.md) - the round-trip regression decodes through it diff --git a/quest/m2/audio-encode-mediafoundation.md b/quest/m2/audio-encode-mediafoundation.md index b1c3440f6bc..8347dd1854f 100644 --- a/quest/m2/audio-encode-mediafoundation.md +++ b/quest/m2/audio-encode-mediafoundation.md @@ -21,7 +21,6 @@ behind the encode seam on Windows. ## Required - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins -- [Layout](/quest/m1/audio-codecs/layout.md) - the input layout the encoder accepts - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the round-trip regression decodes through it ## Related diff --git a/rs/libmoq/src/audio.rs b/rs/libmoq/src/audio.rs index b23a57c18e0..d389abc0fee 100644 --- a/rs/libmoq/src/audio.rs +++ b/rs/libmoq/src/audio.rs @@ -67,6 +67,9 @@ pub struct moq_audio_encoder_input { /// `moq_audio_sample_format` discriminant. pub format: u32, pub sample_rate: u32, + /// Interleaved channel count, which also names the speaker layout by the + /// WAVE convention: 1 mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, + /// 8 7.1, in front left, front right, center, LFE, back, side order. pub channels: u32, } @@ -98,7 +101,9 @@ pub struct moq_audio_decoder_output { pub format: u32, /// 0 = deliver at the codec's native sample rate. pub sample_rate: u32, - /// 0 = deliver at the codec's native channel count. + /// 0 = deliver at the codec's native channel count. A count names its + /// layout as `moq_audio_encoder_input.channels` describes, and the decoder + /// remixes to it. pub channels: u32, /// Upper bound on buffering before skipping a stalled group, in /// microseconds. Same congestion-control knob as diff --git a/rs/moq-audio/src/aec.rs b/rs/moq-audio/src/aec.rs index 607d0111cae..da24dda60a1 100644 --- a/rs/moq-audio/src/aec.rs +++ b/rs/moq-audio/src/aec.rs @@ -40,7 +40,7 @@ use sonora::config::{EchoCanceller, GainController2, NoiseSuppression, NoiseSupp use sonora::{AudioProcessing, StreamConfig}; use crate::Error; -use crate::playback::{self, BUS_CHANNELS}; +use crate::playback::{self, REFERENCE_CHANNELS}; /// Sample rate the echo reference is resampled to on its way out of the mixer. /// @@ -187,7 +187,7 @@ impl Control { "echo cancellation needs a microphone between 8 and 384 kHz (got {sample_rate})" ))); } - if channels == 0 || channels > BUS_CHANNELS as u32 { + if channels == 0 || channels > REFERENCE_CHANNELS as u32 { return Err(Error::Unsupported(format!( "echo cancellation accepts a mono or stereo microphone (got {channels} channels)" ))); @@ -381,9 +381,9 @@ impl State { self.pending = Vec::with_capacity(headroom); self.processed = Vec::with_capacity(headroom); - self.reference_frame = vec![0.0; REFERENCE_FRAME * BUS_CHANNELS]; - self.render_in = vec![0.0; REFERENCE_FRAME * BUS_CHANNELS]; - self.render_out = vec![0.0; REFERENCE_FRAME * BUS_CHANNELS]; + self.reference_frame = vec![0.0; REFERENCE_FRAME * REFERENCE_CHANNELS]; + self.render_in = vec![0.0; REFERENCE_FRAME * REFERENCE_CHANNELS]; + self.render_out = vec![0.0; REFERENCE_FRAME * REFERENCE_CHANNELS]; self.capture_in = vec![0.0; frame * channels]; self.capture_out = vec![0.0; frame * channels]; @@ -453,7 +453,7 @@ impl State { if let Some(reference) = reference { while reference.available_frames() >= REFERENCE_FRAME { reference.read_interleaved(reference_frame, false); - deinterleave(reference_frame, render_in, BUS_CHANNELS); + deinterleave(reference_frame, render_in, REFERENCE_CHANNELS); let _ = process_render(processor, render_in, render_out); } } @@ -484,7 +484,7 @@ impl State { /// The format the mixer's reference tap arrives in. fn reference_config() -> StreamConfig { - StreamConfig::new(REFERENCE_RATE, BUS_CHANNELS as u16) + StreamConfig::new(REFERENCE_RATE, REFERENCE_CHANNELS as u16) } /// Feed one reference frame to the echo model. @@ -602,7 +602,7 @@ impl Reference { /// canceller reads it back at [`REFERENCE_RATE`]. fn channel(rate: u32) -> (ResamplingProd, ResamplingCons) { resampling_channel::( - BUS_CHANNELS, + REFERENCE_CHANNELS, rate, REFERENCE_RATE, true, @@ -652,7 +652,7 @@ mod tests { /// Push one 10 ms stereo frame of `value` into the tap. fn play(prod: &mut ResamplingProd, value: f32) { - prod.push_interleaved(&vec![value; REFERENCE_FRAME * BUS_CHANNELS]); + prod.push_interleaved(&vec![value; REFERENCE_FRAME * REFERENCE_CHANNELS]); } #[test] @@ -976,10 +976,10 @@ mod tests { fn round(&mut self) -> (f64, f64) { let played: Vec = (0..self.frame).map(|_| self.noise.next()).collect(); - let mut reference = vec![0.0f32; self.frame * BUS_CHANNELS]; + let mut reference = vec![0.0f32; self.frame * REFERENCE_CHANNELS]; for (i, sample) in played.iter().enumerate() { - reference[i * BUS_CHANNELS] = *sample; - reference[i * BUS_CHANNELS + 1] = *sample; + reference[i * REFERENCE_CHANNELS] = *sample; + reference[i * REFERENCE_CHANNELS + 1] = *sample; } self.prod.push_interleaved(&reference); @@ -1112,13 +1112,13 @@ mod tests { let mut tone = Vec::with_capacity(frames * 2 * 4); for frame in 0..frames { let value = (std::f32::consts::TAU * 440.0 * frame as f32 / 48_000.0).sin() * 0.5; - for _ in 0..BUS_CHANNELS { + for _ in 0..REFERENCE_CHANNELS { tone.extend_from_slice(&value.to_le_bytes()); } } let mut energy = 0.0f64; - let mut buf = vec![0.0f32; REFERENCE_FRAME * BUS_CHANNELS]; + let mut buf = vec![0.0f32; REFERENCE_FRAME * REFERENCE_CHANNELS]; for _ in 0..20 { let _ = sink.write(&tone).expect("write"); diff --git a/rs/moq-audio/src/decode/consumer.rs b/rs/moq-audio/src/decode/consumer.rs index 0f7ee37fb66..5110db70e54 100644 --- a/rs/moq-audio/src/decode/consumer.rs +++ b/rs/moq-audio/src/decode/consumer.rs @@ -5,7 +5,7 @@ use std::collections::VecDeque; use bytes::Bytes; use super::decoder::{Config, Decoder}; -use crate::resample::{Resampler, remix, validate_remix}; +use crate::resample::{Remix, Resampler}; use crate::{Activity, Error, Format, Frame, Layout}; /// Where a consumer starts on a track that already holds groups. @@ -63,6 +63,8 @@ pub struct Consumer { decoder: Decoder, track: moq_mux::container::Consumer, resampler: Option, + /// Converts the decoded layout to the output's, when they differ. + remix: Option, options: Options, max_age: std::time::Duration, resolved_sample_rate: u32, @@ -111,7 +113,9 @@ impl Consumer { let decoder = Decoder::new(catalog, &options.decoder)?; let sample_rate = options.output.sample_rate.unwrap_or_else(|| decoder.sample_rate()); let layout = options.output.layout.unwrap_or_else(|| decoder.layout()); - validate_remix(decoder.layout(), layout)?; + let remix = (decoder.layout() != layout) + .then(|| Remix::new(decoder.layout(), layout)) + .transpose()?; let resampler = if sample_rate == decoder.sample_rate() { None @@ -165,6 +169,7 @@ impl Consumer { decoder, track, resampler, + remix, options, max_age, resolved_sample_rate: sample_rate, @@ -438,10 +443,9 @@ impl Consumer { /// Remix and pack decoded PCM into an output frame. fn frame(&self, pcm: Vec, timestamp: moq_net::Timestamp, activity: Activity) -> Result { - let pcm = if self.decoder.layout() == self.resolved_layout { - pcm - } else { - remix(&pcm, self.decoder.layout(), self.resolved_layout)? + let pcm = match &self.remix { + Some(remix) => remix.process(&pcm), + None => pcm, }; let bytes = self diff --git a/rs/moq-audio/src/encode/capture.rs b/rs/moq-audio/src/encode/capture.rs index e166e19f1a9..32d89909be2 100644 --- a/rs/moq-audio/src/encode/capture.rs +++ b/rs/moq-audio/src/encode/capture.rs @@ -12,7 +12,7 @@ use moq_mux::catalog::hang::CatalogExt; use super::producer::Reserved; use super::{Input, Options, Producer}; use crate::capture; -use crate::resample::{Resampler, remix, validate_remix}; +use crate::resample::{Remix, Resampler}; use crate::{Error, Format, Frame, Layout as PcmLayout}; /// Backoff bounds for reopening a capture source. The quick first retry covers @@ -445,7 +445,7 @@ impl Driver { None => continue, }, }; - let pcm_layout = match PcmLayout::from_channels(layout.channels) { + let pcm_layout = match device_layout(layout.channels) { Ok(layout) => layout, Err(err) => match self.failed(err, track, desired.revision).await { Some(result) => return Some(result), @@ -891,23 +891,29 @@ impl EncoderOutput<'_, E> { } } +/// The layout of a capture device, which reports only a channel count. That names +/// speakers for mono and stereo alone: a six-channel microphone array is not a +/// 5.1 speaker layout, so it stays discrete rather than being remixed as one. +fn device_layout(channels: u32) -> Result { + match channels { + 0..=2 => PcmLayout::from_channels(channels), + channels => Ok(PcmLayout::Discrete(channels)), + } +} + /// Converts one opened stream's native layout into the producer's fixed input /// layout. A new instance per open keeps filter state out of recovery gaps. struct Converter { - input: capture::Layout, - output: capture::Layout, + remix: Option, resampler: Option, anchor_us: Option, } impl Converter { fn new(input: capture::Layout, output: capture::Layout) -> Result { - if input.channels != output.channels { - validate_remix( - PcmLayout::from_channels(input.channels)?, - PcmLayout::from_channels(output.channels)?, - )?; - } + let remix = (input.channels != output.channels) + .then(|| Remix::new(device_layout(input.channels)?, device_layout(output.channels)?)) + .transpose()?; let resampler = if input.sample_rate == output.sample_rate { None @@ -924,8 +930,7 @@ impl Converter { }; Ok(Self { - input, - output, + remix, resampler, anchor_us: None, }) @@ -956,10 +961,8 @@ impl Converter { let data = resampler.process(&samples.data, moq_net::Timestamp::from_micros(timestamp_us)?)?; samples.replace(data); } - if self.input.channels != self.output.channels { - let input = PcmLayout::from_channels(self.input.channels)?; - let output = PcmLayout::from_channels(self.output.channels)?; - let data = remix(&samples.data, input, output)?; + if let Some(remix) = &self.remix { + let data = remix.process(&samples.data); samples.replace(data); } if samples.data.is_empty() { diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index ca2045a6333..90479471db4 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -883,8 +883,10 @@ mod tests { assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); } + /// The catalog carries only a count, so a discrete layout comes back as the + /// count's default one, with the samples untouched. #[test] - fn pcm_preserves_discrete_multichannel_layout() { + fn pcm_passes_discrete_multichannel_samples_through() { let settings = Settings { codec: Codec::Pcm, ..Settings::new(48_000, Layout::Discrete(3)) @@ -895,7 +897,7 @@ mod tests { let input = [0.1, 0.2, 0.3].repeat(encoder.frame_size()); let output = decoder.decode(&encoder.encode(&input).unwrap().payload).unwrap(); - assert_eq!(decoder.layout(), Layout::Discrete(3)); + assert_eq!(decoder.layout(), Layout::TwoPointOne); assert_eq!(output.samples, input); } diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index dcc3d1c9569..1b61545141b 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -10,7 +10,7 @@ use moq_net::Timestamp; use super::encoded::Encoded; use super::encoder::{Encoder, Input, Settings}; -use crate::resample::{Resampler, remix, validate_remix}; +use crate::resample::{Remix, Resampler}; use crate::{Activity, Error, Frame}; /// Encode and publication policy for [`Producer`]. @@ -64,6 +64,8 @@ impl Default for Options { pub struct Producer { encoder: Encoder, input: Input, + /// Converts the input layout to the codec's, when they differ. + remix: Option, resampler: Option, track: moq_mux::container::Producer, _ext: std::marker::PhantomData E>, @@ -140,7 +142,9 @@ impl Reserved { /// Separate from [`encode`](Self::encode), which cannot fail, so a layout the /// codec rejects leaves the reservation intact for another input. pub(crate) fn register(&mut self, input: Input, options: &Options) -> Result { - validate_remix(input.layout, options.settings.layout)?; + let remix = (input.layout != options.settings.layout) + .then(|| Remix::new(input.layout, options.settings.layout)) + .transpose()?; let encoder = Encoder::new(&options.settings)?; let resampler = if input.sample_rate == encoder.codec_rate() { @@ -163,6 +167,7 @@ impl Reserved { Ok(Registered { encoder, input, + remix, resampler, }) } @@ -172,6 +177,7 @@ impl Reserved { Producer { encoder: registered.encoder, input: registered.input, + remix: registered.remix, resampler: registered.resampler, track: self.track, _ext: self._ext, @@ -193,6 +199,7 @@ impl Reserved { pub(crate) struct Registered { encoder: Encoder, input: Input, + remix: Option, resampler: Option, } @@ -337,7 +344,10 @@ impl Producer { let input = &self.input; let (format, channels) = (input.format, input.layout.channels()); let pcm = format.as_interleaved_f32(frame.data.as_ref(), channels)?; - let pcm = remix(&pcm, input.layout, self.encoder.settings().layout)?; + let pcm = match &self.remix { + Some(remix) => remix.process(&pcm), + None => pcm.into_owned(), + }; let pcm: Vec = match self.resampler.as_mut() { Some(r) => r.process(&pcm, frame.timestamp)?, None => pcm, diff --git a/rs/moq-audio/src/layout.rs b/rs/moq-audio/src/layout.rs index 0ff4d0d09b1..90373ed715f 100644 --- a/rs/moq-audio/src/layout.rs +++ b/rs/moq-audio/src/layout.rs @@ -1,6 +1,11 @@ use crate::Error; /// Speaker meaning and interleaving order for PCM channels. +/// +/// Named layouts interleave in the SMPTE/WAVE order (front left, front right, +/// center, LFE, back left, back right, back center, side left, side right), +/// keeping only the speakers the layout has. Every decoder reorders its codec's +/// native order into this one. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] #[non_exhaustive] pub enum Layout { @@ -10,11 +15,47 @@ pub enum Layout { #[default] Stereo, /// Channels with no declared speaker positions, in source order. + /// + /// Passes through unchanged but can't be remixed into another layout. Discrete(u32), + /// 2.1: left, right, LFE. + TwoPointOne, + /// 3.0: left, right, center. + ThreePointZero, + /// Quad: left, right, back left, back right. + Quad, + /// 4.0: left, right, center, back center. + FourPointZero, + /// 5.0: left, right, center, side left, side right. + FivePointZero, + /// 5.1: left, right, center, LFE, side left, side right. + FivePointOne, + /// 6.1: left, right, center, LFE, back center, side left, side right. + SixPointOne, + /// 7.1: left, right, center, LFE, back left, back right, side left, side right. + SevenPointOne, +} + +/// One speaker position, declared in the canonical interleaving order. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Speaker { + FrontLeft, + FrontRight, + FrontCenter, + Lfe, + BackLeft, + BackRight, + BackCenter, + SideLeft, + SideRight, } impl Layout { - /// Infer today's conventional layout from a channel count. + /// The default layout for a channel count, by the WAVE convention: 1 is mono, + /// 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1. + /// + /// Any other count is [`Discrete`](Self::Discrete), since there is no + /// convention to take speaker positions from. pub fn from_channels(channels: u32) -> Result { match channels { 0 => Err(Error::Unsupported( @@ -22,6 +63,12 @@ impl Layout { )), 1 => Ok(Self::Mono), 2 => Ok(Self::Stereo), + 3 => Ok(Self::TwoPointOne), + 4 => Ok(Self::Quad), + 5 => Ok(Self::FivePointZero), + 6 => Ok(Self::FivePointOne), + 7 => Ok(Self::SixPointOne), + 8 => Ok(Self::SevenPointOne), channels => Ok(Self::Discrete(channels)), } } @@ -29,12 +76,40 @@ impl Layout { /// Number of interleaved channels in this layout. pub fn channels(self) -> u32 { match self { - Self::Mono => 1, - Self::Stereo => 2, Self::Discrete(channels) => channels, + layout => layout.speakers().map_or(0, <[_]>::len) as u32, } } + /// The speaker each channel feeds, in interleaved order, or `None` for a + /// discrete layout. + pub(crate) fn speakers(self) -> Option<&'static [Speaker]> { + use Speaker::*; + + Some(match self { + Self::Mono => &[FrontCenter], + Self::Stereo => &[FrontLeft, FrontRight], + Self::Discrete(_) => return None, + Self::TwoPointOne => &[FrontLeft, FrontRight, Lfe], + Self::ThreePointZero => &[FrontLeft, FrontRight, FrontCenter], + Self::Quad => &[FrontLeft, FrontRight, BackLeft, BackRight], + Self::FourPointZero => &[FrontLeft, FrontRight, FrontCenter, BackCenter], + Self::FivePointZero => &[FrontLeft, FrontRight, FrontCenter, SideLeft, SideRight], + Self::FivePointOne => &[FrontLeft, FrontRight, FrontCenter, Lfe, SideLeft, SideRight], + Self::SixPointOne => &[FrontLeft, FrontRight, FrontCenter, Lfe, BackCenter, SideLeft, SideRight], + Self::SevenPointOne => &[ + FrontLeft, + FrontRight, + FrontCenter, + Lfe, + BackLeft, + BackRight, + SideLeft, + SideRight, + ], + }) + } + pub(crate) fn validate(self) -> Result<(), Error> { if self.channels() == 0 { return Err(Error::Unsupported( @@ -44,3 +119,54 @@ impl Layout { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn counts_map_to_the_wave_defaults() { + let expected = [ + Layout::Mono, + Layout::Stereo, + Layout::TwoPointOne, + Layout::Quad, + Layout::FivePointZero, + Layout::FivePointOne, + Layout::SixPointOne, + Layout::SevenPointOne, + ]; + for (count, layout) in (1..).zip(expected) { + assert_eq!(Layout::from_channels(count).unwrap(), layout); + assert_eq!(layout.channels(), count); + } + + assert_eq!(Layout::from_channels(9).unwrap(), Layout::Discrete(9)); + assert!(Layout::from_channels(0).is_err()); + } + + /// Canonical order is the WAVE bit order, so every layout's speakers must + /// ascend through [`Speaker`]'s declaration order without repeating. + #[test] + fn speakers_follow_the_canonical_order() { + let layouts = [ + Layout::Mono, + Layout::Stereo, + Layout::TwoPointOne, + Layout::ThreePointZero, + Layout::Quad, + Layout::FourPointZero, + Layout::FivePointZero, + Layout::FivePointOne, + Layout::SixPointOne, + Layout::SevenPointOne, + ]; + for layout in layouts { + let speakers = layout.speakers().unwrap(); + assert!( + speakers.windows(2).all(|pair| (pair[0] as u8) < (pair[1] as u8)), + "{layout:?} is out of order" + ); + } + } +} diff --git a/rs/moq-audio/src/playback.rs b/rs/moq-audio/src/playback.rs index 49e81de3546..1f5e514a683 100644 --- a/rs/moq-audio/src/playback.rs +++ b/rs/moq-audio/src/playback.rs @@ -50,7 +50,7 @@ pub use sink::{Control, Input, Sink, Write}; #[cfg(feature = "aec")] pub(crate) use driver::Shared; #[cfg(feature = "aec")] -pub(crate) use mixer::BUS_CHANNELS; +pub(crate) use mixer::REFERENCE_CHANNELS; use crate::Error; @@ -137,13 +137,14 @@ impl Engine { /// Add a stream to the mix, taking PCM in the layout `input` describes. /// /// Independent of the device: several sinks can play at different rates and - /// channel counts, and each is resampled on its way to the mix. One device + /// layouts, and each is resampled and remixed to the device's on its way to + /// the mix. One device /// mixes up to 64 of them, past which this returns an error rather than /// handing back a sink that plays nothing. pub fn sink(&self, input: Input) -> Result { let sink = self .shared - .add(|id, rate| sink::new(id, rate, input, self.shared.clone(), self.handle.clone()))?; + .add(|id, rate, bus| sink::new(id, rate, bus, input, self.shared.clone(), self.handle.clone()))?; // Covers the case where the mixer's command queue was momentarily full, // so a sink is never left silently unmixed. diff --git a/rs/moq-audio/src/playback/device.rs b/rs/moq-audio/src/playback/device.rs index 8d55a28a42e..680e6a80f7b 100644 --- a/rs/moq-audio/src/playback/device.rs +++ b/rs/moq-audio/src/playback/device.rs @@ -11,10 +11,9 @@ use crate::Error; /// second choice. const RATES: &[u32] = &[48_000, 44_100]; -/// Channel counts to try, best first. The mixer produces stereo, and mono is -/// the only other count worth naming: anything else is a surround layout we -/// would be guessing the speaker order of. -const CHANNELS: &[u16] = &[2, 1]; +/// The widest channel count with a well-known layout. Wider counts name no +/// speaker positions to mix into, so they are never opened. +const MAX_CHANNELS: u16 = 8; /// Sample formats we can write, best first: `f32` is what the mixer produces, /// and the rest are conversions on the way out. @@ -129,16 +128,24 @@ pub(super) fn open(selector: Option<&str>) -> Result { /// Pick the stream format to open `device` with. /// -/// Only considers formats in [`FORMATS`], and prefers in that order: a channel -/// count in [`CHANNELS`], a rate the pipeline already runs at, then a format we -/// write without converting. Failing all of those it takes the highest rate the -/// device supports, since resampling down is kinder than resampling up. +/// Only considers formats in [`FORMATS`] and well-known layouts, and prefers in +/// that order: the device's own channel count, the widest layout, a rate the +/// pipeline already runs at, then a format we write without converting. Failing +/// the rates it takes the highest the device supports, since resampling down is +/// kinder than resampling up. pub(super) fn negotiate(device: &cpal::Device) -> Result { let supported = device .supported_output_configs() .map_err(|err| Error::Playback(format!("cannot enumerate output configs: {err}")))?; - choose(supported).ok_or_else(|| Error::Unsupported("output device offers no sample format we can write".into())) + // What the device is set up for: a 5.1 HDMI sink reports six channels and a + // headset two. Taken first because a sound server's plugin also accepts + // every count up to its limit, where the widest would upmix a headset. + let native = device.default_output_config().ok().map(|config| config.channels()); + + choose(supported, native).ok_or_else(|| { + Error::Unsupported("output device offers no sample format and speaker layout we can write".into()) + }) } /// Pick the best of the stream configurations a device reports. @@ -146,10 +153,14 @@ pub(super) fn negotiate(device: &cpal::Device) -> Result) -> Option { +fn choose( + supported: impl Iterator, + native: Option, +) -> Option { supported .filter(|config| FORMATS.contains(&config.sample_format())) - .min_by_key(preference) + .filter(|config| (1..=MAX_CHANNELS).contains(&config.channels())) + .min_by_key(|config| preference(config, native)) .map(|config| match preferred_rate(&config) { Some(rate) => config.try_with_sample_rate(rate).expect("a rate the range covers"), None => config.with_max_sample_rate(), @@ -164,11 +175,11 @@ fn choose(supported: impl Iterator) -> /// and a pass per channel count that gave up when neither preferred rate matched /// would hand that to the mono device and downmix. The channel count leads /// because a downmix is audible where a resample is not. -fn preference(config: &cpal::SupportedStreamConfigRange) -> (usize, usize, std::cmp::Reverse, usize) { - let channels = CHANNELS - .iter() - .position(|count| *count == config.channels()) - .unwrap_or(CHANNELS.len()); +fn preference( + config: &cpal::SupportedStreamConfigRange, + native: Option, +) -> ((bool, std::cmp::Reverse), usize, std::cmp::Reverse, usize) { + let channels = (native != Some(config.channels()), std::cmp::Reverse(config.channels())); let rate = match preferred_rate(config) { Some(rate) => (RATES.iter().position(|r| *r == rate).expect("from RATES"), rate), // Nothing we asked for, so take the most the device offers: resampling @@ -223,33 +234,72 @@ mod tests { /// the way to a stereo sink. #[test] fn stereo_wins_even_when_the_device_lists_mono_first() { - let chosen = choose([range(1, 48_000, SampleFormat::F32), range(2, 48_000, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(1, 48_000, SampleFormat::F32), range(2, 48_000, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!(chosen.channels(), 2); } /// Preferring stereo must not refuse a device that has no stereo to offer. #[test] fn mono_is_taken_when_that_is_all_there_is() { - let chosen = choose([range(1, 48_000, SampleFormat::F32)].into_iter()).expect("a config"); + let chosen = choose([range(1, 48_000, SampleFormat::F32)].into_iter(), None).expect("a config"); assert_eq!(chosen.channels(), 1); } - /// Neither preferred count is offered, so the pass that accepts any count - /// has to catch it. Without it a surround-only sink would not open at all. + /// A surround-only sink opens at its own width rather than being refused. #[test] - fn a_count_we_do_not_prefer_still_opens() { - let chosen = choose([range(6, 48_000, SampleFormat::F32)].into_iter()).expect("a config"); + fn a_surround_only_device_opens() { + let chosen = choose([range(6, 48_000, SampleFormat::F32)].into_iter(), None).expect("a config"); assert_eq!(chosen.channels(), 6); } + /// A 5.1 HDMI sink opens at six channels rather than being folded to stereo. + #[test] + fn six_channels_are_chosen_when_offered() { + let offered = || [range(2, 48_000, SampleFormat::F32), range(6, 48_000, SampleFormat::F32)].into_iter(); + assert_eq!(choose(offered(), None).expect("a config").channels(), 6); + assert_eq!(choose(offered(), Some(6)).expect("a config").channels(), 6); + } + + /// A sound server's plugin accepts every count up to its limit, so the + /// device's own count beats the widest: a headset stays at two rather than + /// being upmixed to 7.1 and folded back down by the server. + #[test] + fn the_native_count_beats_the_widest() { + let offered = (1..=8).map(|channels| range(channels, 48_000, SampleFormat::F32)); + assert_eq!(choose(offered, Some(2)).expect("a config").channels(), 2); + } + + /// Past 7.1 there is no convention naming the speakers, so there is nothing + /// to remix into. + #[test] + fn counts_without_a_layout_are_refused() { + assert!(choose([range(12, 48_000, SampleFormat::F32)].into_iter(), Some(12)).is_none()); + let chosen = choose( + [ + range(12, 48_000, SampleFormat::F32), + range(2, 48_000, SampleFormat::F32), + ] + .into_iter(), + Some(12), + ) + .expect("a config"); + assert_eq!(chosen.channels(), 2); + } + /// Rate is preferred within a channel count, not across one: a stereo config /// at an awkward rate beats a mono config at the pipeline's own rate, /// because resampling is inaudible and a downmix is not. #[test] fn channels_outrank_the_sample_rate() { - let chosen = choose([range(1, 48_000, SampleFormat::F32), range(2, 44_100, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(1, 48_000, SampleFormat::F32), range(2, 44_100, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!((chosen.channels(), chosen.sample_rate()), (2, 44_100)); } @@ -257,8 +307,11 @@ mod tests { /// produces rather than one that costs a conversion. #[test] fn f32_is_preferred_over_a_format_we_convert_to() { - let chosen = choose([range(2, 48_000, SampleFormat::I16), range(2, 48_000, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(2, 48_000, SampleFormat::I16), range(2, 48_000, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!(chosen.sample_format(), SampleFormat::F32); } @@ -266,8 +319,8 @@ mod tests { /// stream that plays noise. #[test] fn a_device_we_cannot_write_to_is_rejected() { - assert!(choose([range(2, 48_000, SampleFormat::I8)].into_iter()).is_none()); - assert!(choose(std::iter::empty()).is_none()); + assert!(choose([range(2, 48_000, SampleFormat::I8)].into_iter(), None).is_none()); + assert!(choose(std::iter::empty(), None).is_none()); } /// Channels lead even when neither preferred rate is on offer. @@ -278,8 +331,11 @@ mod tests { /// downmix the whole preference exists to avoid. #[test] fn stereo_at_an_awkward_rate_beats_mono_at_a_preferred_one() { - let chosen = choose([range(1, 48_000, SampleFormat::F32), range(2, 96_000, SampleFormat::F32)].into_iter()) - .expect("a config"); + let chosen = choose( + [range(1, 48_000, SampleFormat::F32), range(2, 96_000, SampleFormat::F32)].into_iter(), + None, + ) + .expect("a config"); assert_eq!((chosen.channels(), chosen.sample_rate()), (2, 96_000)); } @@ -293,6 +349,7 @@ mod tests { range(2, 96_000, SampleFormat::F32), ] .into_iter(), + None, ) .expect("a config"); assert_eq!((chosen.channels(), chosen.sample_rate()), (2, 96_000)); @@ -309,6 +366,7 @@ mod tests { range(2, 32_000, SampleFormat::F32), ] .into_iter(), + None, ) .expect("a config"); assert_eq!( diff --git a/rs/moq-audio/src/playback/driver.rs b/rs/moq-audio/src/playback/driver.rs index ce30e3688df..12bbd70fa5e 100644 --- a/rs/moq-audio/src/playback/driver.rs +++ b/rs/moq-audio/src/playback/driver.rs @@ -19,7 +19,7 @@ use rand::RngExt; use super::mixer::{self, Mixer}; use super::sink::{Registration, Sink}; -use crate::Error; +use crate::{Error, Layout}; /// Backoff bounds for reopening a device that failed. The first retry is quick because the common /// case is a device that came right back (a USB re-enumerate, a sample-rate change); the ceiling @@ -78,6 +78,9 @@ struct State { /// Rate the device is running at, which is what sinks resample to. Zero /// until the first stream opens. rate: u32, + /// Layout the device is running in, which is what sinks remix to. Stereo + /// stands in until the first stream opens. + layout: Layout, /// Registration channel to the live mixer, replaced every time the stream is /// rebuilt. `None` while no stream is running. mixer: Option>, @@ -108,13 +111,14 @@ struct State { impl Shared { /// Build a sink, register it, and start mixing it. /// - /// `build` is handed the sink's id and the rate its channel should target. + /// `build` is handed the sink's id and the rate and layout its channel should + /// target. /// It runs with no device open too: the registration waits for the next /// restart, so a device that is briefly missing doesn't become an error the /// caller has to retry. pub(super) fn add(&self, build: F) -> Result where - F: FnOnce(u64, u32) -> Result<(Sink, Registration), Error>, + F: FnOnce(u64, u32, Layout) -> Result<(Sink, Registration), Error>, { let mut state = self.state.lock().unwrap(); @@ -131,7 +135,7 @@ impl Shared { // 48 kHz stands in until a device opens and the channel is rebuilt at // the real rate. let rate = if state.rate == 0 { 48_000 } else { state.rate }; - let (sink, mut registration) = build(state.next_id, rate)?; + let (sink, mut registration) = build(state.next_id, rate, state.layout)?; state.next_id += 1; if let Some(mixer) = &state.mixer { @@ -270,11 +274,11 @@ impl Shared { } /// Point every sink at a freshly opened stream: rebuild each channel at - /// `rate` and hand the new consumers to `mixer`. - fn rebind(&self, rate: u32, mixer: SyncSender) { + /// `rate` and `layout` and hand the new consumers to `mixer`. + fn rebind(&self, rate: u32, layout: Layout, mixer: SyncSender) { let mut state = self.state.lock().unwrap(); for sink in &mut state.sinks { - sink.rebuild(rate); + sink.rebuild(rate, layout); sink.attach(&mixer); } @@ -289,6 +293,7 @@ impl Shared { } state.rate = rate; + state.layout = layout; state.mixer = Some(mixer); // The old mixer is gone, and with it every sink it was told about. state.detaching.clear(); @@ -787,7 +792,8 @@ impl Driver { // a full retirement channel is the one case where the mixer has to free // on the audio thread after all. let (retired_tx, retired_rx) = sync_channel(COMMAND_QUEUE); - let mixer = Mixer::new(rx, retired_tx, rate, channels); + let layout = Layout::from_channels(channels as u32)?; + let mixer = Mixer::new(rx, retired_tx, rate, layout)?; let failures = Arc::new(Failures::default()); let reporter = FailureReporter { @@ -799,7 +805,7 @@ impl Driver { .play() .map_err(|err| Error::Playback(format!("cannot start output stream: {err}")))?; - self.shared.rebind(rate, tx); + self.shared.rebind(rate, layout, tx); self.stream = Some(stream); self.failures = Some(failures); // Replaces the previous receiver, dropping anything the old stream @@ -807,7 +813,7 @@ impl Driver { self.retired = Some(retired_rx); self.retry = RETRY_MIN; - tracing::info!(rate, channels, ?format, "opened audio output"); + tracing::info!(rate, ?layout, ?format, "opened audio output"); Ok(()) } @@ -1005,7 +1011,7 @@ mod tests { let handle = Arc::new(super::super::Handle { commands }); let (tx, mixer) = sync_channel(depth); - shared.rebind(48_000, tx); + shared.rebind(48_000, Layout::Stereo, tx); Wired { shared, @@ -1016,7 +1022,7 @@ mod tests { } fn add(shared: &Arc, handle: &Arc) -> Result { - shared.add(|id, rate| sink::new(id, rate, Input::default(), shared.clone(), handle.clone())) + shared.add(|id, rate, bus| sink::new(id, rate, bus, Input::default(), shared.clone(), handle.clone())) } /// Long enough that only a lost wake, rather than a loaded machine, trips a diff --git a/rs/moq-audio/src/playback/mixer.rs b/rs/moq-audio/src/playback/mixer.rs index d3f23124e37..6187f2ca9ae 100644 --- a/rs/moq-audio/src/playback/mixer.rs +++ b/rs/moq-audio/src/playback/mixer.rs @@ -19,6 +19,10 @@ use fixed_resample::ResamplingCons; #[cfg(feature = "aec")] use fixed_resample::ResamplingProd; +#[cfg(feature = "aec")] +use crate::resample::Remix; +use crate::{Error, Layout}; + /// Frames mixed per pass. The callback buffer is chunked to this so the scratch /// buffers stay a fixed size no matter what period the device asks for. const CHUNK: usize = 1024; @@ -27,9 +31,10 @@ const CHUNK: usize = 1024; /// inaudible as a click, short enough to feel instant. const RAMP: f32 = 0.003; -/// The mix bus is stereo: sinks resample into it and it fans out to however many -/// channels the device wants. -pub(crate) const BUS_CHANNELS: usize = 2; +/// The echo reference is the mix downmixed to stereo, whatever layout the +/// device plays, so the canceller's model survives a device switch. +#[cfg(feature = "aec")] +pub(crate) const REFERENCE_CHANNELS: usize = 2; /// Sinks one device will mix. The entry list is allocated to this up front and /// never grows, which is what keeps registration off the allocator. @@ -138,36 +143,61 @@ pub(super) struct Mixer { /// Where anything the mixer is done with goes to be dropped, since dropping /// it here would free on the audio thread. retired: SyncSender, - /// Channels the device takes. + /// Channels the device takes, which is also the bus layout's: every sink + /// remixes into it on its way here. channels: usize, /// Per-frame gain step, so any change spans [`RAMP`] regardless of rate. step: f32, - /// Stereo accumulator for one chunk. + /// Accumulator for one chunk, in the device's layout. bus: Vec, - /// Stereo scratch for the sink being read. + /// Scratch for the sink being read, in the device's layout. scratch: Vec, - /// Where echo cancellation reads what was played. Fed the mix after - /// clipping but before it fans out, since that is the signal the speaker - /// gets and therefore the one the microphone hears back. + /// Where echo cancellation reads what was played. Fed the clipped mix, since + /// that is the signal the speaker gets and therefore the one the microphone + /// hears back. #[cfg(feature = "aec")] reference: Option>, + /// Downmixes the bus to the reference's stereo. + #[cfg(feature = "aec")] + downmix: Remix, + /// Stereo scratch for the downmixed reference. + #[cfg(feature = "aec")] + stereo: Vec, } impl Mixer { - /// `rate` and `channels` describe the device, not the sinks: each sink - /// resamples into the bus on its way here. - pub(super) fn new(commands: Receiver, retired: SyncSender, rate: u32, channels: usize) -> Self { - Self { + /// `rate` and `layout` describe the device, not the sinks: each sink + /// resamples and remixes into the bus on its way here. + pub(super) fn new( + commands: Receiver, + retired: SyncSender, + rate: u32, + layout: Layout, + ) -> Result { + // Sinks remix by speaker position, so a bus without any has nothing to + // remix into. + if layout.speakers().is_none() { + return Err(Error::Unsupported(format!( + "cannot mix into an output without speaker positions ({layout:?})" + ))); + } + + let channels = layout.channels() as usize; + Ok(Self { entries: Vec::with_capacity(MAX_SINKS), commands, retired, channels, step: 1.0 / (rate as f32 * RAMP), - bus: vec![0.0; CHUNK * BUS_CHANNELS], - scratch: vec![0.0; CHUNK * BUS_CHANNELS], + bus: vec![0.0; CHUNK * channels], + scratch: vec![0.0; CHUNK * channels], #[cfg(feature = "aec")] reference: None, - } + #[cfg(feature = "aec")] + downmix: Remix::new(layout, Layout::Stereo)?, + #[cfg(feature = "aec")] + stereo: vec![0.0; CHUNK * REFERENCE_CHANNELS], + }) } /// Hand something the mixer is done with back to the driver to drop. @@ -234,6 +264,10 @@ impl Mixer { scratch, #[cfg(feature = "aec")] reference, + #[cfg(feature = "aec")] + downmix, + #[cfg(feature = "aec")] + stereo, .. } = self; let channels = *channels; @@ -242,7 +276,7 @@ impl Mixer { let mut done = 0; while done < total { let frames = (total - done).min(CHUNK); - let samples = frames * BUS_CHANNELS; + let samples = frames * channels; bus[..samples].fill(0.0); @@ -257,13 +291,16 @@ impl Mixer { let mut applied = entry.applied; let mut peak = 0.0f32; - for frame in 0..frames { + for (input, mixed) in scratch[..samples] + .chunks_exact(channels) + .zip(bus[..samples].chunks_exact_mut(channels)) + { applied += (target - applied).clamp(-*step, *step); - let left = scratch[frame * 2] * applied; - let right = scratch[frame * 2 + 1] * applied; - peak = peak.max(left.abs()).max(right.abs()); - bus[frame * 2] += left; - bus[frame * 2 + 1] += right; + for (input, mixed) in input.iter().zip(mixed) { + let sample = input * applied; + peak = peak.max(sample.abs()); + *mixed += sample; + } } entry.applied = applied; @@ -280,26 +317,12 @@ impl Mixer { // useful to do about it from here and nothing may be logged. #[cfg(feature = "aec")] if let Some(reference) = reference.as_mut() { - reference.push_interleaved(&bus[..samples]); + let stereo = &mut stereo[..frames * REFERENCE_CHANNELS]; + downmix.apply(&bus[..samples], stereo); + reference.push_interleaved(stereo); } - let out = &mut out[done * channels..(done + frames) * channels]; - match channels { - 1 => { - for (frame, out) in out.iter_mut().enumerate() { - *out = (bus[frame * 2] + bus[frame * 2 + 1]) * 0.5; - } - } - _ => { - for (frame, out) in out.chunks_exact_mut(channels).enumerate() { - out[0] = bus[frame * 2]; - out[1] = bus[frame * 2 + 1]; - // Surround devices get silence past the front pair, which - // is better than duplicating stereo into the rears. - out[2..].fill(0.0); - } - } - } + out[done * channels..(done + frames) * channels].copy_from_slice(&bus[..samples]); done += frames; } @@ -316,6 +339,9 @@ mod tests { const RATE: u32 = 48_000; + /// The bus most tests mix into. + const STEREO: usize = 2; + /// Frames pushed per test: a tenth of a second, far more than any `fill` /// below drains, so a short read never starves by accident. const FRAMES: usize = RATE as usize / 10; @@ -328,21 +354,23 @@ mod tests { /// audio thread never drops one. retired: Receiver, next: u64, + channels: usize, } impl Harness { - fn new(channels: usize) -> Self { - Self::with_depth(channels, 8) + fn new(layout: Layout) -> Self { + Self::with_depth(layout, 8) } - fn with_depth(channels: usize, depth: usize) -> Self { + fn with_depth(layout: Layout, depth: usize) -> Self { let (commands, rx) = sync_channel(depth); let (retired_tx, retired) = sync_channel(MAX_SINKS); Self { - mixer: Mixer::new(rx, retired_tx, RATE, channels), + mixer: Mixer::new(rx, retired_tx, RATE, layout).unwrap(), commands, retired, next: 0, + channels: layout.channels() as usize, } } @@ -352,7 +380,7 @@ mod tests { /// pushed samples and the assertions. fn add(&mut self, gain: Arc) -> (u64, ResamplingProd) { let (prod, cons) = resampling_channel::( - BUS_CHANNELS, + self.channels, RATE, RATE, true, @@ -389,12 +417,12 @@ mod tests { /// Push `frames` stereo frames of a constant sample. fn push(prod: &mut ResamplingProd, value: f32, frames: usize) { - prod.push_interleaved(&vec![value; frames * BUS_CHANNELS]); + prod.push_interleaved(&vec![value; frames * STEREO]); } #[test] fn discards_writes_until_the_device_reads() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let (_, mut prod) = harness.add(Arc::new(Gain::new())); // Nothing has read yet, so these samples are dropped rather than queued @@ -411,7 +439,7 @@ mod tests { #[test] fn sums_sinks_and_clips() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; // Three sinks at 0.5 sum to 1.5, which must clip to 1.0. @@ -433,7 +461,7 @@ mod tests { #[test] fn volume_ramps_instead_of_stepping() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let gain = Arc::new(Gain::new()); @@ -453,8 +481,8 @@ mod tests { // The whole ramp is monotonic, not just its endpoints. let ramp = (RATE as f32 * RAMP) as usize; assert!(ramp < out.len() / 2, "test buffer is shorter than the ramp"); - for frame in out[..ramp * BUS_CHANNELS] - .as_chunks::() + for frame in out[..ramp * STEREO] + .as_chunks::() .0 .iter() .collect::>() @@ -484,7 +512,7 @@ mod tests { /// device buffer into NaN. #[test] fn output_stays_finite_after_a_non_finite_volume() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let gain = Arc::new(Gain::new()); @@ -500,7 +528,7 @@ mod tests { #[test] fn peak_reports_the_loudest_sample_then_resets() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let gain = Arc::new(Gain::new()); @@ -517,7 +545,7 @@ mod tests { #[test] fn removed_sinks_stop_mixing() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let (id, mut prod) = harness.add(Arc::new(Gain::new())); @@ -536,7 +564,7 @@ mod tests { /// callback would free on the audio thread, so it goes back to the driver. #[test] fn removed_sinks_are_handed_back_rather_than_dropped() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 512]; let (id, _prod) = harness.add(Arc::new(Gain::new())); @@ -561,7 +589,7 @@ mod tests { #[cfg(feature = "aec")] #[test] fn the_echo_reference_gets_the_mix() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 2048]; let (reference, mut tap) = reference_channel(); @@ -574,7 +602,7 @@ mod tests { // does, so that nothing queues up while nobody is listening. That first // read also primes the channel with its configured latency in silence, // which is why the buffer below is read past it rather than at it. - let mut heard = vec![0.0f32; 2048 * BUS_CHANNELS]; + let mut heard = vec![0.0f32; 2048 * REFERENCE_CHANNELS]; tap.read_interleaved(&mut heard, false); for prod in &mut prods { @@ -596,7 +624,7 @@ mod tests { #[cfg(feature = "aec")] #[test] fn a_replaced_echo_reference_is_handed_back() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 512]; harness @@ -629,7 +657,7 @@ mod tests { #[cfg(feature = "aec")] fn reference_channel() -> (ResamplingProd, ResamplingCons) { resampling_channel::( - BUS_CHANNELS, + REFERENCE_CHANNELS, RATE, RATE, true, @@ -645,7 +673,7 @@ mod tests { /// thread, which is why the driver caps registrations at MAX_SINKS. #[test] fn the_entry_list_never_grows() { - let mut harness = Harness::with_depth(2, MAX_SINKS); + let mut harness = Harness::with_depth(Layout::Stereo, MAX_SINKS); let mut out = vec![0.0f32; 512]; let capacity = harness.mixer.entries.capacity(); @@ -660,50 +688,65 @@ mod tests { #[test] fn silence_when_no_sink_is_registered() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![1.0f32; 512]; harness.fill(&mut out); assert!(out.iter().all(|s| *s == 0.0), "callback buffer was not overwritten"); } + /// The bus is the device's own layout, so every speaker of a surround device + /// carries what the sinks put there rather than silence past the front pair. #[test] - fn mono_device_gets_the_stereo_average() { - let mut harness = Harness::new(1); - let mut out = vec![0.0f32; 1024]; + fn a_surround_bus_mixes_every_speaker() { + let mut harness = Harness::new(Layout::FivePointOne); + let mut out = vec![0.0f32; 6 * 512]; let (_, mut prod) = harness.add(Arc::new(Gain::new())); harness.fill(&mut out); - - // Hard left, so a mono device should hear half of it. - let mut samples = vec![0.0f32; FRAMES * BUS_CHANNELS]; - for frame in samples.as_chunks_mut::().0.iter_mut() { - frame[0] = 1.0; - } - prod.push_interleaved(&samples); + let frame = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]; + prod.push_interleaved(&frame.repeat(FRAMES)); harness.settle(&mut out); - assert!((out[out.len() - 1] - 0.5).abs() < 1e-5, "got {}", out[out.len() - 1]); + + let last = &out[out.len() - 6..]; + for (got, want) in last.iter().zip(frame) { + assert!((got - want).abs() < 1e-5, "got {last:?}"); + } } + /// The canceller models a stereo reference whatever the device plays, so a + /// surround bus reaches it downmixed rather than cut to its front pair. + #[cfg(feature = "aec")] #[test] - fn surround_devices_get_silence_past_the_front_pair() { - let mut harness = Harness::new(6); - let mut out = vec![0.0f32; 6 * 512]; + fn the_echo_reference_downmixes_a_surround_bus() { + let mut harness = Harness::new(Layout::FivePointOne); + let mut out = vec![0.0f32; 6 * 2048]; + let (reference, mut tap) = reference_channel(); + harness.commands.send(Command::Reference(Some(reference))).unwrap(); let (_, mut prod) = harness.add(Arc::new(Gain::new())); harness.fill(&mut out); - push(&mut prod, 1.0, FRAMES); + let mut heard = vec![0.0f32; 2048 * REFERENCE_CHANNELS]; + tap.read_interleaved(&mut heard, false); + + // Center only, which a stereo downmix splits evenly at -3 dB. + prod.push_interleaved(&[0.0, 0.0, 0.5, 0.0, 0.0, 0.0].repeat(FRAMES)); harness.settle(&mut out); + tap.read_interleaved(&mut heard, false); - let last = &out[out.len() - 6..]; - assert!(last[0] > 0.9 && last[1] > 0.9, "front pair was silent"); - assert!(last[2..].iter().all(|s| *s == 0.0), "rear channels were not silent"); + let want = 0.5 * std::f32::consts::FRAC_1_SQRT_2; + let tail = &heard[heard.len() - 64..]; + assert!( + tail.iter().all(|s| (*s - want).abs() < 1e-5), + "the tap saw {:?}, not the downmix", + &tail[..4] + ); } #[test] fn underflow_reads_as_silence_rather_than_stale_samples() { - let mut harness = Harness::new(2); + let mut harness = Harness::new(Layout::Stereo); let mut out = vec![0.0f32; 8192]; let (_, mut prod) = harness.add(Arc::new(Gain::new())); diff --git a/rs/moq-audio/src/playback/sink.rs b/rs/moq-audio/src/playback/sink.rs index 41b8d0f79de..74bed20c192 100644 --- a/rs/moq-audio/src/playback/sink.rs +++ b/rs/moq-audio/src/playback/sink.rs @@ -1,6 +1,5 @@ //! [`Sink`]: one stream of PCM on its way to the speaker. -use std::borrow::Cow; use std::sync::mpsc::{SyncSender, TrySendError}; use std::sync::{Arc, Mutex}; use std::time::Duration; @@ -8,8 +7,8 @@ use std::time::Duration; use fixed_resample::{PushStatus, ResamplingChannelConfig, ResamplingCons, ResamplingProd, resampling_channel}; use super::driver::Shared; -use super::mixer::{self, BUS_CHANNELS, Gain}; -use crate::resample::remix; +use super::mixer::{self, Gain}; +use crate::resample::Remix; use crate::{Error, Format, Layout}; /// Default for [`Input::latency`]: audio buffered between [`Sink::write`] and @@ -28,7 +27,7 @@ const HEADROOM: f64 = 3.0; /// /// The playback counterpart to [`encode::Input`](crate::encode::Input): it /// describes the buffers you hand in, not the device, which is free to run at -/// its own rate and channel count. +/// its own rate and layout. /// /// `#[non_exhaustive]`: construct via [`Input::default`] and set fields, so new /// options can be added without breaking callers. @@ -40,7 +39,8 @@ pub struct Input { /// Samples per second per channel. Resampled to the device rate if they /// differ. pub sample_rate: u32, - /// Speaker meaning and channel order. + /// Speaker meaning and channel order. Remixed to the device's layout, so it + /// must name speaker positions rather than be [`Layout::Discrete`]. pub layout: Layout, /// How much audio to hold between [`Sink::write`] and the speaker (default: @@ -77,9 +77,9 @@ impl Input { if self.sample_rate == 0 { return Err(Error::Unsupported("sample rate must be > 0".into())); } - if !matches!(self.layout, Layout::Mono | Layout::Stereo) { + if self.layout.speakers().is_none() { return Err(Error::Unsupported(format!( - "playback accepts named mono or stereo input (got {:?})", + "playback needs speaker positions to remix (got {:?})", self.layout ))); } @@ -120,13 +120,13 @@ impl Write { /// One stream of PCM being played, mixed with every other sink on the device. /// /// Write decoded samples with [`write`](Self::write) and drop the sink to stop. -/// Writes are cheap and never block on the device: they hand samples to a ring -/// buffer that the audio thread drains on its own clock, resampling to the -/// device rate on the way. +/// Writes are cheap and never block on the device: they remix to the device's +/// layout and hand samples to a ring buffer that the audio thread drains on its +/// own clock, resampling to the device rate on the way. pub struct Sink { id: u64, input: Input, - prod: Arc>>, + channel: Arc>, control: Control, /// Whether the last write overflowed, so a writer that stays ahead of the /// device logs once rather than on every write. @@ -151,17 +151,21 @@ impl Sink { /// The returned [`Write`] counts input sample frames accepted and dropped; /// dropped live audio should be observed for telemetry, not retried. pub fn write(&mut self, samples: &[u8]) -> Result { - let pcm = self - .input - .format - .as_interleaved_f32(samples, self.input.layout.channels())?; - let pcm = match self.input.layout.channels() as usize { - BUS_CHANNELS => pcm, - _ => Cow::Owned(remix(&pcm, self.input.layout, Layout::Stereo)?), + let channels = self.input.layout.channels(); + let pcm = self.input.format.as_interleaved_f32(samples, channels)?; + let requested_sample_frames = pcm.len() / channels as usize; + + let mut channel = self.channel.lock().unwrap(); + let mixed; + let pcm = match &channel.remix { + Some(remix) => { + mixed = remix.process(&pcm); + &mixed + } + None => pcm.as_ref(), }; - let requested_sample_frames = pcm.len() / BUS_CHANNELS; - let accepted_sample_frames = match self.prod.lock().unwrap().push_interleaved(&pcm) { + let accepted_sample_frames = match channel.prod.push_interleaved(pcm) { // OutputNotReady means the device has not read yet, so these samples // are dropped rather than queued to play late. PushStatus::Ok => { @@ -201,7 +205,7 @@ impl Sink { /// climbs when the writer runs ahead, and falls toward zero when it falls /// behind. pub fn buffered(&self) -> Duration { - Duration::from_secs_f64(self.prod.lock().unwrap().occupied_seconds().max(0.0)) + Duration::from_secs_f64(self.channel.lock().unwrap().prod.occupied_seconds().max(0.0)) } /// The PCM layout this sink was built with. @@ -278,15 +282,21 @@ impl Control { } } +/// The ring into the mixer and the remix that fills it, swapped together when +/// the device changes rate or layout. +struct Channel { + prod: ResamplingProd, + /// Converts the sink's layout to the device's, when they differ. + remix: Option, +} + /// A sink as the driver sees it: enough to rebuild its channel when the device /// changes underneath it. pub(super) struct Registration { pub(super) id: u64, - /// The caller's rate, which is the input side of the channel. - rate: u32, - /// The depth the rebuilt channel has to keep, from the caller's [`Input`]. - latency: Duration, - prod: Arc>>, + /// The caller's rate, layout, and latency: the input side of the channel. + input: Input, + channel: Arc>, gain: Arc, /// The consumer waiting to be handed to a mixer. Taken once it is attached, /// and refilled by [`rebuild`](Self::rebuild). @@ -319,34 +329,36 @@ impl Registration { } } - /// Re-create the channel for a device now running at `rate`, swapping the - /// producer the caller's [`Sink`] writes into. - pub(super) fn rebuild(&mut self, rate: u32) { - let (prod, cons) = channel(self.rate, rate, self.latency); - *self.prod.lock().unwrap() = prod; + /// Re-create the channel for a device now running at `rate` in `bus`, + /// swapping the producer the caller's [`Sink`] writes into. + pub(super) fn rebuild(&mut self, rate: u32, bus: Layout) { + let (channel, cons) = channel(&self.input, rate, bus); + *self.channel.lock().unwrap() = channel; self.pending = Some(cons); } } /// Build a sink and its registration. The device may not be open yet, in which -/// case `rate` is a placeholder the driver replaces on the next rebuild. +/// case `rate` and `bus` are placeholders the driver replaces on the next +/// rebuild. pub(super) fn new( id: u64, rate: u32, + bus: Layout, input: Input, shared: Arc, engine: Arc, ) -> Result<(Sink, Registration), Error> { input.validate()?; - let (prod, cons) = channel(input.sample_rate, rate, input.latency); - let prod = Arc::new(Mutex::new(prod)); + let (channel, cons) = self::channel(&input, rate, bus); + let channel = Arc::new(Mutex::new(channel)); let gain = Arc::new(Gain::new()); let sink = Sink { id, input, - prod: prod.clone(), + channel: channel.clone(), control: Control { gain: gain.clone() }, overflowing: false, shared, @@ -355,9 +367,8 @@ pub(super) fn new( let registration = Registration { id, - rate: sink.input.sample_rate, - latency: sink.input.latency, - prod, + input: sink.input.clone(), + channel, gain, pending: Some(cons), }; @@ -365,14 +376,17 @@ pub(super) fn new( Ok((sink, registration)) } -/// The ring buffer between a writer and the audio thread, resampling the -/// caller's rate to the device's. -fn channel(from: u32, to: u32, latency: Duration) -> (ResamplingProd, ResamplingCons) { - let latency = latency.as_secs_f64(); - resampling_channel::( - BUS_CHANNELS, - from, - to, +/// The ring buffer between a writer and the audio thread, remixing the caller's +/// layout to the device's and resampling its rate to the device's. +fn channel(input: &Input, rate: u32, bus: Layout) -> (Channel, ResamplingCons) { + let remix = + (input.layout != bus).then(|| Remix::new(input.layout, bus).expect("sink and bus layouts name their speakers")); + + let latency = input.latency.as_secs_f64(); + let (prod, cons) = resampling_channel::( + bus.channels() as usize, + input.sample_rate, + rate, // We only ever push interleaved, which lets the channel skip its planar // staging buffer. true, @@ -385,7 +399,9 @@ fn channel(from: u32, to: u32, latency: Duration) -> (ResamplingProd, Resam overflow_autocorrect_percent_threshold: Some(75.0), ..Default::default() }, - ) + ); + + (Channel { prod, remix }, cons) } #[cfg(test)] @@ -393,20 +409,65 @@ mod tests { use super::*; fn sink(input: Input, output_rate: u32) -> (Sink, ResamplingCons) { + sink_into(input, output_rate, Layout::Stereo) + } + + fn sink_into(input: Input, output_rate: u32, bus: Layout) -> (Sink, ResamplingCons) { let shared = Arc::new(Shared::default()); let engine = Arc::new(super::super::Handle { commands: super::super::driver::Commands::default(), }); - let (sink, mut registration) = new(0, output_rate, input, shared, engine).unwrap(); + let (sink, mut registration) = new(0, output_rate, bus, input, shared, engine).unwrap(); (sink, registration.pending.take().unwrap()) } + /// Write `frame` repeated as `input` and read back what the bus got. + fn mix(input: Layout, bus: Layout, frame: &[f32]) -> Vec { + let input = Input { + layout: input, + ..Default::default() + }; + let (mut sink, mut cons) = sink_into(input, 48_000, bus); + let channels = bus.channels() as usize; + cons.read_interleaved(&mut vec![0.0; channels], false); + + let pcm: Vec = frame.repeat(4800).iter().flat_map(|s| s.to_le_bytes()).collect(); + assert_eq!(sink.write(&pcm).unwrap().dropped_sample_frames, 0); + + let mut out = vec![0.0; 4800 * channels]; + cons.read_interleaved(&mut out, false); + out[out.len() - channels..].to_vec() + } + + fn close(got: &[f32], want: &[f32]) { + assert_eq!(got.len(), want.len(), "{got:?} vs {want:?}"); + for (g, w) in got.iter().zip(want) { + assert!((g - w).abs() < 1e-4, "{got:?} vs {want:?}"); + } + } + + /// A 5.1 track on a stereo device plays its downmix, center and surrounds + /// folded into the front pair at -3 dB. + #[test] + fn a_surround_sink_downmixes_into_a_stereo_bus() { + let h = std::f32::consts::FRAC_1_SQRT_2; + let got = mix(Layout::FivePointOne, Layout::Stereo, &[0.1, 0.2, 0.3, 0.4, 0.05, 0.06]); + close(&got, &[0.1 + h * 0.3 + h * 0.05, 0.2 + h * 0.3 + h * 0.06]); + } + + /// A stereo track on a 5.1 device plays from the front pair alone. + #[test] + fn a_stereo_sink_fills_the_front_of_a_surround_bus() { + let got = mix(Layout::Stereo, Layout::FivePointOne, &[0.25, 0.75]); + close(&got, &[0.25, 0.75, 0.0, 0.0, 0.0, 0.0]); + } + fn s16(frames: usize, channels: usize) -> Vec { vec![0; frames * channels * 2] } fn ready(cons: &mut ResamplingCons) { - cons.read_interleaved(&mut [0.0; BUS_CHANNELS], false); + cons.read_interleaved(&mut [0.0; 2], false); } #[test] @@ -495,7 +556,7 @@ mod tests { #[test] fn rejects_layouts_it_cannot_mix() { - for layout in [Layout::Discrete(0), Layout::Discrete(6)] { + for layout in [Layout::Discrete(0), Layout::Discrete(2), Layout::Discrete(6)] { let input = Input { layout, ..Default::default() @@ -511,8 +572,13 @@ mod tests { } #[test] - fn accepts_mono_and_stereo() { - for layout in [Layout::Mono, Layout::Stereo] { + fn accepts_named_layouts() { + for layout in [ + Layout::Mono, + Layout::Stereo, + Layout::FivePointOne, + Layout::SevenPointOne, + ] { let input = Input { layout, ..Default::default() diff --git a/rs/moq-audio/src/resample.rs b/rs/moq-audio/src/resample.rs index 173a3b1ba27..402f0b85295 100644 --- a/rs/moq-audio/src/resample.rs +++ b/rs/moq-audio/src/resample.rs @@ -2,14 +2,17 @@ //! //! Wraps [`rubato`] with a small interleaved-`f32` interface so the //! producer/consumer doesn't have to convert to planar on every call. -//! The resampler keeps the channel layout unchanged; [`remix`] converts mono -//! and stereo after sample-rate conversion. +//! The resampler keeps the channel layout unchanged; [`Remix`] converts between +//! layouts after sample-rate conversion. + +use std::f32::consts::FRAC_1_SQRT_2; use rubato::audioadapter_buffers::direct::SequentialSliceOfVecs; use rubato::{ Async, FixedAsync, Resampler as RubatoTrait, SincInterpolationParameters, SincInterpolationType, WindowFunction, }; +use crate::layout::Speaker; use crate::{Error, Layout}; #[derive(Debug, thiserror::Error)] @@ -301,50 +304,147 @@ impl Resampler { } } -/// Convert between known layouts without assigning positions to discrete channels. -pub(crate) fn remix(samples: &[f32], input: Layout, output: Layout) -> Result, Error> { - validate_remix(input, output)?; - match (input, output) { - (input, output) if input == output => Ok(samples.to_vec()), - (Layout::Mono, Layout::Stereo) => { - let mut output = Vec::with_capacity(samples.len() * 2); - for &sample in samples { - output.extend_from_slice(&[sample, sample]); +/// A channel mix from one layout to another, each output channel a weighted sum +/// of the input channels. +/// +/// Refuses to give discrete channels speaker positions, though it will drop +/// them from named channels. +/// +/// Downmixing uses the ITU-R BS.775 coefficients: center and surrounds fold into +/// the front pair at -3 dB and the LFE is dropped. Upmixing leaves the speakers +/// the input lacks silent. Mono is the exception both ways: it plays at full +/// level from both front speakers when there is no center, and a mono output +/// averages the stereo downmix. +pub(crate) struct Remix { + inputs: usize, + outputs: usize, + /// One row of `inputs` weights per output channel. + weights: Vec, +} + +impl Remix { + pub(crate) fn new(input: Layout, output: Layout) -> Result { + input.validate()?; + output.validate()?; + + let (inputs, outputs) = (input.channels() as usize, output.channels() as usize); + // Dropping speaker positions is always safe; inventing them is not. + let unchanged = input == output || output == Layout::Discrete(inputs as u32); + let weights = match (input.speakers(), output.speakers()) { + _ if unchanged => (0..outputs) + .flat_map(|o| (0..inputs).map(move |i| if i == o { 1.0 } else { 0.0 })) + .collect(), + (Some(from), Some(to)) => weights(from, to), + _ => { + return Err(Error::Unsupported(format!( + "cannot convert audio layout {input:?} to {output:?} without speaker positions" + ))); + } + }; + + Ok(Self { + inputs, + outputs, + weights, + }) + } + + /// Mix whole interleaved input frames into `output`, which holds as many + /// frames. Never allocates, so the audio thread can call it. + pub(crate) fn apply(&self, input: &[f32], output: &mut [f32]) { + for (frame, out) in input + .chunks_exact(self.inputs) + .zip(output.chunks_exact_mut(self.outputs)) + { + for (sample, row) in out.iter_mut().zip(self.weights.chunks_exact(self.inputs)) { + *sample = row.iter().zip(frame).map(|(weight, input)| weight * input).sum(); } - Ok(output) } - (Layout::Stereo, Layout::Mono) => Ok(samples - .as_chunks::<2>() - .0 - .iter() - .map(|pair| (pair[0] + pair[1]) * 0.5) - .collect()), - _ => Err(Error::Unsupported(format!( - "cannot convert audio layout {input:?} to {output:?} without speaker positions" - ))), + } + + /// Mix whole interleaved input frames into a new buffer. + pub(crate) fn process(&self, input: &[f32]) -> Vec { + let mut output = vec![0.0; input.len() / self.inputs * self.outputs]; + self.apply(input, &mut output); + output } } -/// Check that [`remix`] can convert between two layouts. -pub(crate) fn validate_remix(input: Layout, output: Layout) -> Result<(), Error> { - input.validate()?; - output.validate()?; - if input == output - || matches!( - (input, output), - (Layout::Mono, Layout::Stereo) | (Layout::Stereo, Layout::Mono) - ) { - return Ok(()); - } - Err(Error::Unsupported(format!( - "cannot convert audio layout {input:?} to {output:?} without speaker positions" - ))) +/// The weights mixing `input` speakers into `output` speakers, one row per output. +fn weights(input: &[Speaker], output: &[Speaker]) -> Vec { + use Speaker::*; + + // The only layout without a front pair. Average the stereo downmix rather + // than invent a center weight for every speaker. + if output == [FrontCenter] && input != [FrontCenter] { + let stereo = weights(input, &[FrontLeft, FrontRight]); + let (left, right) = stereo.split_at(input.len()); + return left.iter().zip(right).map(|(l, r)| (l + r) * 0.5).collect(); + } + + let mut weights = vec![0.0; output.len() * input.len()]; + let has = |speaker| output.contains(&speaker); + + for (i, &speaker) in input.iter().enumerate() { + let mut feed = |to: Speaker, weight: f32| { + if let Some(o) = output.iter().position(|s| *s == to) { + weights[o * input.len() + i] += weight; + } + }; + + if has(speaker) { + feed(speaker, 1.0); + continue; + } + + match speaker { + FrontCenter => { + let weight = if input == [FrontCenter] { 1.0 } else { FRAC_1_SQRT_2 }; + feed(FrontLeft, weight); + feed(FrontRight, weight); + } + Lfe => {} + SideLeft | BackLeft | SideRight | BackRight => { + let (front, side, back) = match speaker { + SideLeft | BackLeft => (FrontLeft, SideLeft, BackLeft), + _ => (FrontRight, SideRight, BackRight), + }; + // Side and back are the same surround to a layout with only one of them. + let other = if speaker == side { back } else { side }; + if has(other) { + feed(other, 1.0); + } else { + feed(front, FRAC_1_SQRT_2); + } + } + BackCenter => { + if has(BackLeft) { + feed(BackLeft, FRAC_1_SQRT_2); + feed(BackRight, FRAC_1_SQRT_2); + } else if has(SideLeft) { + feed(SideLeft, FRAC_1_SQRT_2); + feed(SideRight, FRAC_1_SQRT_2); + } else { + feed(FrontLeft, 0.5); + feed(FrontRight, 0.5); + } + } + // Every output but mono, handled above, has a front pair. + FrontLeft | FrontRight => unreachable!("{output:?} has no front pair"), + } + } + + weights } #[cfg(test)] mod tests { use super::*; + fn remix(samples: &[f32], input: Layout, output: Layout) -> Result, Error> { + Ok(Remix::new(input, output)?.process(samples)) + } + /// `frames` into a stream at `rate`, as a timestamp in the source's own scale. fn at(frames: u64, rate: u64) -> moq_net::Timestamp { moq_net::Timestamp::from_scale(frames, rate).unwrap() @@ -548,4 +648,105 @@ mod tests { [2.0, 3.0] ); } + + /// One frame of 5.1 with a distinct level per speaker, in canonical order. + const FIVE_ONE: [f32; 6] = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]; + + fn close(got: &[f32], want: &[f32]) { + assert_eq!(got.len(), want.len(), "{got:?} vs {want:?}"); + for (g, w) in got.iter().zip(want) { + assert!((g - w).abs() < 1e-6, "{got:?} vs {want:?}"); + } + } + + #[test] + fn remix_downmixes_five_one_to_stereo_by_bs775() { + let h = FRAC_1_SQRT_2; + let [l, r, c, _lfe, ls, rs] = FIVE_ONE; + close( + &remix(&FIVE_ONE, Layout::FivePointOne, Layout::Stereo).unwrap(), + &[l + h * c + h * ls, r + h * c + h * rs], + ); + } + + #[test] + fn remix_downmixes_five_one_to_mono_through_stereo() { + let stereo = remix(&FIVE_ONE, Layout::FivePointOne, Layout::Stereo).unwrap(); + close( + &remix(&FIVE_ONE, Layout::FivePointOne, Layout::Mono).unwrap(), + &[(stereo[0] + stereo[1]) * 0.5], + ); + } + + #[test] + fn remix_upmixes_stereo_into_the_front_pair() { + close( + &remix(&[0.25, 0.75], Layout::Stereo, Layout::FivePointOne).unwrap(), + &[0.25, 0.75, 0.0, 0.0, 0.0, 0.0], + ); + } + + #[test] + fn remix_upmixes_mono_into_the_center() { + close( + &remix(&[0.5], Layout::Mono, Layout::FivePointOne).unwrap(), + &[0.0, 0.0, 0.5, 0.0, 0.0, 0.0], + ); + // Quad has no center, so mono plays from both fronts as it does in stereo. + close( + &remix(&[0.5], Layout::Mono, Layout::Quad).unwrap(), + &[0.5, 0.5, 0.0, 0.0], + ); + } + + /// 7.1 to 5.1 folds the back pair into the sides at full level, since a 5.1 + /// surround pair is the only surround it has. + #[test] + fn remix_folds_back_into_side_surrounds() { + let seven = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]; + close( + &remix(&seven, Layout::SevenPointOne, Layout::FivePointOne).unwrap(), + &[0.1, 0.2, 0.3, 0.4, 0.5 + 0.7, 0.6 + 0.8], + ); + } + + #[test] + fn remix_refuses_positions_it_would_invent() { + for (input, output) in [ + (Layout::Discrete(6), Layout::Stereo), + (Layout::Mono, Layout::Discrete(2)), + (Layout::Discrete(0), Layout::Discrete(0)), + ] { + assert!( + matches!(Remix::new(input, output), Err(Error::Unsupported(_))), + "{input:?} -> {output:?}" + ); + } + + assert_eq!( + remix(&[1.0, 2.0, 3.0], Layout::Discrete(3), Layout::Discrete(3)).unwrap(), + [1.0, 2.0, 3.0] + ); + assert_eq!( + remix(&[1.0, 2.0, 3.0], Layout::TwoPointOne, Layout::Discrete(3)).unwrap(), + [1.0, 2.0, 3.0] + ); + assert!(Remix::new(Layout::Discrete(3), Layout::TwoPointOne).is_err()); + } + + /// Every pair of named layouts converts, and the mix is sized to the output. + #[test] + fn remix_converts_between_every_named_layout() { + let layouts: Vec = (1..=8) + .map(|n| Layout::from_channels(n).unwrap()) + .chain([Layout::ThreePointZero, Layout::FourPointZero]) + .collect(); + for &input in &layouts { + for &output in &layouts { + let frame = vec![0.5; input.channels() as usize * 2]; + let mixed = remix(&frame, input, output).unwrap(); + assert_eq!(mixed.len(), output.channels() as usize * 2, "{input:?} -> {output:?}"); + } + } + } } diff --git a/rs/moq-audio/tests/roundtrip.rs b/rs/moq-audio/tests/roundtrip.rs index 570dacac45f..55e09bfab39 100644 --- a/rs/moq-audio/tests/roundtrip.rs +++ b/rs/moq-audio/tests/roundtrip.rs @@ -203,3 +203,71 @@ async fn pcm_round_trip_is_lossless() { assert_eq!(decoded, samples); assert!(consumer.read().await.unwrap().is_none()); } + +/// A 5.1 broadcast carries only its channel count in the catalog, which reads +/// back as 5.1 in canonical order and remixes to whatever the subscriber asks +/// for. +#[tokio::test] +async fn pcm_five_one_reads_back_at_any_layout() { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let mut catalog_consumer = catalog.consume().unwrap(); + let broadcast_consumer = broadcast.consume(); + + let input = encode::Input::new(48_000, Layout::FivePointOne); + let mut options = encode::Options::default(); + options.track = Some("pcm".to_string()); + options.settings = encode::Settings::new(48_000, Layout::FivePointOne); + options.settings.codec = encode::Codec::Pcm; + + // A level per speaker: left, right, center, LFE, side left, side right. + let speakers = [0.1f32, 0.2, 0.3, 0.4, 0.05, 0.06]; + let mut producer = encode::Producer::new(&mut broadcast, catalog.clone(), input, &options).unwrap(); + producer + .write(&Frame::new( + f32_bytes(&speakers.repeat(960)), + Timestamp::from_micros(0).unwrap(), + )) + .unwrap(); + + let snapshot = catalog_consumer.next().await.unwrap().unwrap(); + let rendition = snapshot.audio.renditions.get("pcm").unwrap(); + assert_eq!(rendition.channel_count, 6); + + let h = std::f32::consts::FRAC_1_SQRT_2; + let left = 0.1 + h * 0.3 + h * 0.05; + let right = 0.2 + h * 0.3 + h * 0.06; + let cases = [ + (None, Layout::FivePointOne, speakers.to_vec()), + (Some(Layout::Stereo), Layout::Stereo, vec![left, right]), + (Some(Layout::Mono), Layout::Mono, vec![(left + right) * 0.5]), + ]; + + let mut consumers = Vec::new(); + for (output, _, _) in &cases { + let mut options = decode::Options::default(); + options.output.layout = *output; + consumers.push( + decode::Consumer::new(&broadcast_consumer, rendition, "pcm", options) + .await + .unwrap(), + ); + } + producer.finish().unwrap(); + + for (mut consumer, (_, layout, frame)) in consumers.into_iter().zip(cases) { + assert_eq!(consumer.layout(), layout); + let decoded = consumer.read().await.unwrap().unwrap(); + let samples: Vec = decoded + .data + .as_chunks::<4>() + .0 + .iter() + .map(|sample| f32::from_le_bytes([sample[0], sample[1], sample[2], sample[3]])) + .collect(); + assert_eq!(samples.len(), 960 * frame.len(), "{layout:?}"); + for (got, want) in samples.iter().zip(frame.iter().cycle()) { + assert!((got - want).abs() < 1e-6, "{layout:?}: got {got}, want {want}"); + } + } +} diff --git a/rs/moq-ffi/src/audio.rs b/rs/moq-ffi/src/audio.rs index 16750c1375e..5e8bea78ce4 100644 --- a/rs/moq-ffi/src/audio.rs +++ b/rs/moq-ffi/src/audio.rs @@ -76,6 +76,9 @@ impl MoqAudioCodec { pub struct MoqAudioEncoderInput { pub format: MoqAudioSampleFormat, pub sample_rate: u32, + /// Interleaved channel count, which also names the speaker layout by the + /// WAVE convention: 1 mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, + /// 8 7.1, in front left, front right, center, LFE, back, side order. pub channels: u32, } @@ -105,7 +108,9 @@ pub struct MoqAudioDecoderOutput { /// `None` delivers samples at the codec's native rate. #[uniffi(default = None)] pub sample_rate: Option, - /// `None` delivers samples at the codec's native channel count. + /// `None` delivers samples at the codec's native channel count. A count + /// names its layout as [`MoqAudioEncoderInput::channels`] describes, and + /// the decoder remixes to it. #[uniffi(default = None)] pub channels: Option, /// Upper bound on buffering before skipping a stalled group, in From c1cdb920b72a889fc48b7972a23f225e4178bad3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 08:10:26 -0700 Subject: [PATCH 05/20] feat(audio): a decode backend seam that prefers the platform codec (#4131) Co-authored-by: Claude Opus 5.5 --- doc/bin/cli.md | 10 +- doc/lib/rs/moq-audio.md | 17 +- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/decode-audiotoolbox.md | 18 +- quest/m1/audio-codecs/decode-backend.md | 41 --- quest/m1/audio-codecs/encode-backend.md | 6 +- quest/m2/audio-decode-mediacodec.md | 4 - quest/m2/audio-decode-mediafoundation.md | 4 - rs/moq-audio/src/decode/backend/libopus.rs | 126 +++++++ rs/moq-audio/src/decode/backend/mod.rs | 303 +++++++++++++++++ rs/moq-audio/src/decode/backend/pcm.rs | 78 +++++ rs/moq-audio/src/decode/backend/symphonia.rs | 106 ++++++ rs/moq-audio/src/decode/consumer.rs | 5 + rs/moq-audio/src/decode/decoder.rs | 325 +++---------------- rs/moq-audio/src/decode/mod.rs | 5 +- rs/moq-cli/src/play/media.rs | 2 +- 16 files changed, 692 insertions(+), 359 deletions(-) delete mode 100644 quest/m1/audio-codecs/decode-backend.md create mode 100644 rs/moq-audio/src/decode/backend/libopus.rs create mode 100644 rs/moq-audio/src/decode/backend/mod.rs create mode 100644 rs/moq-audio/src/decode/backend/pcm.rs create mode 100644 rs/moq-audio/src/decode/backend/symphonia.rs diff --git a/doc/bin/cli.md b/doc/bin/cli.md index cbd4d9d03a4..723b80a3e46 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -87,11 +87,11 @@ moq --connect https://relay.example.com/anon --broadcast my-stream.hang play moq ... play --delay 500ms # trade latency for a jittery link ``` -Decodes H.264, H.265, and AV1 video and Opus, PCM, and AAC-LC audio using -the platform hardware decoder where available. `--video-name` and -`--audio-name` pick a rendition. -HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) plays as its -half-rate AAC-LC core. +Decodes H.264, H.265, and AV1 video using the platform hardware decoder where +available, and Opus, PCM, and AAC-LC (mono or stereo) audio in software. The +log names the decoder each track opened. `--video-name` and `--audio-name` +pick a rendition. HE-AAC signaled only in band (implicit SBR, as over MPEG-TS) +plays as its half-rate AAC-LC core. Playback runs on a clock it owns. `--delay` (default 100 ms) is how far it trails the live edge, which is both the jitter a late frame may absorb and the diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 2d57b4a5bc5..637343dd611 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -33,9 +33,20 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | -AAC decoding refuses HE-AAC its config declares. HE-AAC signaled only in band -(implicit SBR, as over MPEG-TS) goes undetected and plays as its half-rate -AAC-LC core. +`decode` picks a backend per track the way `moq-video` does: a platform decoder +first, then software. `decode::Config::kind` forces one (`Kind::Software`, or +`Kind::Named` with a name below), and `Decoder::name()` reports what opened. + +| Backend | Decodes | Hosts | +| --- | --- | --- | +| `libopus` | Opus, mono or stereo | all | +| `pcm` | PCM | all | +| `symphonia` | AAC-LC, mono or stereo (the default-on `aac` feature) | all | + +No platform decoder is wired in yet, so multichannel AAC and HE-AAC declared in +its config are refused at construction on every host. HE-AAC signaled only in +band plays as its half-rate LC core. Linux has no OS audio decoder, so it will +stay that way there. Highlights: diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 78912f392c9..9b3763d7f15 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,7 +44,6 @@ its own decode and encode quest so verification stays per host. ## Quests - [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - `decode::backend` selects a platform decoder before symphonia, mirroring moq-video - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC diff --git a/quest/m1/audio-codecs/decode-audiotoolbox.md b/quest/m1/audio-codecs/decode-audiotoolbox.md index e81db0d5543..cda8f73b5cc 100644 --- a/quest/m1/audio-codecs/decode-audiotoolbox.md +++ b/quest/m1/audio-codecs/decode-audiotoolbox.md @@ -10,9 +10,14 @@ documents as unsupported. ## Plan An `AudioConverter` from the packetized format to interleaved `f32` at the -codec's native rate and layout, behind the decode seam as the first platform -candidate on `target_os = "macos"` and `"ios"`. `objc2-audio-toolbox` is the -binding, alongside the `objc2-core-audio-types` the crate already carries. +codec's native rate and layout, behind the decode seam +(`rs/moq-audio/src/decode/backend`) as the first platform candidate on +`target_os = "macos"` and `"ios"`. `objc2-audio-toolbox` is the binding, +alongside the `objc2-core-audio-types` the crate already carries. + +- With a platform tier in place, `Auto` falling past a refusing platform + decoder to software should warn, as moq-video's `select` does; the seam + only aggregates the refusals into its error today. - Build the `AudioStreamBasicDescription` and magic cookie from the catalog description; the converter reports the output layout, which maps to @@ -20,7 +25,8 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. - HE-AAC: the converter reads SBR in band and reports the doubled rate; the seam passes it through. No config-level guessing. - Priming and remainder: AudioToolbox reports `kAudioConverterPrimeInfo`; - trim it so timestamps line up with symphonia's output on the same stream. + report it as the backend's startup delay, which the front end trims, so + timestamps line up with symphonia's output on the same stream. - Every codec the backend advertises has a fixture and a decode test, and the test asserts the layout order matches the canonical one (the LFE and centre end up where `Layout` says). @@ -29,10 +35,6 @@ binding, alongside the `objc2-core-audio-types` the crate already carries. - Docs: `doc/bin/obs.md` drops the HE-AAC and multichannel caveat on macOS, and the backend table names what this host decodes. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the same shape on Windows diff --git a/quest/m1/audio-codecs/decode-backend.md b/quest/m1/audio-codecs/decode-backend.md deleted file mode 100644 index 271ef84b526..00000000000 --- a/quest/m1/audio-codecs/decode-backend.md +++ /dev/null @@ -1,41 +0,0 @@ -# [M] A decode backend seam that prefers the platform codec - -## Goal - -`moq_audio::decode` selects a backend per codec the way `moq_video::decode` -does: platform first, software fallback, and a `Kind` to force one. Opus, -PCM, and symphonia AAC-LC become the software backends, and the crate -documents which codecs each host decodes. - -## Plan - -Mirror `rs/moq-video/src/decode/backend` in name and shape: a crate-private -`Backend` trait (`decode`, `flush`, `name`), an `open(codec, config)` that -walks the platform candidates before the software ones and refuses when none -takes the track. Use the backend-selection configuration and Decoder -constructor settled in main rather than adding a conflicting public shape. -`Decoder::name()` reports what was opened, which the OBS -stats and `moq play` surface. - -- The seam is generic over `hang::catalog::AudioCodec`, so a backend advertises - the set it opens and the selector asks each in order. Symphonia advertises - AAC-LC mono/stereo only; the platform backends that follow advertise what - their framework opens and has a fixture for. -- Move today's Opus, PCM, and symphonia code behind the trait without changing - behavior. Document per host that symphonia plays implicit-SBR HE-AAC as its - half-rate LC core: finding the in-band SBR element needs a full Huffman walk - of the channel elements, and symphonia detects it internally without - exposing or refusing it. -- A backend's output rate and layout are what it produced, not what the - catalog said (HE-AAC doubles the rate); `Consumer` already resamples and - remixes to the requested output, so that stays the seam's contract. -- The `aac` feature keeps gating symphonia. Platform backends are - `cfg(target_os)` like their video counterparts. Audio has no MediaCodec - feature yet; its Android quest introduces the optional dependency following - the settled media build policy. -- Docs: `doc/lib/rs/moq-audio.md` gains the backend table `moq-video.md` has, - and states the Linux gap. `doc/bin/cli.md` and `doc/bin/obs.md` follow. -- Regression: the selection order and `Named` refusal, tested with a stub - backend like the video seam's `probe`. - -The FFI does not expose `Kind` until a consumer asks. diff --git a/quest/m1/audio-codecs/encode-backend.md b/quest/m1/audio-codecs/encode-backend.md index 5c6dbc10738..4f73ed98f26 100644 --- a/quest/m1/audio-codecs/encode-backend.md +++ b/quest/m1/audio-codecs/encode-backend.md @@ -9,7 +9,7 @@ host with no AAC encoder refuses it at construction. ## Plan -Mirror the decode seam: `encode::backend` with a crate-private `Backend` +Mirror the decode seam (`rs/moq-audio/src/decode/backend`): `encode::backend` with a crate-private `Backend` trait (`encode`, `flush`, `set_bitrate`, `name`), an `open(codec, config)` that walks platform candidates before software ones, using the public settings and selection contract settled in main. Opus and PCM retain their behavior. This @@ -35,10 +35,6 @@ quest adds AAC through platform encoders; no software AAC dependency is selected - Regression: the selection order with a stub backend; `Codec::Aac` refused on a host with no backend; the Opus and PCM paths unchanged. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the naming and shape this mirrors - ## Related - [OBS audio publishing](/quest/m1/obs-moq-video/audio-publish.md) - the OBS encoder adapter can offer AAC once this lands diff --git a/quest/m2/audio-decode-mediacodec.md b/quest/m2/audio-decode-mediacodec.md index 5b7df87c71c..1446550bf03 100644 --- a/quest/m2/audio-decode-mediacodec.md +++ b/quest/m2/audio-decode-mediacodec.md @@ -23,10 +23,6 @@ behind a new optional audio `mediacodec` feature and the decode seam, on `target - The binding ships in the moq-ffi Android slice, which is how Kotlin and Dart reach it. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Android capture](/quest/m2/mobile-capture-android.md) - the video MediaCodec family this sits beside diff --git a/quest/m2/audio-decode-mediafoundation.md b/quest/m2/audio-decode-mediafoundation.md index 1da6a8337c0..37fc31fca75 100644 --- a/quest/m2/audio-decode-mediafoundation.md +++ b/quest/m2/audio-decode-mediafoundation.md @@ -22,10 +22,6 @@ ones. Behind the decode seam as the first candidate on `target_os = - Verification runs on a Windows host; the per-PR CI only compiles the platform code, and `just rs windows` runs nightly. -## Required - -- [Decode seam](/quest/m1/audio-codecs/decode-backend.md) - the candidate order this backend joins - ## Related - [Runtime QA hosts](/quest/m2/runtime-qa-hosts.md) - where the Windows run happens diff --git a/rs/moq-audio/src/decode/backend/libopus.rs b/rs/moq-audio/src/decode/backend/libopus.rs new file mode 100644 index 00000000000..0fa9f78b69d --- /dev/null +++ b/rs/moq-audio/src/decode/backend/libopus.rs @@ -0,0 +1,126 @@ +//! Opus through libopus, the software decoder for every host. + +use unsafe_libopus::{ + OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, + opus_decoder_destroy, varargs, +}; + +use super::Backend; +use crate::decode::Decoded; +use crate::{Error, Layout, opus}; + +pub(super) const NAME: &str = "libopus"; + +/// Opus packets cap at 120 ms (RFC 6716 §2.1.4). +const MAX_FRAME_MS: usize = 120; + +pub(super) struct Libopus { + inner: *mut OpusDecoder, + sample_rate: u32, + layout: Layout, + pre_skip: usize, + max_frame_size: usize, + in_dtx: bool, +} + +// SAFETY: the decoder is owned exclusively and libopus keeps no thread-local state. +unsafe impl Send for Libopus {} + +impl Libopus { + /// Parses the OpusHead `description` if present; falls back to the catalog's + /// declared sample rate / channel count. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { + let mut buf = desc.as_ref(); + match moq_mux::codec::opus::Config::parse(&mut buf) { + Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), + Err(_) => (catalog.sample_rate, catalog.channel_count, 0), + } + } else { + (catalog.sample_rate, catalog.channel_count, 0) + }; + + opus::validate_rate(sample_rate)?; + let channels = opus::validate_channels(channel_count)?; + let layout = Layout::from_channels(channel_count)?; + + let mut err = 0i32; + // SAFETY: out-pointer is valid; inner is checked for null below. + let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; + if err != OPUS_OK || inner.is_null() { + return Err(opus::error(err, "opus_decoder_create")); + } + + Ok(Box::new(Self { + inner, + sample_rate, + layout, + // OpusHead counts pre-skip at 48 kHz whatever rate the decoder runs at. + pre_skip: (pre_skip as usize * sample_rate as usize) / 48_000, + max_frame_size: (sample_rate as usize * MAX_FRAME_MS) / 1000, + in_dtx: false, + })) + } +} + +impl Backend for Libopus { + /// Empty packets invoke packet-loss concealment. Loss during DTX remains + /// classified as DTX, while loss during active audio remains active. + fn decode(&mut self, packet: &[u8]) -> Result { + let channels = self.layout.channels() as usize; + let mut out = vec![0.0f32; self.max_frame_size * channels]; + // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are bounded by + // the lengths we pass. + let samples = unsafe { + opus_decode_float( + &mut *self.inner, + packet.as_ptr(), + packet.len() as i32, + out.as_mut_ptr(), + self.max_frame_size as i32, + 0, + ) + }; + if samples < 0 { + return Err(opus::decode_error(samples)); + } + out.truncate(samples as usize * channels); + + let activity = opus::activity(packet, self.in_dtx); + self.in_dtx = activity.is_dtx(); + Ok(Decoded { samples: out, activity }) + } + + fn reset(&mut self) -> Result<(), Error> { + // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. + let rc = unsafe { opus_decoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, "OPUS_RESET_STATE")); + } + self.in_dtx = false; + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn delay(&self) -> usize { + self.pre_skip + } + + fn name(&self) -> &str { + NAME + } +} + +impl Drop for Libopus { + fn drop(&mut self) { + // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. + unsafe { opus_decoder_destroy(self.inner) }; + } +} diff --git a/rs/moq-audio/src/decode/backend/mod.rs b/rs/moq-audio/src/decode/backend/mod.rs new file mode 100644 index 00000000000..f39bb4c170c --- /dev/null +++ b/rs/moq-audio/src/decode/backend/mod.rs @@ -0,0 +1,303 @@ +//! Pluggable audio decoder backends. +//! +//! The audio mirror of `moq-video`'s decode backends. [`Backend`] is the seam +//! between the codec and the [`Decoder`](super::Decoder) front end, which owns +//! what every codec shares: trimming the startup delay a backend reports. +//! +//! [`open`] tries the platform decoders before the software ones, skipping any +//! that does not advertise the catalog codec, and refuses when none opens the +//! track. The software tier is libopus for Opus, a passthrough for PCM, and +//! symphonia for AAC-LC mono and stereo (behind the `aac` feature). No platform +//! decoder is wired in yet. + +use hang::catalog::{AudioCodec, AudioConfig}; + +use super::Decoded; +use super::decoder::{Config, Kind}; +use crate::{Error, Layout}; + +mod libopus; +mod pcm; +#[cfg(feature = "aac")] +mod symphonia; + +/// An opened decoder: packets in, interleaved `f32` PCM out. +pub(crate) trait Backend: Send { + /// Decode one packet into interleaved samples at [`sample_rate`](Self::sample_rate) + /// and [`layout`](Self::layout), untrimmed. + fn decode(&mut self, packet: &[u8]) -> Result; + + /// Drop codec history after a discontinuity, so the next packet does not + /// predict from audio that is no longer adjacent. + fn reset(&mut self) -> Result<(), Error>; + + /// The rate this backend decodes to, which may differ from the catalog's. + fn sample_rate(&self) -> u32; + + /// The layout this backend decodes to, which may differ from the catalog's. + fn layout(&self) -> Layout; + + /// Frames at the start of the stream that are codec priming, not media. + fn delay(&self) -> usize { + 0 + } + + /// The stable lowercase name [`Kind::Named`] selects this backend by. + fn name(&self) -> &str; +} + +type Open = fn(&AudioConfig) -> Result, Error>; + +/// A backend constructor: its name, the catalog codecs it advertises, and an opener. +struct Candidate { + name: &'static str, + supports: fn(&AudioCodec) -> bool, + open: Open, +} + +/// Operating-system decoders, in priority order. +const PLATFORM: &[Candidate] = &[]; + +const SOFTWARE: &[Candidate] = &[ + Candidate { + name: libopus::NAME, + supports: |codec| matches!(codec, AudioCodec::Opus), + open: libopus::Libopus::open, + }, + Candidate { + name: pcm::NAME, + supports: |codec| matches!(codec, AudioCodec::Pcm), + open: pcm::Pcm::open, + }, + // Claims every AAC profile and refuses at open what it can't decode: the + // profile that matters is the description's, which the catalog string can + // contradict. + #[cfg(feature = "aac")] + Candidate { + name: symphonia::NAME, + supports: |codec| matches!(codec, AudioCodec::AAC(_)), + open: symphonia::Symphonia::open, + }, +]; + +/// Open the first backend that advertises the catalog codec and accepts the track. +pub(crate) fn open(catalog: &AudioConfig, config: &Config) -> Result, Error> { + select(catalog, &config.kind, candidates(&config.kind, PLATFORM, SOFTWARE)) +} + +/// The candidates `kind` allows, in the order to try them. +/// +/// Takes the tiers as arguments so a test can supply stubs instead of whatever +/// this host compiles in. +fn candidates<'a>(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { + match kind { + Kind::Auto => platform.iter().chain(software).collect(), + Kind::Software => software.iter().collect(), + Kind::Named(name) => platform.iter().chain(software).filter(|c| c.name == name).collect(), + } +} + +fn select(catalog: &AudioConfig, kind: &Kind, candidates: Vec<&Candidate>) -> Result, Error> { + let codec = &catalog.codec; + let mut refused = Vec::new(); + + for candidate in candidates { + if !(candidate.supports)(codec) { + continue; + } + match (candidate.open)(catalog) { + Ok(backend) => return Ok(backend), + Err(err) => refused.push((candidate.name, err)), + } + } + + // One refusal is the whole answer, so keep its variant: a malformed + // description stays a container error rather than becoming a string. + if refused.len() == 1 { + let (_, err) = refused.remove(0); + return Err(err); + } + if !refused.is_empty() { + let reasons: Vec = refused.iter().map(|(name, err)| format!("{name}: {err}")).collect(); + return Err(Error::Unsupported(reasons.join(", "))); + } + + match kind { + Kind::Named(name) => { + let available: Vec<&str> = PLATFORM + .iter() + .chain(SOFTWARE) + .filter(|c| (c.supports)(codec)) + .map(|c| c.name) + .collect(); + Err(Error::Unsupported(format!( + "no audio decoder named {name:?} for {codec} (this build has: {})", + available.join(", ") + ))) + } + _ => Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::Activity; + + /// Opens anything it advertises and reports which candidate it came from. + struct Stub(&'static str); + + impl Backend for Stub { + fn decode(&mut self, _packet: &[u8]) -> Result { + Ok(Decoded { + samples: Vec::new(), + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + Ok(()) + } + + fn sample_rate(&self) -> u32 { + 48_000 + } + + fn layout(&self) -> Layout { + Layout::Stereo + } + + fn name(&self) -> &str { + self.0 + } + } + + const PLATFORM_STUB: Candidate = Candidate { + name: "platform", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Ok(Box::new(Stub("platform"))), + }; + + /// Compiled in but refusing the track, like a platform decoder asked for a + /// layout its framework does not open. + const REFUSING: Candidate = Candidate { + name: "refusing", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Err(Error::Unsupported("not this track".into())), + }; + + const SOFTWARE_STUB: Candidate = Candidate { + name: "software", + supports: |codec| matches!(codec, AudioCodec::Opus), + open: |_| Ok(Box::new(Stub("software"))), + }; + + /// Advertises nothing but PCM, so an Opus track never reaches its opener. + const PCM_ONLY: Candidate = Candidate { + name: "pcm-only", + supports: |codec| matches!(codec, AudioCodec::Pcm), + open: |_| panic!("opened for a codec it does not advertise"), + }; + + fn opus() -> AudioConfig { + AudioConfig::new(AudioCodec::Opus, 48_000, 2) + } + + fn pick(kind: Kind, platform: &[Candidate], software: &[Candidate]) -> Result { + let backend = select(&opus(), &kind, candidates(&kind, platform, software))?; + Ok(backend.name().to_owned()) + } + + #[test] + fn auto_prefers_platform() { + let name = pick(Kind::Auto, &[PCM_ONLY, PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); + } + + #[test] + fn auto_falls_back_to_software() { + let name = pick(Kind::Auto, &[REFUSING], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn software_skips_platform() { + let name = pick(Kind::Software, &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn named_forces_one() { + let name = pick( + Kind::Named("software".into()), + &[PLATFORM_STUB], + &[PCM_ONLY, SOFTWARE_STUB], + ) + .unwrap(); + assert_eq!(name, "software"); + } + + /// A named backend that refuses the track is the answer: nothing else is tried. + #[test] + fn named_refusal_does_not_fall_back() { + let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); + assert!(err.to_string().contains("not this track"), "{err}"); + } + + #[test] + fn every_refusal_is_reported() { + const ALSO_REFUSING: Candidate = Candidate { + name: "also-refusing", + ..REFUSING + }; + + let err = pick(Kind::Auto, &[REFUSING], &[ALSO_REFUSING]).unwrap_err(); + let message = err.to_string(); + assert!( + message.contains("refusing: ") && message.contains("also-refusing: "), + "{message}" + ); + } + + /// An unknown name says what this build has for the codec instead. + #[test] + fn unknown_name_lists_the_alternatives() { + let err = open( + &opus(), + &Config { + kind: Kind::Named("opus".into()), + }, + ) + .err() + .expect("no backend is named after its codec"); + let message = err.to_string(); + assert!( + message.contains("\"opus\"") && message.contains(libopus::NAME), + "{message}" + ); + } + + /// Asking for a real backend that does not decode the codec is refused, not + /// quietly swapped for one that does. + #[test] + fn named_backend_for_another_codec_is_refused() { + let config = Config { + kind: Kind::Named(pcm::NAME.into()), + }; + assert!(matches!(open(&opus(), &config), Err(Error::Unsupported(_)))); + } + + #[test] + fn software_backends_open_by_name() { + let pcm = AudioConfig::new(AudioCodec::Pcm, 48_000, 2); + let config = Config { + kind: Kind::Named(pcm::NAME.into()), + }; + assert_eq!(open(&pcm, &config).unwrap().name(), pcm::NAME); + + let config = Config { + kind: Kind::Named(libopus::NAME.into()), + }; + assert_eq!(open(&opus(), &config).unwrap().name(), libopus::NAME); + } +} diff --git a/rs/moq-audio/src/decode/backend/pcm.rs b/rs/moq-audio/src/decode/backend/pcm.rs new file mode 100644 index 00000000000..bead86fb5e0 --- /dev/null +++ b/rs/moq-audio/src/decode/backend/pcm.rs @@ -0,0 +1,78 @@ +//! Uncompressed little-endian `f32` PCM, described entirely by the catalog. + +use super::Backend; +use crate::decode::Decoded; +use crate::{Activity, Error, Layout, pcm}; + +pub(super) const NAME: &str = "pcm"; + +pub(super) struct Pcm { + sample_rate: u32, + layout: Layout, + bytes_per_frame: usize, +} + +impl Pcm { + /// Uses the catalog's rate and channel count, and requires an absent `description`. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + if catalog.sample_rate == 0 { + return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); + } + if catalog.channel_count == 0 { + return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); + } + if catalog.description.is_some() { + return Err(Error::Unsupported("pcm catalog description must be absent".into())); + } + let bitrate = pcm::bitrate(catalog.sample_rate, catalog.channel_count)?; + if catalog.bitrate.is_some_and(|declared| declared != bitrate) { + return Err(Error::Unsupported(format!( + "pcm catalog bitrate must be {bitrate} bits per second" + ))); + } + + Ok(Box::new(Self { + sample_rate: catalog.sample_rate, + layout: Layout::from_channels(catalog.channel_count)?, + bytes_per_frame: pcm::frame_bytes(1, catalog.channel_count)?, + })) + } +} + +impl Backend for Pcm { + fn decode(&mut self, packet: &[u8]) -> Result { + if packet.is_empty() || !packet.len().is_multiple_of(self.bytes_per_frame) { + return Err(Error::Misaligned { + got: packet.len(), + expected: packet.len().max(1).next_multiple_of(self.bytes_per_frame), + }); + } + + let samples = packet + .as_chunks::<{ pcm::BYTES_PER_SAMPLE }>() + .0 + .iter() + .map(|sample| f32::from_le_bytes(*sample)) + .collect(); + Ok(Decoded { + samples, + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/decode/backend/symphonia.rs b/rs/moq-audio/src/decode/backend/symphonia.rs new file mode 100644 index 00000000000..e2a87c6253d --- /dev/null +++ b/rs/moq-audio/src/decode/backend/symphonia.rs @@ -0,0 +1,106 @@ +//! AAC-LC through symphonia, the pure-Rust fallback for hosts without a +//! platform AAC decoder. + +use symphonia_core::codecs::audio::AudioDecoder; + +use super::Backend; +use crate::decode::Decoded; +use crate::{Activity, Error, Layout, aac}; + +pub(super) const NAME: &str = "symphonia"; + +pub(super) struct Symphonia { + inner: symphonia_codec_aac::AacDecoder, + sample_rate: u32, + layout: Layout, +} + +impl Symphonia { + /// AAC-LC in mono or stereo only, which is what every gateway that feeds + /// this crate publishes. + /// + /// HE-AAC is rejected however its config spells it: leading with SBR or PS + /// (mp4a.40.5 / .29), or leading with LC and declaring SBR in a sync extension + /// after the core. Symphonia decodes no SBR either way, so the alternative is + /// half-rate audio that sounds like a fault rather than an unsupported codec. + /// A stream that signals SBR only in band is indistinguishable from LC in the + /// config, and does decode as the core. + pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { + use symphonia_core::codecs::audio::well_known::CODEC_ID_AAC; + use symphonia_core::codecs::audio::{AudioCodecParameters, AudioDecoderOptions}; + + let hang::catalog::AudioCodec::AAC(codec) = &catalog.codec else { + return Err(Error::Unsupported(format!("symphonia cannot decode {}", catalog.codec))); + }; + let description = aac::description(catalog, codec.profile)?; + + let mut params = AudioCodecParameters::new(); + params + .for_codec(CODEC_ID_AAC) + .with_extra_data(description.to_vec().into_boxed_slice()); + + let inner = symphonia_codec_aac::AacDecoder::try_new(¶ms, &AudioDecoderOptions::default()) + .map_err(|err| Error::Unsupported(format!("aac decoder: {err}")))?; + + // Resolved by the decoder from the config, so this is what it will emit + // even when the catalog's own fields say otherwise. + let params = inner.codec_params(); + let sample_rate = params + .sample_rate + .ok_or_else(|| Error::Unsupported("aac config declares no sample rate".into()))?; + let channel_count = params + .channels + .as_ref() + .map(|channels| channels.count()) + .ok_or_else(|| Error::Unsupported("aac config declares no channels".into()))?; + + Ok(Box::new(Self { + inner, + sample_rate, + layout: Layout::from_channels(channel_count as u32)?, + })) + } +} + +impl Backend for Symphonia { + fn decode(&mut self, packet: &[u8]) -> Result { + // The packet is a raw AAC frame, not ADTS, so there is nothing to timestamp + // it with here: the container carries the timestamp and the decoder only + // reads the payload. + let packet = symphonia_core::packet::PacketRef::new( + 0, + symphonia_core::units::Timestamp::ZERO, + symphonia_core::units::Duration::ZERO, + packet, + ); + + let decoded = self + .inner + .decode_ref(&packet) + .map_err(|err| Error::Decode(format!("aac: {err}")))?; + + let mut samples = Vec::new(); + decoded.copy_to_vec_interleaved(&mut samples); + Ok(Decoded { + samples, + activity: Activity::Active, + }) + } + + fn reset(&mut self) -> Result<(), Error> { + self.inner.reset(); + Ok(()) + } + + fn sample_rate(&self) -> u32 { + self.sample_rate + } + + fn layout(&self) -> Layout { + self.layout + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/decode/consumer.rs b/rs/moq-audio/src/decode/consumer.rs index 5110db70e54..b5f8ccdce89 100644 --- a/rs/moq-audio/src/decode/consumer.rs +++ b/rs/moq-audio/src/decode/consumer.rs @@ -187,6 +187,11 @@ impl Consumer { }) } + /// The decoder backend name in use, e.g. `"libopus"` or `"symphonia"`. + pub fn name(&self) -> &str { + self.decoder.name() + } + /// The options this consumer was built with. pub fn options(&self) -> &Options { &self.options diff --git a/rs/moq-audio/src/decode/decoder.rs b/rs/moq-audio/src/decode/decoder.rs index 5f85d66c1cf..9cb9d7954f7 100644 --- a/rs/moq-audio/src/decode/decoder.rs +++ b/rs/moq-audio/src/decode/decoder.rs @@ -1,36 +1,24 @@ //! Audio decoder front end. //! -//! Mirror of [`encode::Encoder`](crate::encode::Encoder): dispatches over the -//! catalog codec and produces interleaved `f32` PCM. - -use unsafe_libopus::{ - OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, - opus_decoder_destroy, varargs, -}; - -#[cfg(feature = "aac")] -use symphonia_core::codecs::audio::AudioDecoder; +//! Mirror of [`encode::Encoder`](crate::encode::Encoder): opens a +//! [`Backend`](super::backend::Backend) for the catalog codec and trims its +//! startup delay, producing interleaved `f32` PCM. use super::Decoded; -#[cfg(feature = "aac")] -use crate::aac; -use crate::opus; -use crate::pcm; -use crate::{Activity, Error, Layout}; - -/// Opus packets cap at 120 ms (RFC 6716 §2.1.4). -const MAX_FRAME_MS: usize = 120; +use super::backend::{self, Backend}; +use crate::{Error, Layout}; /// Decoder backend selection. #[derive(Clone, Debug, Default, PartialEq, Eq)] #[non_exhaustive] pub enum Kind { - /// Pick the available backend automatically. + /// Prefer a platform decoder, falling back to software. #[default] Auto, - /// Require the built-in software backend. + /// Require a software backend. Software, - /// Require a backend by its stable lowercase name. + /// Require a backend by its stable lowercase name: `"libopus"`, `"pcm"`, or + /// `"symphonia"`. Named(String), } @@ -54,184 +42,42 @@ impl Config { /// The bring-your-own-payload layer under [`Consumer`](super::Consumer): use it /// when the packets don't come from a plain track subscription. pub struct Decoder { - backend: Backend, - sample_rate: u32, - layout: Layout, + backend: Box, + /// Startup delay in native-rate frames, and how much of it is left to trim. delay: usize, -} - -enum Backend { - Opus(Opus), - Pcm { - bytes_per_frame: usize, - }, - #[cfg(feature = "aac")] - Aac(Box), -} - -struct Opus { - inner: *mut OpusDecoder, - pre_skip_remaining: usize, - max_frame_size: usize, - in_dtx: bool, -} - -// SAFETY: see Encoder. -unsafe impl Send for Opus {} - -/// Boxed in [`Backend`]: the symphonia decoder carries its own filterbank state, -/// which is far larger than the other backends' handles. -#[cfg(feature = "aac")] -struct Aac { - inner: symphonia_codec_aac::AacDecoder, + delay_remaining: usize, } impl Decoder { /// Build a decoder from a catalog [`AudioConfig`](hang::catalog::AudioConfig). /// - /// Parses the OpusHead `description` if present; falls back to the catalog's - /// declared sample rate / channel count. PCM uses those catalog fields - /// directly and requires an absent `description`. + /// Opus parses the OpusHead `description` if present, falling back to the + /// catalog's declared sample rate and channel count. PCM uses those catalog + /// fields directly and requires an absent `description`. AAC reads its + /// AudioSpecificConfig, synthesizing one from the catalog when absent. pub fn new(catalog: &hang::catalog::AudioConfig, config: &Config) -> Result { - let name = match &catalog.codec { - hang::catalog::AudioCodec::Opus => "opus", - hang::catalog::AudioCodec::Pcm => "pcm", - #[cfg(feature = "aac")] - hang::catalog::AudioCodec::AAC(_) => "aac", - codec => return Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), - }; - match &config.kind { - Kind::Auto | Kind::Software => {} - Kind::Named(requested) if requested == name => {} - Kind::Named(requested) => { - return Err(Error::Unsupported(format!( - "audio decoder backend {requested:?} is unavailable for {name}" - ))); - } - } - match &catalog.codec { - hang::catalog::AudioCodec::Opus => Self::new_opus(catalog), - hang::catalog::AudioCodec::Pcm => Self::new_pcm(catalog), - #[cfg(feature = "aac")] - hang::catalog::AudioCodec::AAC(aac) => Self::new_aac(catalog, aac.profile), - codec => Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), - } - } - - fn new_opus(catalog: &hang::catalog::AudioConfig) -> Result { - let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { - let mut buf = desc.as_ref(); - match moq_mux::codec::opus::Config::parse(&mut buf) { - Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), - Err(_) => (catalog.sample_rate, catalog.channel_count, 0), - } - } else { - (catalog.sample_rate, catalog.channel_count, 0) - }; - - opus::validate_rate(sample_rate)?; - let channels = opus::validate_channels(channel_count)?; - - let mut err = 0i32; - // SAFETY: out-pointer is valid; inner is checked for null below. - let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; - if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_decoder_create")); - } - - let max_frame_size = (sample_rate as usize * MAX_FRAME_MS) / 1000; - let pre_skip_remaining = (pre_skip as usize * sample_rate as usize) / 48_000; - + let backend = backend::open(catalog, config)?; + let delay = backend.delay(); Ok(Self { - backend: Backend::Opus(Opus { - inner, - pre_skip_remaining, - max_frame_size, - in_dtx: false, - }), - sample_rate, - layout: Layout::from_channels(channel_count)?, - delay: pre_skip_remaining, + backend, + delay, + delay_remaining: delay, }) } - /// AAC-LC only, which is what every gateway that feeds this crate publishes. - /// - /// HE-AAC is rejected however its config spells it: leading with SBR or PS - /// (mp4a.40.5 / .29), or leading with LC and declaring SBR in a sync extension - /// after the core. Symphonia decodes no SBR either way, so the alternative is - /// half-rate audio that sounds like a fault rather than an unsupported codec. - /// A stream that signals SBR only in band is indistinguishable from LC in the - /// config, and does decode as the core. - #[cfg(feature = "aac")] - fn new_aac(catalog: &hang::catalog::AudioConfig, profile: u8) -> Result { - use symphonia_core::codecs::audio::well_known::CODEC_ID_AAC; - use symphonia_core::codecs::audio::{AudioCodecParameters, AudioDecoderOptions}; - - let description = aac::description(catalog, profile)?; - - let mut params = AudioCodecParameters::new(); - params - .for_codec(CODEC_ID_AAC) - .with_extra_data(description.to_vec().into_boxed_slice()); - - let inner = symphonia_codec_aac::AacDecoder::try_new(¶ms, &AudioDecoderOptions::default()) - .map_err(|err| Error::Unsupported(format!("aac decoder: {err}")))?; - - // Resolved by the decoder from the config, so this is what it will emit - // even when the catalog's own fields say otherwise. - let params = inner.codec_params(); - let sample_rate = params - .sample_rate - .ok_or_else(|| Error::Unsupported("aac config declares no sample rate".into()))?; - let channel_count = params - .channels - .as_ref() - .map(|channels| channels.count()) - .ok_or_else(|| Error::Unsupported("aac config declares no channels".into()))?; - - Ok(Self { - backend: Backend::Aac(Box::new(Aac { inner })), - sample_rate, - layout: Layout::from_channels(channel_count as u32)?, - delay: 0, - }) + /// The decoder backend name in use, e.g. `"libopus"` or `"symphonia"`. + pub fn name(&self) -> &str { + self.backend.name() } - fn new_pcm(catalog: &hang::catalog::AudioConfig) -> Result { - if catalog.sample_rate == 0 { - return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); - } - if catalog.channel_count == 0 { - return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); - } - if catalog.description.is_some() { - return Err(Error::Unsupported("pcm catalog description must be absent".into())); - } - let bitrate = pcm::bitrate(catalog.sample_rate, catalog.channel_count)?; - if catalog.bitrate.is_some_and(|declared| declared != bitrate) { - return Err(Error::Unsupported(format!( - "pcm catalog bitrate must be {bitrate} bits per second" - ))); - } - let bytes_per_frame = pcm::frame_bytes(1, catalog.channel_count)?; - - Ok(Self { - backend: Backend::Pcm { bytes_per_frame }, - sample_rate: catalog.sample_rate, - layout: Layout::from_channels(catalog.channel_count)?, - delay: 0, - }) - } - - /// The rate the codec decodes at, read from the catalog. + /// The rate the codec decodes at, which may differ from the catalog's. pub fn sample_rate(&self) -> u32 { - self.sample_rate + self.backend.sample_rate() } - /// The PCM layout decoded from the catalog. + /// The PCM layout the codec decodes to. pub fn layout(&self) -> Layout { - self.layout + self.backend.layout() } /// Reset codec history and reapply startup delay for a new discontinuous epoch. @@ -243,27 +89,12 @@ impl Decoder { /// Reapply catalog startup delay for a new playhead epoch without resetting codec prediction. pub(super) fn reapply_delay(&mut self) { - if let Backend::Opus(opus) = &mut self.backend { - opus.pre_skip_remaining = self.delay; - } + self.delay_remaining = self.delay; } /// Reset codec prediction after packet loss without reapplying stream startup delay. pub(super) fn reset_prediction(&mut self) -> Result<(), Error> { - match &mut self.backend { - Backend::Opus(opus) => { - // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_decoder_ctl_impl(opus.inner, OPUS_RESET_STATE, varargs![]) }; - if rc != OPUS_OK { - return Err(crate::opus::error(rc, "OPUS_RESET_STATE")); - } - opus.in_dtx = false; - } - Backend::Pcm { .. } => {} - #[cfg(feature = "aac")] - Backend::Aac(aac) => aac.inner.reset(), - } - Ok(()) + self.backend.reset() } /// How much startup delay is still to be trimmed, in native-rate frames. @@ -272,12 +103,7 @@ impl Decoder { /// it, so a caller tracking where a packet ends has to add back whatever this /// dropped across the call. pub(super) fn delay_remaining(&self) -> usize { - match &self.backend { - Backend::Opus(opus) => opus.pre_skip_remaining, - Backend::Pcm { .. } => 0, - #[cfg(feature = "aac")] - Backend::Aac(_) => 0, - } + self.delay_remaining } /// Decode one packet into interleaved `f32` PCM and report its codec activity. @@ -285,88 +111,14 @@ impl Decoder { /// Empty Opus packets invoke packet-loss concealment. Loss during DTX remains /// classified as DTX, while loss during active audio remains active. pub fn decode(&mut self, packet: &[u8]) -> Result { - match &mut self.backend { - Backend::Opus(opus) => { - let channels = self.layout.channels() as usize; - let mut out = vec![0.0f32; opus.max_frame_size * channels]; - // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are - // bounded by the lengths we pass. - let samples = unsafe { - opus_decode_float( - &mut *opus.inner, - packet.as_ptr(), - packet.len() as i32, - out.as_mut_ptr(), - opus.max_frame_size as i32, - 0, - ) - }; - if samples < 0 { - return Err(crate::opus::decode_error(samples)); - } - out.truncate(samples as usize * channels); - let trim_frames = opus.pre_skip_remaining.min(samples as usize); - if trim_frames > 0 { - let trim_samples = trim_frames * channels; - out.copy_within(trim_samples.., 0); - out.truncate(out.len() - trim_samples); - opus.pre_skip_remaining -= trim_frames; - } - let activity = crate::opus::activity(packet, opus.in_dtx); - opus.in_dtx = activity.is_dtx(); - Ok(Decoded { samples: out, activity }) - } - Backend::Pcm { bytes_per_frame } => { - if packet.is_empty() || !packet.len().is_multiple_of(*bytes_per_frame) { - return Err(Error::Misaligned { - got: packet.len(), - expected: packet.len().max(1).next_multiple_of(*bytes_per_frame), - }); - } - - let out = packet - .as_chunks::<{ pcm::BYTES_PER_SAMPLE }>() - .0 - .iter() - .map(|sample| f32::from_le_bytes([sample[0], sample[1], sample[2], sample[3]])) - .collect(); - Ok(Decoded { - samples: out, - activity: Activity::Active, - }) - } - #[cfg(feature = "aac")] - Backend::Aac(aac) => { - // The packet is a raw AAC frame, not ADTS, so there is nothing to - // timestamp it with here: the container carries the timestamp and the - // decoder only reads the payload. - let packet = symphonia_core::packet::PacketRef::new( - 0, - symphonia_core::units::Timestamp::ZERO, - symphonia_core::units::Duration::ZERO, - packet, - ); - - let decoded = aac - .inner - .decode_ref(&packet) - .map_err(|err| Error::Decode(format!("aac: {err}")))?; - - let mut out = Vec::new(); - decoded.copy_to_vec_interleaved(&mut out); - Ok(Decoded { - samples: out, - activity: Activity::Active, - }) - } + let mut decoded = self.backend.decode(packet)?; + let channels = self.backend.layout().channels() as usize; + let trim = self.delay_remaining.min(decoded.samples.len() / channels); + if trim > 0 { + decoded.samples.drain(..trim * channels); + self.delay_remaining -= trim; } - } -} - -impl Drop for Opus { - fn drop(&mut self) { - // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. - unsafe { opus_decoder_destroy(self.inner) }; + Ok(decoded) } } @@ -405,6 +157,7 @@ mod tests { #[test] fn aac_decodes_a_sine() { let mut decoder = Decoder::new(&aac_catalog(), &Config::default()).unwrap(); + assert_eq!(decoder.name(), "symphonia"); assert_eq!(decoder.sample_rate(), 44_100); assert_eq!(decoder.layout(), Layout::Mono); diff --git a/rs/moq-audio/src/decode/mod.rs b/rs/moq-audio/src/decode/mod.rs index 7e3cbd14a71..6f4500054c4 100644 --- a/rs/moq-audio/src/decode/mod.rs +++ b/rs/moq-audio/src/decode/mod.rs @@ -10,8 +10,11 @@ //! [`Decoded`] interleaved `f32` samples. //! //! [`Options`] keeps subscription and output policy separate from the -//! lower-level decoder [`Config`]. +//! lower-level decoder [`Config`], whose [`Kind`] picks the backend: a +//! platform decoder first where one takes the track, then software (libopus, +//! PCM, and symphonia for AAC-LC). +mod backend; mod consumer; mod decoded; mod decoder; diff --git a/rs/moq-cli/src/play/media.rs b/rs/moq-cli/src/play/media.rs index 0c12887e131..c47b3a9c3c7 100644 --- a/rs/moq-cli/src/play/media.rs +++ b/rs/moq-cli/src/play/media.rs @@ -224,7 +224,7 @@ impl Media { decode.output.format = moq_audio::Format::F32; match moq_audio::decode::Consumer::new(&rendition, &config, &name, decode).await { Ok(consumer) => { - tracing::info!(track = name, "playing audio rendition"); + tracing::info!(track = name, decoder = consumer.name(), "playing audio rendition"); if engine.is_none() { engine = Some(Engine::open(Default::default()).await?); } From 057bdec27fbea878b37f1d59ddcef3aae446aab5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 16:01:32 -0700 Subject: [PATCH 06/20] fix(mux): TS export carries an AAC track's program config element (#4178) Co-authored-by: Claude Opus 5.5 Co-authored-by: Grok 4.7 --- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/ts-export-pce.md | 17 --- rs/moq-ffi/src/binary.rs | 9 +- rs/moq-mux/src/codec/aac/mod.rs | 165 ++++++++++++++------- rs/moq-mux/src/container/ts/adts.rs | 10 +- rs/moq-mux/src/container/ts/export.rs | 39 +++-- rs/moq-mux/src/container/ts/export_test.rs | 59 ++++++++ rs/moq-mux/src/container/ts/import.rs | 8 +- 8 files changed, 215 insertions(+), 93 deletions(-) delete mode 100644 quest/m1/audio-codecs/ts-export-pce.md diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 9b3763d7f15..fdeca617059 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -43,7 +43,6 @@ its own decode and encode quest so verification stays per host. ## Quests -- [TS export PCE](/quest/m1/audio-codecs/ts-export-pce.md) - a TS export of a PCE-described AAC track writes channel_config 0 and the PCE instead of a count-derived config - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder - [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC diff --git a/quest/m1/audio-codecs/ts-export-pce.md b/quest/m1/audio-codecs/ts-export-pce.md deleted file mode 100644 index 31e9b76b251..00000000000 --- a/quest/m1/audio-codecs/ts-export-pce.md +++ /dev/null @@ -1,17 +0,0 @@ -# [S] TS export carries an AAC track's program config element - -## Goal - -`moq-mux`'s MPEG-TS export writes an AAC track described by a program config -element (channelConfiguration 0) as ADTS with channel_config 0 and the PCE at -the start of the first raw data block, so a TS round trip keeps the layout. -Today the export derives the ADTS channel_config from the channel count, which -mislabels such a track. - -## Plan - -Take the PCE from the track's AudioSpecificConfig description with the parser -that reads it on import, and write it as ffmpeg does: once, leading the first -raw data block. Tracks with a nonzero channelConfiguration are unchanged. Test -a round trip of the quad fixture from the PCE import (`aac_quad.ts`): import, -export, and import again, asserting the same description and channel count. diff --git a/rs/moq-ffi/src/binary.rs b/rs/moq-ffi/src/binary.rs index 8e343bd2ad4..cf9e4000943 100644 --- a/rs/moq-ffi/src/binary.rs +++ b/rs/moq-ffi/src/binary.rs @@ -8,6 +8,7 @@ use std::sync::Arc; +use moq_mux::binary::Config; use moq_mux::catalog::hang::Extra; use crate::error::MoqError; @@ -25,9 +26,9 @@ pub struct MoqBinaryConfig { pub mime: Option, } -impl From for moq_mux::binary::Config { +impl From for Config { fn from(config: MoqBinaryConfig) -> Self { - let mut out = moq_mux::binary::Config::default().with_compression(config.compression); + let mut out = Config::default().with_compression(config.compression); if let Some(mime) = config.mime { out = out.with_mime(mime); } @@ -48,7 +49,7 @@ impl MoqBroadcastProducer { let _guard = crate::ffi::enter(); self.with_state(|state| { let track = state.broadcast.create_track(name, None)?; - let producer = state.catalog.binary_snapshot(track, config.into())?; + let producer = state.catalog.binary_snapshot(track, Config::from(config))?; Ok(Arc::new(MoqBinarySnapshotProducer { inner: std::sync::Mutex::new(Some(producer)), })) @@ -66,7 +67,7 @@ impl MoqBroadcastProducer { let _guard = crate::ffi::enter(); self.with_state(|state| { let track = state.broadcast.create_track(name, None)?; - let producer = state.catalog.binary_stream(track, config.into())?; + let producer = state.catalog.binary_stream(track, Config::from(config))?; Ok(Arc::new(MoqBinaryStreamProducer { inner: std::sync::Mutex::new(Some(producer)), })) diff --git a/rs/moq-mux/src/codec/aac/mod.rs b/rs/moq-mux/src/codec/aac/mod.rs index 6d7f7cb67a5..5ccb24a1103 100644 --- a/rs/moq-mux/src/codec/aac/mod.rs +++ b/rs/moq-mux/src/codec/aac/mod.rs @@ -70,53 +70,10 @@ impl Config { /// that follows, and a reserved one is refused. Any SBR/PS extension bits after the /// core fields are consumed. pub fn parse(buf: &mut T) -> Result { - if buf.remaining() < 2 { - return Err(Error::ConfigTooShort); - } - let mut reader = BitReader::new(buf); - let object_type = read_object_type(&mut reader)?; - - // samplingFrequencyIndex: 4 bits; index 15 means an explicit 24-bit rate follows. - let freq_index = reader.read(4, Error::IncompleteConfig)? as u8; - let sample_rate = if freq_index == 15 { - reader.read(24, Error::ExplicitSampleRateTooShort)? - } else { - *SAMPLE_RATES - .get(freq_index as usize) - .ok_or(Error::UnsupportedSampleRateIndex(freq_index))? - }; - - // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. - let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; + let (object_type, sample_rate, channel_config) = read_header(&mut reader)?; let channel_count = match channel_config { - 0 => { - // Explicit SBR and PS name their core object type after an extension rate; the - // GASpecificConfig carrying the program config element follows that core type. - let mut core = object_type; - if matches!(object_type, 5 | 29) { - if reader.read(4, Error::IncompleteConfig)? == 15 { - reader.read(24, Error::IncompleteConfig)?; - } - core = read_object_type(&mut reader)?; - if core == 22 { - // extensionChannelConfiguration, only for ER BSAC. - reader.read(4, Error::IncompleteConfig)?; - } - } - if !GENERAL_AUDIO.contains(&core) { - return Err(Error::ProgramConfigUnsupported(core)); - } - - // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit - // coreCoderDelay), and extensionFlag precede the element. - reader.read(1, Error::IncompleteConfig)?; - if reader.read(1, Error::IncompleteConfig)? == 1 { - reader.read(14, Error::IncompleteConfig)?; - } - reader.read(1, Error::IncompleteConfig)?; - program_config(&mut reader)? - } + 0 => program_config(&mut reader)?, _ => channel_count_from_config(channel_config)?, }; @@ -178,13 +135,65 @@ impl Config { } } +/// Read an AudioSpecificConfig up to its channels: the audioObjectType, sample rate, and +/// channelConfiguration. For a channelConfiguration of 0 the reader stops at the program config +/// element. +fn read_header(reader: &mut BitReader) -> Result<(u8, u32, u8)> { + if reader.buf.remaining() < 2 { + return Err(Error::ConfigTooShort); + } + let object_type = read_object_type(reader)?; + + // samplingFrequencyIndex: 4 bits; index 15 means an explicit 24-bit rate follows. + let freq_index = reader.read(4, Error::IncompleteConfig)? as u8; + let sample_rate = if freq_index == 15 { + reader.read(24, Error::ExplicitSampleRateTooShort)? + } else { + *SAMPLE_RATES + .get(freq_index as usize) + .ok_or(Error::UnsupportedSampleRateIndex(freq_index))? + }; + + // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. + let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; + if channel_config != 0 { + return Ok((object_type, sample_rate, channel_config)); + } + + // Explicit SBR and PS name their core object type after an extension rate; the + // GASpecificConfig carrying the program config element follows that core type. + let mut core = object_type; + if matches!(object_type, 5 | 29) { + if reader.read(4, Error::IncompleteConfig)? == 15 { + reader.read(24, Error::IncompleteConfig)?; + } + core = read_object_type(reader)?; + if core == 22 { + // extensionChannelConfiguration, only for ER BSAC. + reader.read(4, Error::IncompleteConfig)?; + } + } + if !GENERAL_AUDIO.contains(&core) { + return Err(Error::ProgramConfigUnsupported(core)); + } + + // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit + // coreCoderDelay), and extensionFlag precede the element. + reader.read(1, Error::IncompleteConfig)?; + if reader.read(1, Error::IncompleteConfig)? == 1 { + reader.read(14, Error::IncompleteConfig)?; + } + reader.read(1, Error::IncompleteConfig)?; + Ok((object_type, sample_rate, 0)) +} + /// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does. /// /// A `channel_config` of 0 means a program config element leads `block`, the stream's first raw -/// data block; it moves into the config, so the config describes the channels on its own. Any -/// other value leaves `block` unread. An element placed anywhere else is refused: reaching past -/// the channel data needs a full Huffman decode. -pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, block: &[u8]) -> Result { +/// data block; it moves into the config, so the config describes the channels on its own, and +/// `block` advances past it. Any other value leaves `block` untouched. An element placed anywhere +/// else is refused: reaching past the channel data needs a full Huffman decode. +pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, block: &mut &[u8]) -> Result { let mut out = BitWriter::default(); out.write(5, u32::from(profile & 0x1F)); match SAMPLE_RATES.iter().position(|&rate| rate == sample_rate) { @@ -200,8 +209,8 @@ pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder, and extensionFlag, all clear. out.write(3, 0); - let mut block = block; - let mut reader = BitReader::new(&mut block); + // The element ends byte aligned to the block, so the reader stops exactly past it. + let mut reader = BitReader::new(block); if reader.read(3, Error::ProgramConfigMissing)? != ID_PCE { return Err(Error::ProgramConfigMissing); } @@ -213,6 +222,27 @@ pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, Ok(Bytes::from(out.bytes)) } +/// Split an AudioSpecificConfig into the channel signaling of a stream that carries it per frame, +/// as ADTS does; the inverse of [`in_band_config`]. +/// +/// Returns the channelConfiguration and, when that is 0, the program config element framed to +/// lead a raw data block: its element ID, then the element aligned to the block. +pub(crate) fn in_band_channels(asc: &[u8]) -> Result<(u8, Option)> { + let mut asc = asc; + let mut reader = BitReader::new(&mut asc); + let (_, _, channel_config) = read_header(&mut reader)?; + if channel_config != 0 { + return Ok((channel_config, None)); + } + + let mut out = BitWriter::default(); + out.write(3, ID_PCE); + reader.record = Some(out); + program_config(&mut reader)?; + let out = reader.record.take().expect("recording set above"); + Ok((0, Some(Bytes::from(out.bytes)))) +} + /// The raw data block element ID of a program config element (ISO 14496-3 Table 4.85). const ID_PCE: u32 = 5; @@ -597,23 +627,48 @@ mod tests { let mut block = BitWriter::default(); block.write(3, ID_PCE); write_pce(&mut block, &[false, true], &[], &[true], 1); + let pce = block.bytes.clone(); // The channel elements that follow are never read. block.write(8, 0xFF); - let asc = in_band_config(2, 48_000, 0, &block.bytes).unwrap(); + let mut rest = block.bytes.as_slice(); + let asc = in_band_config(2, 48_000, 0, &mut rest).unwrap(); assert_eq!(asc, pce_asc(2, &[false, true], &[], &[true], 1)); assert_eq!(Config::parse(&mut asc.as_ref()).unwrap().channel_count, 6); + assert_eq!(rest, [0xFF], "the element moves out of the block"); + + // And back: the element leads the block again, byte for byte. + assert_eq!(in_band_channels(&asc).unwrap(), (0, Some(Bytes::from(pce)))); + } + + #[test] + fn in_band_channels_of_ffmpeg_program_config_element() { + // Round trip ffmpeg's own element through a raw data block and back into a config. The + // trailing SBR sync extension is not part of the element, so it drops. + let (config, pce) = in_band_channels(&FFMPEG_QUAD_ASC).unwrap(); + assert_eq!(config, 0); + let pce = pce.unwrap(); + let mut block = pce.as_ref(); + let asc = in_band_config(2, 48_000, 0, &mut block).unwrap(); + assert!(block.is_empty(), "the element is all that was framed"); + assert_eq!(asc, FFMPEG_QUAD_ASC[..FFMPEG_QUAD_ASC.len() - 3]); + } + + #[test] + fn in_band_channels_without_a_program_config_element() { + assert_eq!(in_band_channels(&[0x11, 0x90]).unwrap(), (2, None)); + assert_eq!(in_band_channels(&[0x11, 0xE0]).unwrap(), (12, None)); } #[test] fn in_band_config_refuses_a_block_without_a_leading_pce() { // ID_CPE first: any element past the channel data is out of reach. assert!(matches!( - in_band_config(2, 48_000, 0, &[0x20, 0x00]), + in_band_config(2, 48_000, 0, &mut [0x20, 0x00].as_slice()), Err(Error::ProgramConfigMissing) )); assert!(matches!( - in_band_config(2, 48_000, 0, &[]), + in_band_config(2, 48_000, 0, &mut [].as_slice()), Err(Error::ProgramConfigMissing) )); } @@ -621,7 +676,7 @@ mod tests { #[test] fn in_band_config_matches_encode() { // Without a PCE the block is unread and the config is the plain two-byte form. - let asc = in_band_config(2, 44_100, 2, &[]).unwrap(); + let asc = in_band_config(2, 44_100, 2, &mut [].as_slice()).unwrap(); let encoded = Config { profile: 2, sample_rate: 44_100, diff --git a/rs/moq-mux/src/container/ts/adts.rs b/rs/moq-mux/src/container/ts/adts.rs index 0166c059372..ac1aec68232 100644 --- a/rs/moq-mux/src/container/ts/adts.rs +++ b/rs/moq-mux/src/container/ts/adts.rs @@ -63,13 +63,17 @@ impl Header { pub(super) fn write_header( object_type: u8, sample_rate: u32, - channel_count: u32, + channel_config: u8, raw_len: usize, ) -> anyhow::Result<[u8; 7]> { // ADTS `profile` is the 2-bit audioObjectType - 1. let profile = object_type.saturating_sub(1) & 0x03; let freq_index = freq_index_from_rate(sample_rate)?; - let channel_config = channel_config_from_count(channel_count); + // ADTS has 3 bits for it; the higher configurations only fit an AudioSpecificConfig. + anyhow::ensure!( + channel_config < 8, + "channelConfiguration {channel_config} not representable in ADTS" + ); let frame_len = raw_len + 7; anyhow::ensure!(frame_len < (1 << 13), "AAC frame too large for ADTS framing"); @@ -98,7 +102,7 @@ fn freq_index_from_rate(sample_rate: u32) -> anyhow::Result { } /// Map a channel count to an AAC `channel_config` (ISO 14496-3 Table 1.19). -fn channel_config_from_count(channel_count: u32) -> u8 { +pub(super) fn channel_config_from_count(channel_count: u32) -> u8 { match channel_count { 1..=6 => channel_count as u8, 8 => 7, diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index e1380502ae8..c53a7c141f5 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -254,10 +254,14 @@ impl Track { enum Kind { /// Video carries its TS stream type (H.264 = 0x1B, H.265 = 0x24). Video(StreamType), + /// AAC, framed as ADTS. A `channel_config` of 0 defers the layout to a program config + /// element, which leads the next raw data block written and is then taken. A catalog update + /// rebuilds the kind and so repeats it once, which a decoder tuning in mid-stream welcomes. Aac { object_type: u8, sample_rate: u32, - channel_count: u32, + channel_config: u8, + program_config: Option, }, /// Opus (private stream_type 0x06). Each frame is one Opus packet, prefixed with /// the Opus-in-TS access-unit control header and announced with the 'Opus' @@ -1341,9 +1345,12 @@ impl Export { if self.span_counters.is_none() { self.span_counters = Some(self.counters.clone()); } - let track = self.tracks.get(name).context("missing track")?; + let track = self.tracks.get_mut(name).context("missing track")?; let pid = track.pid; let kind = track.kind.clone(); + if let Kind::Aac { program_config, .. } = &mut track.kind { + program_config.take(); + } let is_video = matches!(kind, Kind::Video(_)); let timestamp = frame.timestamp; let keyframe = frame.keyframe; @@ -1356,11 +1363,15 @@ impl Export { Kind::Aac { object_type, sample_rate, - channel_count, + channel_config, + program_config, } => { - let header = adts::write_header(*object_type, *sample_rate, *channel_count, frame.payload.len())?; - let mut framed = Vec::with_capacity(7 + frame.payload.len()); + let pce = program_config.as_deref().unwrap_or_default(); + let raw_len = pce.len() + frame.payload.len(); + let header = adts::write_header(*object_type, *sample_rate, *channel_config, raw_len)?; + let mut framed = Vec::with_capacity(header.len() + raw_len); framed.extend_from_slice(&header); + framed.extend_from_slice(pce); framed.extend_from_slice(&frame.payload); Some(framed) } @@ -2086,11 +2097,19 @@ fn video_es_payload(stream_type: StreamType, description: Option<&Bytes>, frame: fn audio_kind(config: &AudioConfig, name: &str) -> anyhow::Result { ensure_raw(&config.container, "audio", name)?; match &config.codec { - AudioCodec::AAC(aac) => Ok(Kind::Aac { - object_type: aac.profile, - sample_rate: config.sample_rate, - channel_count: config.channel_count, - }), + AudioCodec::AAC(aac) => { + // The description names the layout exactly; without one, the count is all there is. + let (channel_config, program_config) = match &config.description { + Some(asc) => crate::codec::aac::in_band_channels(asc)?, + None => (adts::channel_config_from_count(config.channel_count), None), + }; + Ok(Kind::Aac { + object_type: aac.profile, + sample_rate: config.sample_rate, + channel_config, + program_config, + }) + } AudioCodec::Mp2 => Ok(Kind::Mp2 { sample_rate: config.sample_rate, }), diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 19dd6f9fae3..00d9294308c 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1425,6 +1425,65 @@ async fn ac3_roundtrip_byte_exact() { assert_eq!(roundtripped, ingested, "AC-3 frames must survive byte-for-byte"); } +/// The first ADTS frame of the first AAC PES: its header and raw data block. +fn first_adts_frame(ts: &[u8]) -> (super::adts::Header, Vec) { + let mut pes = PesPacketReader::new(TsPacketReader::new(Cursor::new(ts))); + let packet = pes.read_pes_packet().unwrap().expect("an AAC PES"); + let header = super::adts::Header::parse(&packet.data).unwrap(); + (header, packet.data[header.header_len..header.frame_len].to_vec()) +} + +/// ffmpeg's quad AAC fixture has no channelConfiguration, so its layout rides in a program +/// config element. Import moves it into the description and export puts it back: channel_config +/// 0 in ADTS and the element leading the first raw data block, exactly as ffmpeg wrote it. +#[tokio::test(start_paused = true)] +async fn aac_program_config_roundtrip() { + let data = include_bytes!("test_data/aac_quad.ts"); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = crate::container::ts::Import::new(broadcast, catalog.reserve()); + import.decode(&BytesMut::from(&data[..])).unwrap(); + import.finish().unwrap(); + + let snapshot = catalog.snapshot(); + let (name, audio) = snapshot.audio.renditions.iter().next().expect("an AAC track"); + assert_eq!(audio.channel_count, 4); + let ingested = read_frames(&consumer, name, Kind::Audio).await; + assert!(!ingested.is_empty(), "no AAC frames"); + + let ts = drain(consumer).await; + assert_packet_aligned(&ts); + + let (header, block) = first_adts_frame(&ts); + assert_eq!(header.channel_config, 0, "the layout is not a channelConfiguration"); + assert_eq!( + block, + first_adts_frame(data).1, + "the first raw data block, element and all" + ); + + let mut broadcast2 = moq_net::broadcast::Info::new().produce(); + let consumer2 = broadcast2.consume(); + let catalog2 = crate::catalog::Producer::new(&mut broadcast2, crate::catalog::Config::default()).unwrap(); + let mut import2 = crate::container::ts::Import::new(broadcast2, catalog2.reserve()); + import2.decode(&BytesMut::from(ts.as_ref())).unwrap(); + import2.finish().unwrap(); + + let snapshot2 = catalog2.snapshot(); + let (name2, audio2) = snapshot2 + .audio + .renditions + .iter() + .next() + .expect("round-trip lost the AAC track"); + assert_eq!(audio2.channel_count, 4); + assert_eq!(audio2.description, audio.description); + let roundtripped = read_frames(&consumer2, name2, Kind::Audio).await; + assert_eq!(roundtripped, ingested, "the element is written once, not per frame"); +} + /// The ffmpeg E-AC-3 fixture must survive TS -> MoQ -> TS byte-for-byte in an /// audio-only program; the PMT re-announces ATSC 0x87 with the 'EAC3' /// registration descriptor. diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 643069c77dd..057d998aa1e 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -2132,17 +2132,19 @@ impl AacStream { } }; + let mut block = &data[offset + header.header_len..end]; let import = match &mut self.import { Some(import) => import, None => { // Synthesize the AudioSpecificConfig `description` so out-of-band consumers // (fMP4/MKV export, WebCodecs) can configure the decoder. A channel_config of 0 - // moves the program config element out of this first frame into it. + // moves the program config element out of this first frame into it, as + // ffmpeg's aac_adtstoasc does; the TS export puts it back. let asc = aac::in_band_config( header.object_type, header.sample_rate, header.channel_config, - &data[offset + header.header_len..end], + &mut block, )?; let mut config = aac::config(&asc)?; config.container = self.container.clone(); @@ -2157,7 +2159,7 @@ impl AacStream { } }; - import.decode(&data[offset + header.header_len..end], pts)?; + import.decode(block, pts)?; // Count only completed frames; input gaps and unfinished tails are not a media burst. burst += std::time::Duration::from_nanos((1024_u64 * 1_000_000_000).div_ceil(header.sample_rate as u64)); // The importer accumulates; cut each ADTS frame into its own group (one QUIC stream) From c6ec067760efd46a646ac1b20825bf4a98c008c0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 16:25:21 -0700 Subject: [PATCH 07/20] feat(audio): an encode backend seam and an AAC output codec (#4183) Co-authored-by: Claude Opus 5.5 Co-authored-by: Grok 4.7 --- doc/lib/c/index.md | 2 +- doc/lib/go/index.md | 6 + doc/lib/kt/index.md | 7 +- doc/lib/py/index.md | 7 +- doc/lib/rs/moq-audio.md | 21 +- doc/lib/swift/index.md | 7 +- go/wrapper/publish.go | 2 +- go/wrapper/types.go | 10 +- .../kotlin/dev/moq/Aliases.kt | 2 +- py/moq-rs/README.md | 2 +- py/moq-rs/moq/publish.py | 4 +- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/encode-audiotoolbox.md | 6 +- quest/m1/audio-codecs/encode-backend.md | 40 -- quest/m1/obs-moq-video/audio-publish.md | 6 +- quest/m2/audio-encode-mediacodec.md | 4 +- quest/m2/audio-encode-mediafoundation.md | 1 - rs/libmoq/src/audio.rs | 7 +- rs/moq-audio/Cargo.toml | 2 +- rs/moq-audio/src/aac.rs | 4 +- rs/moq-audio/src/encode/backend/libopus.rs | 207 ++++++ rs/moq-audio/src/encode/backend/mod.rs | 323 +++++++++ rs/moq-audio/src/encode/backend/pcm.rs | 46 ++ rs/moq-audio/src/encode/backend/stub.rs | 59 ++ rs/moq-audio/src/encode/encoder.rs | 671 ++++++++++-------- rs/moq-audio/src/encode/mod.rs | 7 +- rs/moq-audio/src/encode/producer.rs | 76 +- rs/moq-audio/src/lib.rs | 7 +- rs/moq-ffi/src/audio.rs | 25 +- rs/moq-ffi/src/test.rs | 54 ++ swift/Sources/Moq/Aliases.swift | 2 +- swift/Sources/Moq/Broadcast.swift | 2 +- 32 files changed, 1221 insertions(+), 399 deletions(-) delete mode 100644 quest/m1/audio-codecs/encode-backend.md create mode 100644 rs/moq-audio/src/encode/backend/libopus.rs create mode 100644 rs/moq-audio/src/encode/backend/mod.rs create mode 100644 rs/moq-audio/src/encode/backend/pcm.rs create mode 100644 rs/moq-audio/src/encode/backend/stub.rs diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index b8425b713e6..5a4fcc0e9c4 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -35,7 +35,7 @@ and `target/include/moq.h`. - **Threading.** Any function from any thread. Raw publish calls block until the codec takes the frame, which paces a publisher. - **Connection health.** `moq_session_stats()` reports available metrics with per-field validity flags. `moq_session_snapshot()` samples those metrics and the negotiated draft name together from the same connection. Its protocol string is backed by static storage. Both return an offline error between reconnects and leave the destination untouched. `moq_session_bandwidth()` mints an allocator over the send estimate; `moq_bandwidth_reserve` claims a share for an app-owned track, and `moq_encode_video` / `moq_encode_audio` take the same handle so the built-in video encoder follows the grant. - **Raw playback.** Raw audio and video consumers start at the newest cached group when opened, so rebuilding a live decoder skips the retained backlog. -- **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec (only `"opus"` today), and `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the 20 ms default. +- **Raw audio encode.** `moq_audio_encoder_output.codec` names the codec: `"opus"`, `"pcm"`, or `"aac"`. `frame_duration_us` sets the Opus frame length: 2500, 5000, 10000, 20000, 40000, or 60000, with 0 meaning the codec's default (20 ms for Opus, 1024 samples for AAC). AAC-LC encodes through the platform's encoder, so a host without one refuses it. - **Audio channel layouts.** A `channels` count also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. `moq_decode_audio` remixes to the count you ask for; past 8 channels the samples pass through but can't be remixed. - **Raw decode output.** `moq_video_decoder_output` selects the decoded CPU pixel format (`MOQ_VIDEO_PIXEL_FORMAT_I420` or `_RGBA`) and target size (`width`/`height`, both zero for native; otherwise even and non-zero). Unknown formats and invalid sizes fail `moq_decode_video` before subscribing; accepted requests deliver exactly that layout or fail on the terminal callback. - **Encoded video metadata.** `moq_video_init.hint` is a zero-initialized `moq_video_hint` with `has_*` flags for coded dimensions, bitrate (bits per second), frame rate, and latency preference. Hints seed a video codec track's catalog; detected dimensions take precedence. diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index 0540cafb905..8a49af5747c 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -115,6 +115,12 @@ one: `FetchGroup`/`FetchMediaGroup`, `Dynamic()` with `Requests(ctx)`, `AppendDatagram`/`Datagrams(ctx)`, `SetCatalogSection`, `Demand()` for `Used`/`Unused`, `Session().Stats()`. `moq.IsAuthError` and `moq.IsShutdown` classify errors. `moq.ProtocolError(err)` is the structured protocol failure (scope, verbatim code, kind) when the peer sent one. +`EncodeAudio` encodes raw PCM inside the binding. Its codec is `OpusAudioCodec()` +or `AacAudioCodec()`, and `AudioEncoderOutput.FrameDurationUs` sets the Opus +frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. 0 takes +the codec's own frame, which AAC needs. AAC-LC encodes through the platform's +encoder, so a host without one refuses it. + Audio `Channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved front left, front right, center, LFE, back, then side. Decoding remixes to the diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index 8f863b942b1..7f552b30376 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -91,8 +91,11 @@ connection's send estimate; pass it to `encodeVideo` / `encodeAudio` or native side. `encodeAudio` encodes raw PCM inside the binding. Its codec is an object, -`AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus -frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`AudioCodec.opus()` or `AudioCodec.aac()`, and +`AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index 5fe4e12de39..1ce6293a5ec 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -107,8 +107,11 @@ Each server request reports a `moq.Transport` enum, including QUIC, Iroh, WebSocket, TCP, and Unix sockets. `encode_audio` encodes raw PCM inside the binding. Its codec is an object, -`moq.AudioCodec.opus()`, and `AudioEncoderOutput.frame_duration_us` sets the -Opus frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`moq.AudioCodec.opus()` or `moq.AudioCodec.aac()`, and +`AudioEncoderOutput.frame_duration_us` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 637343dd611..c76dc1179e7 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -28,7 +28,7 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | Module | Does | | --- | --- | | `capture` | Microphones via CoreAudio, WASAPI, ALSA (and PipeWire/PulseAudio hosts), plus macOS system audio | -| `encode` | PCM to Opus (with DTX and voice-activity signaling) or raw PCM for the lowest latency | +| `encode` | PCM to Opus (with DTX and voice-activity signaling), raw PCM for the lowest latency, or AAC-LC through a platform encoder | | `decode` | Opus, PCM, and AAC-LC back to PCM, resampled to the rate you want | | `playback` | One output device mixing every track in a call, with click-free volume ramps | | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | @@ -48,6 +48,25 @@ its config are refused at construction on every host. HE-AAC signaled only in band plays as its half-rate LC core. Linux has no OS audio decoder, so it will stay that way there. +`encode` selects the same way, through `encode::Settings::kind`, and +`Encoder::name()` reports what opened. + +| Backend | Encodes | Hosts | +| --- | --- | --- | +| `libopus` | Opus, mono or stereo | all | +| `pcm` | PCM | all | + +`encode::Codec::Aac` is AAC-LC (`mp4a.40.2`) at the input's rate and layout: +mono, stereo, 3.0, 4.0, 5.0, 5.1, or 7.1, the layouts with an AAC +channelConfiguration. Frames are 1024 samples, so `Settings::from_input` sets +`frame_duration` to match. The catalog's AudioSpecificConfig is built from the +settings when the track is registered, and since it has no field for the +encoder's delay, packets are stamped that much earlier so the first input +sample still lands at the first timestamp. There is no software AAC encoder, +and no platform encoder is wired in yet, so `Codec::Aac` is refused at +construction on every host for now. Linux has no OS encoder, so it will stay +that way there. + Highlights: - **`encode::Publication`** advertises the track and opens the microphone only while someone listens. Stop, swap devices, and restart without changing the track subscribers know; read a level meter for the UI. diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index dc02120fe7a..39ab5e152aa 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -95,8 +95,11 @@ divides the connection's send estimate; pass it to `encodeVideo` / (scope, verbatim code, kind) when the peer sent one. `encodeAudio` encodes raw PCM inside the binding. Its codec is an object, -`AudioCodec.opus()`, and `AudioEncoderOutput.frameDurationUs` sets the Opus -frame length: 2500, 5000, 10000, 20000 (the default), 40000, or 60000. +`AudioCodec.opus()` or `AudioCodec.aac()`, and +`AudioEncoderOutput.frameDurationUs` sets the Opus frame length: 2500, 5000, +10000, 20000 (the default), 40000, or 60000. 0 takes the codec's own frame, +which AAC needs. AAC-LC encodes through the platform's encoder, so a host +without one refuses it. Audio `channels` also names the speaker layout, by the WAVE convention: 1 is mono, 2 stereo, 3 2.1, 4 quad, 5 5.0, 6 5.1, 7 6.1, and 8 7.1, interleaved diff --git a/go/wrapper/publish.go b/go/wrapper/publish.go index 6eccc6d8f67..6f46b1ff2dc 100644 --- a/go/wrapper/publish.go +++ b/go/wrapper/publish.go @@ -193,7 +193,7 @@ func (b *BroadcastProducer) PublishContainerStream(format ContainerFormat) (*Con // EncodeAudio publishes a raw-audio track with an in-process encoder. // -// Select the codec with OpusAudioCodec (currently the only constructor). +// Select the codec with OpusAudioCodec or AacAudioCodec. // Pass bandwidth to reserve this track's bitrate against the session's // allocator so a co-resident video encoder sizes itself against what is left. func (b *BroadcastProducer) EncodeAudio(name string, input AudioEncoderInput, output AudioEncoderOutput, bandwidth *Bandwidth) (*AudioProducer, error) { diff --git a/go/wrapper/types.go b/go/wrapper/types.go index 40bf4f53eaa..3c588ba2dfa 100644 --- a/go/wrapper/types.go +++ b/go/wrapper/types.go @@ -8,14 +8,14 @@ import ffi "moq.dev/moq-ffi/moq" type ( // Audio describes one audio rendition in a broadcast catalog: codec, sample rate, channel count, and container. Audio = ffi.MoqAudio - // AudioCodec selects the audio encoder codec. Build one with OpusAudioCodec; + // AudioCodec selects the audio encoder codec. Build one with OpusAudioCodec or AacAudioCodec; // adding a codec later adds a constructor, not a breaking enum change. AudioCodec = ffi.MoqAudioCodec // AudioDecoderOutput configures the PCM format, sample rate, and channels DecodeAudio delivers. AudioDecoderOutput = ffi.MoqAudioDecoderOutput // AudioEncoderInput declares the PCM sample format, sample rate, and channel count of frames written to an audio producer. AudioEncoderInput = ffi.MoqAudioEncoderInput - // AudioEncoderOutput configures the Opus encoder: codec, optional sample rate, channels, bitrate, and frame duration. + // AudioEncoderOutput configures the encoder: codec, optional sample rate, channels, bitrate, and frame duration. AudioEncoderOutput = ffi.MoqAudioEncoderOutput // AudioSampleFormat is a raw PCM sample layout, mirroring WebCodecs AudioData.format. AudioSampleFormat = ffi.MoqAudioSampleFormat @@ -184,6 +184,12 @@ func OpusAudioCodec() *AudioCodec { return ffi.MoqAudioCodecOpus() } +// AacAudioCodec selects AAC-LC through the platform's encoder for EncodeAudio. +// A host without one refuses it. Leave FrameDurationUs at 0 for AAC's own frame. +func AacAudioCodec() *AudioCodec { + return ffi.MoqAudioCodecAac() +} + // VideoPixelFormat values: the raw pixel layout fed to the in-process encoder, // and the one the in-process decoder delivers. const ( diff --git a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt index 38883f6cdaa..f59782cf66e 100644 --- a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt +++ b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt @@ -134,7 +134,7 @@ typealias FetchGroupOptions = uniffi.moq.MoqFetchGroupOptions typealias TrackInfo = uniffi.moq.MoqTrackInfo /** One audio frame: PCM payload bytes plus a presentation timestamp. */ typealias AudioFrame = uniffi.moq.MoqAudioFrame -/** Selects the audio encoder codec. Build one with `AudioCodec.opus()`. */ +/** Selects the audio encoder codec. Build one with `AudioCodec.opus()` or `AudioCodec.aac()`. */ typealias AudioCodec = uniffi.moq.MoqAudioCodec /** A raw PCM sample format, mirroring WebCodecs `AudioData.format`. */ typealias AudioSampleFormat = uniffi.moq.MoqAudioSampleFormat diff --git a/py/moq-rs/README.md b/py/moq-rs/README.md index 6add3e7adb3..0b60f1cb924 100644 --- a/py/moq-rs/README.md +++ b/py/moq-rs/README.md @@ -154,7 +154,7 @@ client = moq.Client( - `.publish_audio(format, init, *, label=None, track=None) → MediaProducer`. `init` is required: an OpusHead or AudioSpecificConfig resolves the whole rendition. `track` names the track; otherwise a unique name is derived from the format. - `.publish_video(format, init=b"", *, label=None, hint=None, track=None) → MediaProducer`. `init` may be empty for a format that resolves in band; a `VideoHint` pins catalog fields the stream can't reveal (bitrate) or publishes the catalog before the first keyframe. `track` names the track as in `publish_audio`. - `.encode_video(input, output, *, bandwidth=None) → VideoProducer`. Encode raw `VideoFrame`s inside the binding; `.write(frame)` each one. - - `.encode_audio(name, input, output, *, bandwidth=None) → AudioProducer`. Encode raw PCM `AudioFrame`s; the codec is `output.codec`, e.g. `AudioCodec.opus()`, with `output.frame_duration_us` setting the Opus frame length. + - `.encode_audio(name, input, output, *, bandwidth=None) → AudioProducer`. Encode raw PCM `AudioFrame`s; the codec is `output.codec`, e.g. `AudioCodec.opus()` or `AudioCodec.aac()`, with `output.frame_duration_us` setting the Opus frame length (0 takes the codec's own frame, which AAC needs). - `.finish()` - **`BroadcastDynamic`**. Async source of tracks requested by subscribers. - `await .requested_track() → TrackRequest`. Call `.accept()` on it for a `TrackProducer`, or `.abort(code)` to reject. diff --git a/py/moq-rs/moq/publish.py b/py/moq-rs/moq/publish.py index cbaf29ba13c..e7e8bb03af3 100644 --- a/py/moq-rs/moq/publish.py +++ b/py/moq-rs/moq/publish.py @@ -749,8 +749,8 @@ def encode_audio( ) -> AudioProducer: """Publish a raw-audio track with an in-process encoder. - Select the codec with ``moq.AudioCodec.opus()`` (currently the only - constructor), placed in ``output``. + Select the codec with ``moq.AudioCodec.opus()`` or + ``moq.AudioCodec.aac()``, placed in ``output``. Pass ``bandwidth`` to reserve this track's bitrate against the session's allocator so a co-resident video encoder sizes itself against what is left. diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index fdeca617059..ee3cca548fd 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -45,7 +45,6 @@ its own decode and encode quest so verification stays per host. - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `encode::backend` and `Codec::Aac`, so a native publisher can produce AAC-LC - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC ## Related diff --git a/quest/m1/audio-codecs/encode-audiotoolbox.md b/quest/m1/audio-codecs/encode-audiotoolbox.md index 2ceda88280e..71267e9b635 100644 --- a/quest/m1/audio-codecs/encode-audiotoolbox.md +++ b/quest/m1/audio-codecs/encode-audiotoolbox.md @@ -15,11 +15,15 @@ the encode seam as the platform candidate on macOS and iOS. `kAudioConverterPrimeInfo` gives the delay the timestamps fold in. - Bitrate through `kAudioConverterEncodeBitRate`, updated live where the converter allows. +- The seam assumes one packet per frame. If the converter holds output back, + the backend needs a `flush` and a zero-or-more return, which changes + `Encoder::encode` and so targets `dev`. +- Gate the seam's "AAC refused without a platform encoder" test to hosts + without one. - Regression: a stereo and a 5.1 encode round-trip through the AudioToolbox decoder and through symphonia (stereo only), with timestamps continuous across the priming. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the round-trip regression decodes through it diff --git a/quest/m1/audio-codecs/encode-backend.md b/quest/m1/audio-codecs/encode-backend.md deleted file mode 100644 index 4f73ed98f26..00000000000 --- a/quest/m1/audio-codecs/encode-backend.md +++ /dev/null @@ -1,40 +0,0 @@ -# [M] An encode backend seam and an AAC output codec - -## Goal - -`moq_audio::encode` selects a backend the way `moq_video::encode` does, and -`Codec::Aac` is a valid output: AAC-LC at the input's sample rate and layout, -published with the AudioSpecificConfig description every player expects. A -host with no AAC encoder refuses it at construction. - -## Plan - -Mirror the decode seam (`rs/moq-audio/src/decode/backend`): `encode::backend` with a crate-private `Backend` -trait (`encode`, `flush`, `set_bitrate`, `name`), an `open(codec, config)` -that walks platform candidates before software ones, using the public settings -and selection contract settled in main. Opus and PCM retain their behavior. This -quest adds AAC through platform encoders; no software AAC dependency is selected. - -- `encode::Codec` gains `Aac`, meaning `mp4a.40.2`, and `as_str` / `FromStr` - accept `"aac"`, which is what libmoq's codec string carries. moq-ffi's - immutable codec object gains an `aac()` constructor and conversion; do not - reintroduce a closed binding enum. The generated bindings, hand-written - wrappers, and docs follow the Cross-Package Sync table. -- Catalog emission: `AudioCodec::AAC` with profile 2, `Container::Legacy`, - and the encoder's reported delay folded into timestamps like Opus pre-skip - is today. `Producer` registers the rendition before the first frame is - written, so the ASC `description` is synthesized at construction from the - config with `moq_mux::codec::aac::Config::encode`, never read back from the - backend's first packet. A backend that reports its own header (a magic - cookie, `csd-0`, `MF_MT_USER_DATA`) must produce one equal to the synthesized - ASC, asserted in its tests. -- Frame size is the codec's (1024 samples for AAC), so `frame_duration` is - validated per codec rather than against the Opus table. -- Bitrate updates go through the backend; one that cannot change rate - mid-stream keeps its opening rate, as the video seam documents. -- Regression: the selection order with a stub backend; `Codec::Aac` refused on - a host with no backend; the Opus and PCM paths unchanged. - -## Related - -- [OBS audio publishing](/quest/m1/obs-moq-video/audio-publish.md) - the OBS encoder adapter can offer AAC once this lands diff --git a/quest/m1/obs-moq-video/audio-publish.md b/quest/m1/obs-moq-video/audio-publish.md index ae2543cd356..3b7cf9981a5 100644 --- a/quest/m1/obs-moq-video/audio-publish.md +++ b/quest/m1/obs-moq-video/audio-publish.md @@ -8,7 +8,7 @@ MoQ publishing can encode OBS's mixed audio with moq-audio Opus while preserving - Expose `moq_audio::encode::Encoder` through a codec-only moq-ffi type with owned encoder/packet handles, so every binding gets it. The existing raw audio producer combines encoding and publication and must not create a second publication alongside the OBS encoded output. Reuse frame sizing, bitrate updates, catalog configuration, and finish/padding behavior. - Implement the OBS audio encoder interface, including fixed input frame size, mono/stereo PCM conversion, timestamps, codec headers, final padding, and packet release. Ask OBS for the input layout the encoder supports; moq-audio does not implement arbitrary channel remapping. Keep capture/mixing/device ownership in OBS. -- Publish Opus initially. Leave AAC with the existing OBS mode and defer PCM publishing UI, since the output currently declares AAC/Opus. Correct stale binding documentation that describes the raw codec parser as Opus-only if that API is touched. +- Publish Opus initially. `encode::Codec::Aac` lets the adapter offer AAC where a platform encoder exists; until then leave AAC with the existing OBS mode and defer PCM publishing UI, since the output currently declares AAC/Opus. Correct stale binding documentation that describes the raw codec parser as Opus-only if that API is touched. - Apply the shared presets and independent audio bitrate. Test 10/20 ms packetization, frame-size changes at stream boundaries, partial final frames, silence, reconnect, saturation, and stop with pending output. Verify decoded audio and A/V timestamps with a real subscriber. - Land the internal adapter independently; the combined Use MoQ encoders UI becomes available when the video adapter also lands. Do not expose a temporary video/audio mix-and-match product UI. @@ -16,7 +16,3 @@ MoQ publishing can encode OBS's mixed audio with moq-audio Opus while preserving - [OBS migration](/quest/m1/cpp/obs.md) - the plugin is on the generated C++ first - [Encoder presets](/quest/m1/obs-moq-video/presets.md) - shared policy and truthful reporting - -## Related - -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - `Codec::Aac` lets the adapter offer AAC instead of leaving it to the OBS encoder mode diff --git a/quest/m2/audio-encode-mediacodec.md b/quest/m2/audio-encode-mediacodec.md index d488ec184ba..8af6ad3e512 100644 --- a/quest/m2/audio-encode-mediacodec.md +++ b/quest/m2/audio-encode-mediacodec.md @@ -13,6 +13,9 @@ behind the `mediacodec` feature and the encode seam. - `audio/mp4a-latm` with `KEY_AAC_PROFILE` = LC. The catalog ASC is synthesized at construction per the encode seam, since `csd-0` only arrives with the first output buffer; assert the two match. +- MediaCodec pipelines output, which the one-packet-per-frame seam does not + allow yet: add a `flush` and a zero-or-more return, a change to + `Encoder::encode` that targets `dev`, unless the AudioToolbox quest already did. - Multichannel is device-dependent; probe the encoder's capabilities at open and refuse a layout it does not list. - Round-trip regression through the MediaCodec decoder; runtime proof on a @@ -20,5 +23,4 @@ behind the `mediacodec` feature and the encode seam. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [MediaCodec decode](/quest/m2/audio-decode-mediacodec.md) - the round-trip regression decodes through it diff --git a/quest/m2/audio-encode-mediafoundation.md b/quest/m2/audio-encode-mediafoundation.md index 8347dd1854f..b2bad529722 100644 --- a/quest/m2/audio-encode-mediafoundation.md +++ b/quest/m2/audio-encode-mediafoundation.md @@ -20,7 +20,6 @@ behind the encode seam on Windows. ## Required -- [Encode seam](/quest/m1/audio-codecs/encode-backend.md) - the candidate order this backend joins - [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - the round-trip regression decodes through it ## Related diff --git a/rs/libmoq/src/audio.rs b/rs/libmoq/src/audio.rs index d389abc0fee..21d3b1c2257 100644 --- a/rs/libmoq/src/audio.rs +++ b/rs/libmoq/src/audio.rs @@ -79,7 +79,8 @@ pub struct moq_audio_encoder_input { #[repr(C)] #[allow(non_camel_case_types)] pub struct moq_audio_encoder_output { - /// Codec id, UTF-8 (currently only "opus"). + /// Codec id, UTF-8: "opus", "pcm", or "aac". AAC encodes through the + /// platform's encoder, so a host without one refuses it. pub codec: *const c_char, pub codec_len: usize, /// 0 = derive from input. @@ -89,8 +90,8 @@ pub struct moq_audio_encoder_output { /// 0 = libopus default. pub bitrate: u32, /// Encoded frame duration in microseconds. Opus accepts exactly - /// 2500/5000/10000/20000/40000/60000 us. 0 = the 20 ms default, which - /// matches the JS publish path. + /// 2500/5000/10000/20000/40000/60000 us. 0 = the codec's default: 20 ms for + /// Opus, which matches the JS publish path, and 1024 samples for AAC. pub frame_duration_us: u32, } diff --git a/rs/moq-audio/Cargo.toml b/rs/moq-audio/Cargo.toml index a270e8dbdec..5980b1bc2c6 100644 --- a/rs/moq-audio/Cargo.toml +++ b/rs/moq-audio/Cargo.toml @@ -26,7 +26,7 @@ default = ["aac"] # AAC-LC decode via symphonia. Pure Rust, so it costs no toolchain: the trade # this crate already refused for Opus. Only a subscriber to an ingest-sourced # broadcast needs it (RTMP, SRT/TS, fmp4, and gstreamer all publish AAC, while -# everything this crate encodes is Opus or PCM), so a publish-only build can +# this crate encodes in software is Opus or PCM), so a publish-only build can # drop it and the two crates it pulls. aac = ["dep:symphonia-codec-aac", "dep:symphonia-core"] # Device capture. Microphones go through cpal (pure-Rust: CoreAudio / WASAPI / diff --git a/rs/moq-audio/src/aac.rs b/rs/moq-audio/src/aac.rs index 7c8b9482334..52fb66e3dd3 100644 --- a/rs/moq-audio/src/aac.rs +++ b/rs/moq-audio/src/aac.rs @@ -1,7 +1,7 @@ //! AAC constraints, the sibling of the `opus` and `pcm` modules. //! -//! Only the decode side exists: there is no Rust AAC encoder, so this crate -//! publishes Opus or PCM and reads AAC that a gateway produced. +//! The decode side. There is no Rust AAC encoder, so `encode` produces AAC only +//! through a platform backend and otherwise reads AAC that a gateway produced. use crate::Error; diff --git a/rs/moq-audio/src/encode/backend/libopus.rs b/rs/moq-audio/src/encode/backend/libopus.rs new file mode 100644 index 00000000000..118c701be46 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/libopus.rs @@ -0,0 +1,207 @@ +//! Opus through libopus 1.3.1, via [`unsafe_libopus`]. + +use bytes::Bytes; +use unsafe_libopus::{ + OPUS_APPLICATION_AUDIO, OPUS_GET_BITRATE_REQUEST, OPUS_GET_LOOKAHEAD_REQUEST, OPUS_OK, OPUS_RESET_STATE, + OPUS_SET_BITRATE_REQUEST, OPUS_SET_DTX_REQUEST, OpusEncoder, opus_encode_float, opus_encoder_create, + opus_encoder_ctl_impl, opus_encoder_destroy, varargs, +}; + +use super::Backend; +use crate::encode::{Encoded, Settings}; +use crate::{Error, opus}; + +pub(super) const NAME: &str = "libopus"; + +/// libopus packet size ceiling per RFC 6716 §3.4. +const MAX_PACKET_BYTES: usize = 4_000; + +pub(super) struct Libopus { + inner: *mut OpusEncoder, + scratch: Vec, + sample_rate: u32, + channels: u32, + frame_size: usize, + bitrate: u64, + lookahead: usize, +} + +// SAFETY: OpusEncoder is heap-allocated state owned exclusively by this +// struct; libopus encoder methods take a single &mut, so a unique owner is +// allowed to move it across threads. +unsafe impl Send for Libopus {} + +impl Libopus { + /// Opens at the settings' rate and layout, which the front end has already + /// checked against what Opus codes. + pub(super) fn open(settings: &Settings) -> Result, Error> { + Ok(Box::new(Self::new(settings)?)) + } + + fn new(settings: &Settings) -> Result { + let sample_rate = settings.sample_rate; + let channels = settings.layout.channels(); + let frame_size = opus::frame_size(sample_rate, settings.frame_duration)?; + + let mut err = 0i32; + // SAFETY: out-pointer `err` is valid; inner is checked for null below. + let inner = unsafe { + opus_encoder_create( + sample_rate as i32, + opus::validate_channels(channels)?, + OPUS_APPLICATION_AUDIO, + &mut err, + ) + }; + if err != OPUS_OK || inner.is_null() { + return Err(opus::error(err, "opus_encoder_create")); + } + + // Owned from here, so an early return below destroys it. + let mut backend = Self { + inner, + scratch: vec![0u8; MAX_PACKET_BYTES], + sample_rate, + channels, + frame_size, + bitrate: 0, + lookahead: 0, + }; + + if let Some(bitrate) = settings.bitrate { + backend.set_rate(bitrate.as_bps())?; + } + backend.set_ctl(OPUS_SET_DTX_REQUEST, i32::from(settings.dtx), "OPUS_SET_DTX")?; + + let bitrate = backend.get_ctl(OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE")?; + backend.bitrate = u64::try_from(bitrate) + .map_err(|_| Error::Unsupported(format!("Opus reported negative bitrate {bitrate}")))?; + let lookahead = backend.get_ctl(OPUS_GET_LOOKAHEAD_REQUEST, "OPUS_GET_LOOKAHEAD")?; + backend.lookahead = usize::try_from(lookahead) + .map_err(|_| Error::Unsupported(format!("Opus reported negative lookahead {lookahead}")))?; + + Ok(backend) + } + + /// Refuse rates libopus would silently clamp, then apply the rest. + fn set_rate(&mut self, bitrate: u64) -> Result<(), Error> { + let (channels, frame_size) = (self.channels, self.frame_size); + let max = 300_000 * channels as u64; + let min = opus::bitrate_floor(self.sample_rate, frame_size).max(500); + if !(min..=max).contains(&bitrate) { + return Err(Error::Unsupported(format!( + "Opus bitrate must be between {min} and {max} bits per second for {channels} channel(s) at {frame_size} samples, got {bitrate}" + ))); + } + self.set_ctl(OPUS_SET_BITRATE_REQUEST, bitrate as i32, "OPUS_SET_BITRATE") + } + + fn set_ctl(&mut self, request: i32, value: i32, name: &'static str) -> Result<(), Error> { + // SAFETY: `inner` owns a live encoder and each request here expects one i32. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, request, varargs![value]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, name)); + } + Ok(()) + } + + fn get_ctl(&self, request: i32, name: &'static str) -> Result { + let mut value = 0; + // SAFETY: `inner` owns a live encoder and each request here expects one + // valid mutable i32 output. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, request, varargs![&mut value]) }; + if rc != OPUS_OK { + return Err(opus::error(rc, name)); + } + Ok(value) + } +} + +impl Backend for Libopus { + fn encode(&mut self, pcm: &[f32]) -> Result { + // SAFETY: `inner` owns a live OpusEncoder; pcm and scratch slices are + // bounded by the lengths we pass, and the front end sized `pcm` to one frame. + let n = unsafe { + opus_encode_float( + self.inner, + pcm.as_ptr(), + self.frame_size as i32, + self.scratch.as_mut_ptr(), + self.scratch.len() as i32, + ) + }; + if n < 0 { + return Err(opus::error(n, "opus_encode_float")); + } + let payload = Bytes::copy_from_slice(&self.scratch[..n as usize]); + let activity = opus::activity(&payload, false); + Ok(Encoded { payload, activity }) + } + + fn reset(&mut self) { + // SAFETY: `inner` owns a live encoder and OPUS_RESET_STATE takes no arguments. + let rc = unsafe { opus_encoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + debug_assert_eq!(rc, OPUS_OK, "OPUS_RESET_STATE failed with {rc}"); + } + + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error> { + if bitrate != self.bitrate { + self.set_rate(bitrate)?; + self.bitrate = bitrate; + } + Ok(()) + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + self.lookahead + } + + fn name(&self) -> &str { + NAME + } +} + +impl Drop for Libopus { + fn drop(&mut self) { + // SAFETY: `inner` is a live OpusEncoder that nothing else aliases. + unsafe { opus_encoder_destroy(self.inner) }; + } +} + +#[cfg(test)] +mod tests { + use unsafe_libopus::OPUS_GET_DTX_REQUEST; + + use super::*; + + #[test] + fn runtime_bitrate_reaches_libopus() { + let mut backend = Libopus::new(&Settings { + bitrate: Some(moq_net::bandwidth::Rate::from_bps(64_000)), + ..Settings::default() + }) + .unwrap(); + + backend.set_bitrate(32_000).unwrap(); + assert_eq!(backend.bitrate(), 32_000); + assert_eq!( + backend.get_ctl(OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE").unwrap(), + 32_000 + ); + } + + #[test] + fn applies_dtx_control() { + let backend = Libopus::new(&Settings { + dtx: true, + ..Settings::default() + }) + .unwrap(); + + assert_eq!(backend.get_ctl(OPUS_GET_DTX_REQUEST, "OPUS_GET_DTX").unwrap(), 1); + } +} diff --git a/rs/moq-audio/src/encode/backend/mod.rs b/rs/moq-audio/src/encode/backend/mod.rs new file mode 100644 index 00000000000..1879d49f37e --- /dev/null +++ b/rs/moq-audio/src/encode/backend/mod.rs @@ -0,0 +1,323 @@ +//! Pluggable audio encoder backends. +//! +//! The mirror of the decode backends. [`Backend`] is the seam between the codec +//! and the [`Encoder`](super::Encoder) front end, which owns what every backend +//! of a codec shares: validating [`Settings`](super::Settings) against the +//! codec, framing the input, draining the startup delay at the end, and the +//! catalog entry, whose description is synthesized from the settings rather than +//! read back from the backend. +//! +//! [`open`] tries the platform encoders before the software ones, skipping any +//! that does not emit the codec, and refuses when none opens. The software tier +//! is libopus for Opus and a passthrough for PCM. AAC has no software encoder, +//! so it is only as available as the platform's, and no platform encoder is +//! wired in yet. + +use super::Encoded; +use super::encoder::{Codec, Kind, Settings}; +use crate::Error; + +mod libopus; +mod pcm; + +#[cfg(test)] +pub(crate) mod stub; + +/// An opened encoder: one frame of interleaved `f32` PCM in, one packet out. +/// +/// Input arrives at the settings' rate, in the crate's canonical channel order +/// for the settings' layout; a codec with another native order reorders it here. +/// +/// One packet per frame is part of the contract: the producer stamps packets by +/// counting frames. A codec that pipelines output (MediaCodec) needs a `flush` +/// and a zero-or-more return, which changes `Encoder::encode` too, so that lands +/// with the first backend that needs it rather than as an always-empty method. +pub(crate) trait Backend: Send { + /// Encode exactly one frame of the codec's frame size. + fn encode(&mut self, pcm: &[f32]) -> Result; + + /// Drop codec history so the next frame codes as if it were the first. + fn reset(&mut self); + + /// Retune the live encoder to `bitrate` bits per second, a no-op at the current + /// rate. + /// + /// No default: a backend that can't change rate mid-stream refuses with + /// [`Error::Unsupported`] and keeps its opening rate, rather than inheriting a + /// silent no-op that ignores congestion. + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error>; + + /// The current target bitrate in bits per second, as the codec resolved it. + fn bitrate(&self) -> u64; + + /// Frames of codec priming at the start of the decoded stream, at the codec + /// rate: Opus lookahead, AAC encoder delay. + fn delay(&self) -> usize; + + /// The stable lowercase name [`Kind::Named`] selects this backend by. + fn name(&self) -> &str; +} + +/// A backend constructor: its name, the codecs it emits, and an opener. +struct Candidate { + name: &'static str, + codecs: &'static [Codec], + open: fn(&Settings) -> Result, Error>, +} + +/// Operating-system encoders, in priority order. +const PLATFORM: &[Candidate] = &[]; + +const SOFTWARE: &[Candidate] = &[ + Candidate { + name: libopus::NAME, + codecs: &[Codec::Opus], + open: libopus::Libopus::open, + }, + Candidate { + name: pcm::NAME, + codecs: &[Codec::Pcm], + open: pcm::Pcm::open, + }, +]; + +/// Test-only backends, in neither tier so `Auto` and `Software` never pick one: +/// they exist to be asked for by name. +#[cfg(test)] +const NAMED_ONLY: &[Candidate] = &[Candidate { + name: stub::NAME, + codecs: &[Codec::Aac], + open: stub::Stub::open, +}]; + +#[cfg(not(test))] +const NAMED_ONLY: &[Candidate] = &[]; + +/// Open the first backend that emits the codec and accepts the settings. +pub(crate) fn open(settings: &Settings) -> Result, Error> { + select(settings, candidates(&settings.kind, PLATFORM, SOFTWARE)) +} + +/// The candidates `kind` allows, in the order to try them. +/// +/// Takes the tiers as arguments so a test can supply stubs instead of whatever +/// this host compiles in. +fn candidates<'a>(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { + match kind { + Kind::Auto => platform.iter().chain(software).collect(), + Kind::Software => software.iter().collect(), + Kind::Named(name) => platform + .iter() + .chain(software) + .chain(NAMED_ONLY) + .filter(|c| c.name == name) + .collect(), + } +} + +fn select(settings: &Settings, candidates: Vec<&Candidate>) -> Result, Error> { + let codec = settings.codec; + let mut refused = Vec::new(); + + for candidate in candidates { + if !candidate.codecs.contains(&codec) { + continue; + } + match (candidate.open)(settings) { + Ok(backend) => return Ok(backend), + Err(err) => refused.push((candidate.name, err)), + } + } + + // One refusal is the whole answer, so keep its variant. + if refused.len() == 1 { + let (_, err) = refused.remove(0); + return Err(err); + } + if !refused.is_empty() { + let reasons: Vec = refused.iter().map(|(name, err)| format!("{name}: {err}")).collect(); + return Err(Error::Unsupported(reasons.join(", "))); + } + + let available: Vec<&str> = PLATFORM + .iter() + .chain(SOFTWARE) + .filter(|c| c.codecs.contains(&codec)) + .map(|c| c.name) + .collect(); + let available = match available.is_empty() { + true => "none".to_owned(), + false => available.join(", "), + }; + Err(Error::Unsupported(match &settings.kind { + Kind::Named(name) => format!("no audio encoder named {name:?} for {codec} (this build has: {available})"), + Kind::Software => format!("no software {codec} audio encoder (this build has: {available})"), + Kind::Auto => format!("no {codec} audio encoder (this build has: {available})"), + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Opens anything it advertises and reports which candidate it came from. + struct Fake(&'static str); + + impl Backend for Fake { + fn encode(&mut self, _pcm: &[f32]) -> Result { + Ok(Encoded::new(bytes::Bytes::new())) + } + + fn reset(&mut self) {} + + fn set_bitrate(&mut self, _bitrate: u64) -> Result<(), Error> { + Ok(()) + } + + fn bitrate(&self) -> u64 { + 0 + } + + fn delay(&self) -> usize { + 0 + } + + fn name(&self) -> &str { + self.0 + } + } + + const PLATFORM_STUB: Candidate = Candidate { + name: "platform", + codecs: &[Codec::Aac], + open: |_| Ok(Box::new(Fake("platform"))), + }; + + /// Compiled in but refusing the settings, like a platform encoder asked for a + /// layout its framework does not open. + const REFUSING: Candidate = Candidate { + name: "refusing", + codecs: &[Codec::Aac], + open: |_| Err(Error::Unsupported("not these settings".into())), + }; + + const SOFTWARE_STUB: Candidate = Candidate { + name: "software", + codecs: &[Codec::Aac], + open: |_| Ok(Box::new(Fake("software"))), + }; + + /// Emits nothing but PCM, so an AAC request never reaches its opener. + const PCM_ONLY: Candidate = Candidate { + name: "pcm-only", + codecs: &[Codec::Pcm], + open: |_| panic!("opened for a codec it does not emit"), + }; + + fn aac(kind: Kind) -> Settings { + Settings { + kind, + ..Settings::from_input(Codec::Aac, &Default::default()) + } + } + + fn pick(kind: Kind, platform: &[Candidate], software: &[Candidate]) -> Result { + let settings = aac(kind); + let backend = select(&settings, candidates(&settings.kind, platform, software))?; + Ok(backend.name().to_owned()) + } + + #[test] + fn auto_prefers_platform() { + let name = pick(Kind::Auto, &[PCM_ONLY, PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); + } + + #[test] + fn auto_falls_back_to_software() { + let name = pick(Kind::Auto, &[REFUSING], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn software_skips_platform() { + let name = pick(Kind::Software, &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "software"); + } + + #[test] + fn named_forces_one() { + let name = pick( + Kind::Named("software".into()), + &[PLATFORM_STUB], + &[PCM_ONLY, SOFTWARE_STUB], + ) + .unwrap(); + assert_eq!(name, "software"); + } + + /// A named backend that refuses the settings is the answer: nothing else is tried. + #[test] + fn named_refusal_does_not_fall_back() { + let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); + assert!(err.to_string().contains("not these settings"), "{err}"); + } + + #[test] + fn every_refusal_is_reported() { + const ALSO_REFUSING: Candidate = Candidate { + name: "also-refusing", + ..REFUSING + }; + + let err = pick(Kind::Auto, &[REFUSING], &[ALSO_REFUSING]).unwrap_err(); + let message = err.to_string(); + assert!( + message.contains("refusing: ") && message.contains("also-refusing: "), + "{message}" + ); + } + + /// No AAC encoder is wired in outside the test stub, so `Auto` refuses at + /// construction and says there is nothing to fall back to. A platform backend + /// gates this to the hosts without one. + #[test] + fn aac_without_a_platform_encoder_is_refused() { + let err = open(&aac(Kind::Auto)).err().expect("no AAC encoder on this host"); + let message = err.to_string(); + assert!(message.contains("aac") && message.contains("none"), "{message}"); + } + + /// An unknown name says what this build has for the codec instead. + #[test] + fn unknown_name_lists_the_alternatives() { + let settings = Settings { + kind: Kind::Named("opus".into()), + ..Settings::default() + }; + let message = open(&settings) + .err() + .expect("no backend is named after its codec") + .to_string(); + assert!( + message.contains("\"opus\"") && message.contains(libopus::NAME), + "{message}" + ); + } + + #[test] + fn software_backends_open_by_name() { + let settings = Settings { + kind: Kind::Named(libopus::NAME.into()), + ..Settings::default() + }; + assert_eq!(open(&settings).unwrap().name(), libopus::NAME); + + let settings = Settings { + codec: Codec::Pcm, + kind: Kind::Named(pcm::NAME.into()), + ..Settings::default() + }; + assert_eq!(open(&settings).unwrap().name(), pcm::NAME); + } +} diff --git a/rs/moq-audio/src/encode/backend/pcm.rs b/rs/moq-audio/src/encode/backend/pcm.rs new file mode 100644 index 00000000000..04ff1f16e24 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/pcm.rs @@ -0,0 +1,46 @@ +//! Uncompressed little-endian `f32` PCM, which needs no codec at all. + +use super::Backend; +use crate::encode::{Encoded, Settings}; +use crate::{Error, pcm}; + +pub(super) const NAME: &str = "pcm"; + +pub(super) struct Pcm { + bitrate: u64, +} + +impl Pcm { + pub(super) fn open(settings: &Settings) -> Result, Error> { + let bitrate = pcm::bitrate(settings.sample_rate, settings.layout.channels())?; + Ok(Box::new(Self { bitrate })) + } +} + +impl Backend for Pcm { + fn encode(&mut self, pcm: &[f32]) -> Result { + let mut payload = Vec::with_capacity(std::mem::size_of_val(pcm)); + for sample in pcm { + payload.extend_from_slice(&sample.to_le_bytes()); + } + Ok(Encoded::new(payload.into())) + } + + fn reset(&mut self) {} + + fn set_bitrate(&mut self, _bitrate: u64) -> Result<(), Error> { + Err(Error::Unsupported("pcm bitrate is fixed".into())) + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + 0 + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/encode/backend/stub.rs b/rs/moq-audio/src/encode/backend/stub.rs new file mode 100644 index 00000000000..c581332c156 --- /dev/null +++ b/rs/moq-audio/src/encode/backend/stub.rs @@ -0,0 +1,59 @@ +//! A stand-in AAC encoder, so the AAC front end is testable on a host with no +//! platform encoder. Selectable only by name, and only in tests. + +use bytes::Bytes; + +use super::Backend; +use crate::Error; +use crate::encode::{Encoded, Settings}; + +pub(crate) const NAME: &str = "stub"; + +/// The AudioToolbox AAC-LC encoder delay, which is what a real backend reports. +pub(crate) const DELAY: usize = 2112; + +/// Emits each frame's index as its payload, and cannot retune. +pub(crate) struct Stub { + bitrate: u64, + frames: u64, +} + +impl Stub { + pub(super) fn open(settings: &Settings) -> Result, Error> { + Ok(Box::new(Self { + bitrate: settings.bitrate.map_or(128_000, |rate| rate.as_bps()), + frames: 0, + })) + } +} + +impl Backend for Stub { + fn encode(&mut self, _pcm: &[f32]) -> Result { + let payload = Bytes::copy_from_slice(&self.frames.to_be_bytes()); + self.frames += 1; + Ok(Encoded::new(payload)) + } + + fn reset(&mut self) { + self.frames = 0; + } + + fn set_bitrate(&mut self, bitrate: u64) -> Result<(), Error> { + match bitrate == self.bitrate { + true => Ok(()), + false => Err(Error::Unsupported("the stub cannot change rate mid-stream".into())), + } + } + + fn bitrate(&self) -> u64 { + self.bitrate + } + + fn delay(&self) -> usize { + DELAY + } + + fn name(&self) -> &str { + NAME + } +} diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index 90479471db4..5f908c8b686 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -1,26 +1,29 @@ //! Audio encoder front end. //! -//! [`Encoder`] dispatches over the closed [`Codec`] set. Opus wraps libopus -//! 1.3.1 via [`unsafe_libopus`], while PCM serializes interleaved `f32` samples -//! directly. +//! [`Encoder`] checks [`Settings`] against the codec, opens a +//! [`Backend`](super::backend::Backend) for it, and owns what every backend of a +//! codec shares: framing, the terminal drain, and the catalog entry. use std::str::FromStr; use std::time::Duration; use bytes::Bytes; -use unsafe_libopus::{ - OPUS_APPLICATION_AUDIO, OPUS_GET_BITRATE_REQUEST, OPUS_GET_LOOKAHEAD_REQUEST, OPUS_OK, OPUS_RESET_STATE, - OPUS_SET_BITRATE_REQUEST, OPUS_SET_DTX_REQUEST, OpusEncoder, opus_encode_float, opus_encoder_create, - opus_encoder_ctl_impl, opus_encoder_destroy, varargs, -}; use super::Encoded; +use super::backend::{self, Backend}; use crate::opus; use crate::pcm; use crate::{Error, Format, Layout}; -/// libopus packet size ceiling per RFC 6716 §3.4. -const MAX_PACKET_BYTES: usize = 4_000; +/// Samples per channel in one AAC-LC frame. +const AAC_FRAME_SIZE: usize = 1024; + +/// The audioObjectType of AAC-LC (ISO 14496-3 Table 1.17), `mp4a.40.2`. +const AAC_LC: u8 = 2; + +/// The widest sample rate an AudioSpecificConfig can name: the escape from the +/// frequency table is a 24-bit field. +const AAC_MAX_SAMPLE_RATE: u32 = 0xFF_FFFF; /// Output audio codec. `#[non_exhaustive]` so new codecs can be added without /// breaking external `match`es. @@ -32,6 +35,9 @@ pub enum Codec { Opus, /// Uncompressed interleaved little-endian IEEE-754 binary32 PCM. Pcm, + /// AAC-LC (`mp4a.40.2`), through the platform's encoder. A host without one + /// refuses it at construction. + Aac, } impl Codec { @@ -41,6 +47,7 @@ impl Codec { match self { Self::Opus => "opus", Self::Pcm => "pcm", + Self::Aac => "aac", } } } @@ -58,11 +65,25 @@ impl FromStr for Codec { match s { "opus" => Ok(Self::Opus), "pcm" => Ok(Self::Pcm), + "aac" => Ok(Self::Aac), other => Err(Error::Unsupported(format!("unknown codec: {other}"))), } } } +/// Encoder backend selection. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum Kind { + /// Prefer a platform encoder, falling back to software. + #[default] + Auto, + /// Require a software backend. + Software, + /// Require a backend by its stable lowercase name: `"libopus"` or `"pcm"`. + Named(String), +} + /// PCM supplied to [`Producer::write`](super::Producer::write). #[derive(Clone, Debug)] #[non_exhaustive] @@ -101,9 +122,12 @@ pub struct Settings { /// Sample rate accepted by the codec. pub sample_rate: u32, /// Layout accepted by the codec. + /// + /// AAC takes the layouts its channelConfiguration names: mono, stereo, 3.0, + /// 4.0, 5.0, 5.1, and 7.1. pub layout: Layout, - /// Bitrate in bits per second. `None` lets Opus pick. PCM requires `None` - /// because its bitrate is fixed by the sample rate and channel count. + /// Bitrate in bits per second. `None` lets the codec pick. PCM requires + /// `None` because its bitrate is fixed by the sample rate and channel count. /// /// Rates too low for Opus to code anything at the chosen /// [`frame_duration`](Self::frame_duration) are rejected. The floor is 1200 @@ -113,8 +137,13 @@ pub struct Settings { /// Enable Opus discontinuous transmission during silence. pub dtx: bool, /// Encoded frame duration. Opus accepts 2.5 / 5 / 10 / 20 / 40 / 60 ms. - /// PCM accepts any duration containing a whole number of samples. + /// PCM accepts any duration containing a whole number of samples. AAC frames + /// are 1024 samples, so it accepts the duration that rounds to that at + /// [`sample_rate`](Self::sample_rate), which [`from_input`](Self::from_input) + /// fills in. pub frame_duration: Duration, + /// Which encoder implementation to use. + pub kind: Kind, } impl Settings { @@ -127,6 +156,7 @@ impl Settings { bitrate: None, dtx: false, frame_duration: Duration::from_millis(20), + kind: Kind::Auto, } } @@ -134,13 +164,72 @@ impl Settings { pub fn from_input(codec: Codec, input: &Input) -> Self { let sample_rate = match codec { Codec::Opus => opus::pick_rate(input.sample_rate), - Codec::Pcm => input.sample_rate, + Codec::Pcm | Codec::Aac => input.sample_rate, + }; + let defaults = Self::new(sample_rate, input.layout); + let frame_duration = match codec { + Codec::Aac => aac_frame_duration(sample_rate), + Codec::Opus | Codec::Pcm => defaults.frame_duration, }; Self { codec, - sample_rate, - layout: input.layout, - ..Self::default() + frame_duration, + ..defaults + } + } + + /// Check the settings against the codec, returning its frame size. + /// + /// Codec rules live here rather than in a backend, so every backend of a + /// codec refuses the same settings. + fn frame_size(&self) -> Result { + self.layout.validate()?; + let (rate, channels) = (self.sample_rate, self.layout.channels()); + + match self.codec { + Codec::Opus => { + opus::validate_rate(rate)?; + if !matches!(self.layout, Layout::Mono | Layout::Stereo) { + return Err(Error::Unsupported("opus requires a named mono or stereo layout".into())); + } + opus::frame_size(rate, self.frame_duration) + } + Codec::Pcm => { + if self.bitrate.is_some() { + return Err(Error::Unsupported( + "pcm bitrate is fixed; leave Settings::bitrate unset".into(), + )); + } + if self.dtx { + return Err(Error::Unsupported( + "pcm does not support discontinuous transmission".into(), + )); + } + if rate == 0 { + return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); + } + let frame_size = pcm::frame_size(rate, self.frame_duration)?; + pcm::frame_bytes(frame_size, channels)?; + pcm::bitrate(rate, channels)?; + Ok(frame_size) + } + Codec::Aac => { + if self.dtx { + return Err(Error::Unsupported( + "aac does not support discontinuous transmission".into(), + )); + } + aac_config(self)?; + let frames = (self.frame_duration.as_nanos() * u128::from(rate) + 500_000_000) / 1_000_000_000; + if frames != AAC_FRAME_SIZE as u128 { + return Err(Error::Unsupported(format!( + "aac frames are {AAC_FRAME_SIZE} samples, {:?} at {rate} Hz (got {:?})", + aac_frame_duration(rate), + self.frame_duration + ))); + } + Ok(AAC_FRAME_SIZE) + } } } } @@ -151,6 +240,52 @@ impl Default for Settings { } } +/// One AAC frame at `sample_rate`, to the nearest nanosecond. +fn aac_frame_duration(sample_rate: u32) -> Duration { + if sample_rate == 0 { + return Duration::ZERO; + } + let rate = u64::from(sample_rate); + Duration::from_nanos((AAC_FRAME_SIZE as u64 * 1_000_000_000 + rate / 2) / rate) +} + +/// The AudioSpecificConfig fields for AAC-LC at the settings' rate and layout. +/// +/// Only layouts with a channelConfiguration are accepted, since synthesizing +/// one from a bare count would mislabel the rest: config 3 is 3.0 where the +/// count's default layout is 2.1, and 6.1 has no config the encoder writes. +/// 7.1 takes config 7, the one every decoder reads as eight channels. +fn aac_config(settings: &Settings) -> Result { + let layout = settings.layout; + if !matches!( + layout, + Layout::Mono + | Layout::Stereo + | Layout::ThreePointZero + | Layout::FourPointZero + | Layout::FivePointZero + | Layout::FivePointOne + | Layout::SevenPointOne + ) { + return Err(Error::Unsupported(format!( + "aac has no channelConfiguration for {layout:?}; use mono, stereo, 3.0, 4.0, 5.0, 5.1, or 7.1" + ))); + } + + let sample_rate = settings.sample_rate; + if !(1..=AAC_MAX_SAMPLE_RATE).contains(&sample_rate) { + return Err(Error::Unsupported(format!( + "aac sample rate must be between 1 and {AAC_MAX_SAMPLE_RATE} Hz (got {sample_rate})" + ))); + } + + Ok(moq_mux::codec::aac::Config { + profile: AAC_LC, + sample_rate, + channel_count: layout.channels(), + }) +} + /// Audio encoder over codec-sized interleaved `f32` PCM. /// /// Build one with [`Encoder::new`], feed full PCM frames via @@ -158,38 +293,16 @@ impl Default for Settings { /// [`finish`](Self::finish). Publish every packet either call returns and apply /// the terminal [`Finish::discard_padding`] when the container supports it. pub struct Encoder { - backend: Backend, + backend: Box, settings: Settings, - /// Codec sample rate. - codec_rate: u32, - /// Codec channel count. - codec_channels: u32, - /// Current libopus target bitrate. - bitrate: u64, - /// Encoder lookahead expressed in the OpusHead 48 kHz timebase. - pre_skip: u16, - /// Encoder lookahead in codec-rate frames. - lookahead: usize, frame_size: usize, + /// The catalog description, synthesized from the settings at construction so + /// the rendition can be registered before the first packet exists. + description: Option, /// Whether input has reached the codec, since a fresh encoder owes no drain. started: bool, } -enum Backend { - Opus(Opus), - Pcm, -} - -struct Opus { - inner: *mut OpusEncoder, - scratch: Vec, -} - -// SAFETY: OpusEncoder is heap-allocated state owned exclusively by this -// struct; libopus encoder methods take a single &mut, so a unique owner is -// allowed to move it across threads. -unsafe impl Send for Opus {} - /// Packets emitted by [`Encoder::finish`] and the decoded padding at their end. pub struct Finish { packets: Vec, @@ -214,158 +327,40 @@ impl Finish { } impl Encoder { - /// Open an encoder for `settings`. + /// Open an encoder for `settings`, refusing a codec no backend on this host + /// encodes. pub fn new(settings: &Settings) -> Result { - settings.layout.validate()?; - match settings.codec { - Codec::Opus => Self::new_opus(settings.clone()), - Codec::Pcm => Self::new_pcm(settings.clone()), - } - } - - fn new_opus(settings: Settings) -> Result { - let codec_rate = settings.sample_rate; - opus::validate_rate(codec_rate)?; + let frame_size = settings.frame_size()?; + let backend = backend::open(settings)?; - let codec_channels = settings.layout.channels(); - if !matches!(settings.layout, Layout::Mono | Layout::Stereo) { - return Err(Error::Unsupported("opus requires a named mono or stereo layout".into())); - } - let channels = opus::validate_channels(codec_channels)?; - - let frame_size = opus::frame_size(codec_rate, settings.frame_duration)?; - - let mut err = 0i32; - // SAFETY: out-pointer `err` is valid; inner is checked for null below. - let inner = unsafe { opus_encoder_create(codec_rate as i32, channels, OPUS_APPLICATION_AUDIO, &mut err) }; - if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_encoder_create")); - } - - let configured = Self::configure_opus(inner, &settings, codec_rate, codec_channels, frame_size); - let (bitrate, lookahead, pre_skip) = match configured { - Ok(configured) => configured, - Err(err) => { - // SAFETY: `inner` was created above and not yet handed out. - unsafe { opus_encoder_destroy(inner) }; - return Err(err); + let description = match settings.codec { + Codec::Opus => { + // OpusHead carries the lookahead in the 48 kHz timebase. + let lookahead = backend.delay() as u64; + let pre_skip = u16::try_from((lookahead * 48_000) / u64::from(settings.sample_rate)) + .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in OpusHead")))?; + let head = moq_mux::codec::opus::Config::new(settings.sample_rate, settings.layout.channels()) + .with_pre_skip(pre_skip) + .encode() + .map_err(moq_mux::Error::from)?; + Some(head) } + Codec::Aac => Some(aac_config(settings)?.encode()), + Codec::Pcm => None, }; Ok(Self { - backend: Backend::Opus(Opus { - inner, - scratch: vec![0u8; MAX_PACKET_BYTES], - }), - settings, - codec_rate, - codec_channels, - bitrate, - pre_skip, - lookahead, + backend, + settings: settings.clone(), frame_size, + description, started: false, }) } - fn new_pcm(settings: Settings) -> Result { - if settings.bitrate.is_some() { - return Err(Error::Unsupported( - "pcm bitrate is fixed; leave Settings::bitrate unset".into(), - )); - } - if settings.dtx { - return Err(Error::Unsupported( - "pcm does not support discontinuous transmission".into(), - )); - } - - let codec_rate = settings.sample_rate; - if codec_rate == 0 { - return Err(Error::Unsupported("pcm sample rate must be greater than zero".into())); - } - - let codec_channels = settings.layout.channels(); - if codec_channels == 0 { - return Err(Error::Unsupported("pcm channel count must be greater than zero".into())); - } - let frame_size = pcm::frame_size(codec_rate, settings.frame_duration)?; - pcm::frame_bytes(frame_size, codec_channels)?; - let bitrate = pcm::bitrate(codec_rate, codec_channels)?; - Ok(Self { - backend: Backend::Pcm, - settings, - codec_rate, - codec_channels, - bitrate, - pre_skip: 0, - lookahead: 0, - frame_size, - started: false, - }) - } - - fn configure_opus( - inner: *mut OpusEncoder, - settings: &Settings, - codec_rate: u32, - codec_channels: u32, - frame_size: usize, - ) -> Result<(u64, usize, u16), Error> { - if let Some(bitrate) = settings.bitrate { - Self::set_opus_bitrate(inner, codec_channels, bitrate.as_bps(), codec_rate, frame_size)?; - } - Self::set_opus_ctl(inner, OPUS_SET_DTX_REQUEST, i32::from(settings.dtx), "OPUS_SET_DTX")?; - - let bitrate = Self::get_opus_ctl(inner, OPUS_GET_BITRATE_REQUEST, "OPUS_GET_BITRATE")?; - let bitrate = u64::try_from(bitrate) - .map_err(|_| Error::Unsupported(format!("Opus reported negative bitrate {bitrate}")))?; - let lookahead = Self::get_opus_ctl(inner, OPUS_GET_LOOKAHEAD_REQUEST, "OPUS_GET_LOOKAHEAD")?; - let lookahead = u64::try_from(lookahead) - .map_err(|_| Error::Unsupported(format!("Opus reported negative lookahead {lookahead}")))?; - let pre_skip = u16::try_from((lookahead * 48_000) / codec_rate as u64) - .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in OpusHead")))?; - let lookahead = usize::try_from(lookahead) - .map_err(|_| Error::Unsupported(format!("Opus lookahead {lookahead} does not fit in memory")))?; - - Ok((bitrate, lookahead, pre_skip)) - } - - fn set_opus_bitrate( - inner: *mut OpusEncoder, - channels: u32, - bitrate: u64, - codec_rate: u32, - frame_size: usize, - ) -> Result<(), Error> { - let max = 300_000 * channels as u64; - let min = opus::bitrate_floor(codec_rate, frame_size).max(500); - if !(min..=max).contains(&bitrate) { - return Err(Error::Unsupported(format!( - "Opus bitrate must be between {min} and {max} bits per second for {channels} channel(s) at {frame_size} samples, got {bitrate}" - ))); - } - Self::set_opus_ctl(inner, OPUS_SET_BITRATE_REQUEST, bitrate as i32, "OPUS_SET_BITRATE") - } - - fn set_opus_ctl(inner: *mut OpusEncoder, request: i32, value: i32, name: &'static str) -> Result<(), Error> { - // SAFETY: `inner` owns a live encoder and each request here expects one i32. - let rc = unsafe { opus_encoder_ctl_impl(inner, request, varargs![value]) }; - if rc != OPUS_OK { - return Err(opus::error(rc, name)); - } - Ok(()) - } - - fn get_opus_ctl(inner: *mut OpusEncoder, request: i32, name: &'static str) -> Result { - let mut value = 0; - // SAFETY: `inner` owns a live encoder and each request here expects one - // valid mutable i32 output. - let rc = unsafe { opus_encoder_ctl_impl(inner, request, varargs![&mut value]) }; - if rc != OPUS_OK { - return Err(opus::error(rc, name)); - } - Ok(value) + /// The encoder backend name in use, e.g. `"libopus"`. + pub fn name(&self) -> &str { + self.backend.name() } /// The encoder settings, including the latest accepted runtime bitrate. @@ -382,13 +377,13 @@ impl Encoder { /// Sample rate the codec actually runs at, which is /// [`Settings::sample_rate`]. pub fn codec_rate(&self) -> u32 { - self.codec_rate + self.settings.sample_rate } /// Channel count the codec actually runs at, which is /// [`Settings::layout`]'s channel count. pub fn codec_channels(&self) -> u32 { - self.codec_channels + self.settings.layout.channels() } /// Number of samples per channel the codec consumes per call to @@ -399,23 +394,21 @@ impl Encoder { /// Current target bitrate. pub fn bitrate(&self) -> moq_net::bandwidth::Rate { - moq_net::bandwidth::Rate::from_bps(self.bitrate) + moq_net::bandwidth::Rate::from_bps(self.backend.bitrate()) } - /// Retune the live Opus encoder to `bitrate`. + /// Retune the live encoder to `bitrate`. + /// + /// # Errors + /// + /// Returns [`Error::Unsupported`] when the codec's rate is fixed (PCM) or the + /// backend can't change it mid-stream. The encoder keeps running at its + /// opening rate, so a caller driving a control loop should stop adapting + /// rather than stop encoding. pub fn set_bitrate(&mut self, bitrate: moq_net::bandwidth::Rate) -> Result<(), Error> { - let Backend::Opus(opus) = &mut self.backend else { - return Err(Error::Unsupported("pcm bitrate is fixed".into())); - }; - if bitrate.as_bps() != self.bitrate { - Self::set_opus_bitrate( - opus.inner, - self.codec_channels, - bitrate.as_bps(), - self.codec_rate, - self.frame_size, - )?; - self.bitrate = bitrate.as_bps(); + let previous = self.backend.bitrate(); + self.backend.set_bitrate(bitrate.as_bps())?; + if bitrate.as_bps() != previous { self.settings.bitrate = Some(bitrate); } Ok(()) @@ -423,11 +416,7 @@ impl Encoder { /// Drop all codec history so a later epoch cannot emit audio from this one. pub(super) fn reset(&mut self) { - if let Backend::Opus(opus) = &mut self.backend { - // SAFETY: `inner` owns a live encoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_encoder_ctl_impl(opus.inner, OPUS_RESET_STATE, varargs![]) }; - debug_assert_eq!(rc, OPUS_OK, "OPUS_RESET_STATE failed with {rc}"); - } + self.backend.reset(); self.started = false; } @@ -436,47 +425,34 @@ impl Encoder { self.started } + /// Codec priming the catalog can't signal, in codec-rate frames, which the + /// producer folds into its timestamps instead. + /// + /// Opus declares its lookahead as OpusHead pre-skip, which the decoder trims, + /// so nothing is folded. An AudioSpecificConfig has no such field, so each + /// AAC packet is stamped that much earlier and the priming lands before the + /// first input sample rather than delaying it. + pub(super) fn folded_delay(&self) -> usize { + match self.settings.codec { + Codec::Aac => self.backend.delay(), + Codec::Opus | Codec::Pcm => 0, + } + } + /// Encode one frame of interleaved `f32` PCM at [`codec_rate`](Self::codec_rate). /// /// `pcm.len()` must equal `frame_size() * codec_channels()`. The /// [`Producer`](super::Producer) handles format conversion and resampling /// before calling this; for direct use, the caller does the same. pub fn encode(&mut self, pcm: &[f32]) -> Result { - let expected = self.frame_size * self.codec_channels as usize; + let expected = self.frame_size * self.codec_channels() as usize; if pcm.len() != expected { return Err(Error::Misaligned { got: std::mem::size_of_val(pcm), expected: expected * std::mem::size_of::(), }); } - let encoded = match &mut self.backend { - Backend::Opus(opus) => { - // SAFETY: `inner` owns a live OpusEncoder; pcm and scratch slices - // are bounded by the lengths we pass. - let n = unsafe { - opus_encode_float( - opus.inner, - pcm.as_ptr(), - self.frame_size as i32, - opus.scratch.as_mut_ptr(), - opus.scratch.len() as i32, - ) - }; - if n < 0 { - return Err(crate::opus::error(n, "opus_encode_float")); - } - let payload = Bytes::copy_from_slice(&opus.scratch[..n as usize]); - let activity = crate::opus::activity(&payload, false); - Encoded { payload, activity } - } - Backend::Pcm => { - let mut payload = Vec::with_capacity(std::mem::size_of_val(pcm)); - for sample in pcm { - payload.extend_from_slice(&sample.to_le_bytes()); - } - Encoded::new(payload.into()) - } - }; + let encoded = self.backend.encode(pcm)?; self.started = true; Ok(encoded) } @@ -496,7 +472,7 @@ impl Encoder { /// Same drain as [`finish`](Self::finish), without consuming the encoder. pub(super) fn drain(&mut self, pcm: &[f32]) -> Result { - let channels = self.codec_channels as usize; + let channels = self.codec_channels() as usize; let frame_samples = self.frame_size * channels; if pcm.len() > frame_samples || !pcm.len().is_multiple_of(channels) { return Err(Error::Misaligned { @@ -529,7 +505,8 @@ impl Encoder { }); } - let drain = self.lookahead.saturating_sub(padding); + let lookahead = self.backend.delay(); + let drain = lookahead.saturating_sub(padding); let silence = vec![0.0; frame_samples]; for _ in 0..drain.div_ceil(self.frame_size) { packets.push(self.encode(&silence)?); @@ -538,7 +515,7 @@ impl Encoder { let discard_padding = packets .len() .saturating_mul(self.frame_size) - .saturating_sub(self.lookahead) + .saturating_sub(lookahead) .saturating_sub(source_frames); Ok(Finish { @@ -549,46 +526,27 @@ impl Encoder { /// hang catalog entry describing this encoder's output stream. pub fn catalog(&self) -> hang::catalog::AudioConfig { - match self.settings.codec { - Codec::Opus => { - // `codec_channels` is validated to mono/stereo at encoder construction, - // so the OpusHead (channel mapping family 0) always encodes. - let head = moq_mux::codec::opus::Config::new(self.codec_rate, self.codec_channels) - .with_pre_skip(self.pre_skip) - .encode() - .expect("opus encoder channels validated to mono/stereo"); - - let mut config = hang::catalog::AudioConfig::new( - hang::catalog::AudioCodec::Opus, - self.codec_rate, - self.codec_channels, - ); - config.bitrate = self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps); - config.description = Some(head); - config.container = hang::catalog::Container::Legacy; - config - } - Codec::Pcm => { - let mut config = hang::catalog::AudioConfig::new( - hang::catalog::AudioCodec::Pcm, - self.codec_rate, - self.codec_channels, - ); - config.bitrate = Some( - pcm::bitrate(self.codec_rate, self.codec_channels) - .expect("pcm encoder bitrate validated at construction"), - ); - config.container = hang::catalog::Container::Legacy; - config - } - } - } -} + let (rate, channels) = (self.codec_rate(), self.codec_channels()); + let (codec, bitrate): (hang::catalog::AudioCodec, _) = match self.settings.codec { + Codec::Opus => ( + hang::catalog::AudioCodec::Opus, + self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps), + ), + Codec::Pcm => ( + hang::catalog::AudioCodec::Pcm, + Some(pcm::bitrate(rate, channels).expect("pcm encoder bitrate validated at construction")), + ), + Codec::Aac => ( + hang::catalog::AAC { profile: AAC_LC }.into(), + self.settings.bitrate.map(moq_net::bandwidth::Rate::as_bps), + ), + }; -impl Drop for Opus { - fn drop(&mut self) { - // SAFETY: `inner` is a live OpusEncoder that nothing else aliases. - unsafe { opus_encoder_destroy(self.inner) }; + let mut config = hang::catalog::AudioConfig::new(codec, rate, channels); + config.bitrate = bitrate; + config.description = self.description.clone(); + config.container = hang::catalog::Container::Legacy; + config } } @@ -609,13 +567,6 @@ mod tests { out } - fn opus_inner(encoder: &Encoder) -> *mut OpusEncoder { - let Backend::Opus(opus) = &encoder.backend else { - panic!("expected Opus encoder"); - }; - opus.inner - } - #[test] fn opus_encode_then_decode_keeps_signal_close() { let mut enc = Encoder::new(&Settings { @@ -675,7 +626,6 @@ mod tests { let desc = cfg.description.expect("OpusHead should be present"); assert_eq!(desc.len(), 19); let head = moq_mux::codec::opus::Config::parse(&mut desc.as_ref()).unwrap(); - assert_eq!(head.pre_skip, enc.pre_skip); assert_eq!(head.pre_skip, 312); } @@ -688,7 +638,7 @@ mod tests { let first = dec.decode(&enc.encode(&frame).unwrap().payload).unwrap(); assert_eq!( first.samples.len(), - (enc.frame_size() - enc.pre_skip as usize) * enc.codec_channels() as usize + (enc.frame_size() - enc.backend.delay()) * enc.codec_channels() as usize ); let second = dec.decode(&enc.encode(&frame).unwrap().payload).unwrap(); @@ -757,15 +707,6 @@ mod tests { enc.set_bitrate(moq_net::bandwidth::Rate::from_bps(32_000)).unwrap(); assert_eq!(enc.bitrate(), moq_net::bandwidth::Rate::from_bps(32_000)); assert_eq!(enc.settings().bitrate, Some(moq_net::bandwidth::Rate::from_bps(32_000))); - assert_eq!( - Encoder::get_opus_ctl( - opus_inner(&enc), - unsafe_libopus::OPUS_GET_BITRATE_REQUEST, - "OPUS_GET_BITRATE" - ) - .unwrap(), - 32_000 - ); } #[test] @@ -777,20 +718,6 @@ mod tests { assert_eq!(enc.bitrate(), original); } - #[test] - fn opus_applies_dtx_control() { - let enc = Encoder::new(&Settings { - dtx: true, - ..Settings::default() - }) - .unwrap(); - - assert_eq!( - Encoder::get_opus_ctl(opus_inner(&enc), unsafe_libopus::OPUS_GET_DTX_REQUEST, "OPUS_GET_DTX").unwrap(), - 1 - ); - } - #[test] fn codec_roundtrips_as_str() { assert_eq!(Codec::Opus.as_str(), "opus"); @@ -799,15 +726,20 @@ mod tests { assert_eq!(Codec::Pcm.as_str(), "pcm"); assert_eq!(Codec::Pcm.to_string(), "pcm"); assert_eq!("pcm".parse::().unwrap(), Codec::Pcm); - assert!("aac".parse::().is_err()); + assert_eq!(Codec::Aac.as_str(), "aac"); + assert_eq!(Codec::Aac.to_string(), "aac"); + assert_eq!("aac".parse::().unwrap(), Codec::Aac); + assert!("mp3".parse::().is_err()); } #[test] fn settings_fix_the_codec_rate() { let enc = Encoder::new(&Settings::new(24_000, Layout::Mono)).unwrap(); assert_eq!(enc.codec_rate(), 24_000); - assert_eq!(enc.catalog().sample_rate, 24_000); - assert_eq!(enc.pre_skip, 312); + let catalog = enc.catalog(); + assert_eq!(catalog.sample_rate, 24_000); + let head = moq_mux::codec::opus::Config::parse(&mut catalog.description.unwrap().as_ref()).unwrap(); + assert_eq!(head.pre_skip, 312); } #[test] @@ -906,4 +838,121 @@ mod tests { let settings = Settings::new(48_000, Layout::Discrete(2)); assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); } + + /// AAC settings routed to the test stub, since this host has no AAC encoder. + fn aac(layout: Layout) -> Settings { + Settings { + kind: Kind::Named(backend::stub::NAME.into()), + ..Settings::from_input(Codec::Aac, &Input::new(48_000, layout)) + } + } + + /// The ASC is synthesized from the settings, so it exists before any packet. + #[test] + fn aac_catalog_carries_the_synthesized_asc() { + let enc = Encoder::new(&aac(Layout::Stereo)).unwrap(); + assert_eq!(enc.name(), backend::stub::NAME); + assert_eq!(enc.frame_size(), 1024); + + let catalog = enc.catalog(); + assert_eq!(catalog.codec, hang::catalog::AAC { profile: 2 }.into()); + assert_eq!(catalog.codec.to_string(), "mp4a.40.2"); + assert_eq!(catalog.sample_rate, 48_000); + assert_eq!(catalog.channel_count, 2); + assert_eq!(catalog.container, hang::catalog::Container::Legacy); + // AAC-LC (2), 48 kHz (index 3), stereo (config 2). + assert_eq!(catalog.description.as_deref(), Some(&[0x11, 0x90][..])); + } + + #[test] + fn aac_takes_the_layouts_with_a_channel_configuration() { + for (layout, config) in [ + (Layout::Mono, 1), + (Layout::Stereo, 2), + (Layout::ThreePointZero, 3), + (Layout::FourPointZero, 4), + (Layout::FivePointZero, 5), + (Layout::FivePointOne, 6), + (Layout::SevenPointOne, 7), + ] { + let catalog = Encoder::new(&aac(layout)).unwrap().catalog(); + let description = catalog.description.unwrap(); + assert_eq!(description[1] >> 3 & 0xF, config, "{layout:?}"); + assert_eq!(catalog.channel_count, layout.channels(), "{layout:?}"); + } + + for layout in [ + Layout::TwoPointOne, + Layout::Quad, + Layout::SixPointOne, + Layout::Discrete(2), + ] { + assert!( + matches!(Encoder::new(&aac(layout)), Err(Error::Unsupported(_))), + "{layout:?}" + ); + } + } + + /// AAC frames are 1024 samples however the duration is spelled. + #[test] + fn aac_frame_duration_is_the_codecs() { + assert_eq!(aac(Layout::Stereo).frame_duration, Duration::from_nanos(21_333_333)); + let settings = Settings { + frame_duration: Duration::from_micros(21_333), + ..aac(Layout::Stereo) + }; + assert_eq!(Encoder::new(&settings).unwrap().frame_size(), 1024); + + let settings = Settings { + frame_duration: Duration::from_millis(20), + ..aac(Layout::Stereo) + }; + let err = Encoder::new(&settings).err().expect("20 ms is 960 samples"); + assert!(err.to_string().contains("1024"), "{err}"); + } + + #[test] + fn aac_refuses_dtx() { + let settings = Settings { + dtx: true, + ..aac(Layout::Stereo) + }; + assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); + } + + /// A backend that can't retune keeps its opening rate. + #[test] + fn fixed_rate_backend_keeps_its_opening_rate() { + let mut enc = Encoder::new(&Settings { + bitrate: Some(moq_net::bandwidth::Rate::from_bps(96_000)), + ..aac(Layout::Stereo) + }) + .unwrap(); + + let err = enc.set_bitrate(moq_net::bandwidth::Rate::from_bps(64_000)); + assert!(matches!(err, Err(Error::Unsupported(_)))); + assert_eq!(enc.bitrate(), moq_net::bandwidth::Rate::from_bps(96_000)); + assert_eq!(enc.settings().bitrate, Some(moq_net::bandwidth::Rate::from_bps(96_000))); + assert_eq!(enc.catalog().bitrate, Some(96_000)); + } + + /// The drain pushes the encoder delay out through whole silent frames. + #[test] + fn aac_finish_drains_the_encoder_delay() { + let mut enc = Encoder::new(&aac(Layout::Mono)).unwrap(); + assert_eq!(enc.folded_delay(), backend::stub::DELAY); + enc.encode(&[0.0; 1024]).unwrap(); + + // 2112 frames of delay take three 1024-frame packets. + let finish = enc.finish(&[]).unwrap(); + assert_eq!(finish.packets().len(), 3); + assert_eq!(finish.discard_padding(), 3 * 1024 - backend::stub::DELAY); + } + + /// Opus signals its lookahead as pre-skip, so the producer folds none of it. + #[test] + fn opus_folds_no_delay() { + assert_eq!(Encoder::new(&Settings::default()).unwrap().folded_delay(), 0); + } } diff --git a/rs/moq-audio/src/encode/mod.rs b/rs/moq-audio/src/encode/mod.rs index 9c918b97445..10c3125e416 100644 --- a/rs/moq-audio/src/encode/mod.rs +++ b/rs/moq-audio/src/encode/mod.rs @@ -1,6 +1,8 @@ //! Encode raw PCM and publish it as a moq audio track. //! -//! The output codec is selected via [`Codec`]. +//! The output codec is selected via [`Codec`], and the implementation behind it +//! via [`Kind`]: a platform encoder when the host has one, software otherwise. +//! AAC has no software encoder, so a host without a platform one refuses it. //! //! Entry points, high to low level: //! - `publish_capture` captures a microphone (or system audio) and publishes @@ -15,6 +17,7 @@ //! `publish_capture` is unlinked above because it only exists with the `capture` //! feature, so a default-feature rustdoc build has nothing to link to. +mod backend; mod encoded; mod encoder; mod producer; @@ -23,7 +26,7 @@ mod producer; mod capture; pub use encoded::Encoded; -pub use encoder::{Codec, Encoder, Finish, Input, Settings}; +pub use encoder::{Codec, Encoder, Finish, Input, Kind, Settings}; pub use producer::{Options, Producer}; #[cfg(feature = "capture")] diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index 1b61545141b..afb0b737221 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -366,7 +366,12 @@ impl Producer { let chunk: Vec = self.pending.drain(..frame_samples).collect(); let packet = self.encoder.encode(&chunk)?; - let timestamp = Self::timestamp(epoch_us, self.frames_produced, self.encoder.codec_rate())?; + let timestamp = Self::timestamp( + epoch_us, + self.frames_produced, + self.encoder.folded_delay(), + self.encoder.codec_rate(), + )?; self.frames_produced += self.encoder.frame_size() as u64; self.activity = packet.activity; Self::publish(&mut self.track, packet, timestamp)?; @@ -376,10 +381,18 @@ impl Producer { Ok(()) } - /// PTS of the next frame: the epoch plus the samples emitted since it. - fn timestamp(epoch_us: u64, frames_produced: u64, codec_rate: u32) -> Result { - let offset_us = (frames_produced * 1_000_000) / codec_rate as u64; - Ok(Timestamp::from_micros(epoch_us + offset_us)?) + /// PTS of the frame `frames` samples past the epoch, stamped `delay` samples + /// earlier to fold in codec priming the catalog can't signal. + /// + /// Priming that would land before a zero epoch is stamped at zero instead: it + /// decodes to the codec's warm-up rather than to input, so only its spacing is + /// lost. + fn timestamp(epoch_us: u64, frames: u64, delay: usize, codec_rate: u32) -> Result { + let frames = i128::from(frames) - delay as i128; + let offset_us = (frames * 1_000_000).div_euclid(i128::from(codec_rate)); + let micros = (i128::from(epoch_us) + offset_us).max(0); + let micros = u64::try_from(micros).map_err(|_| moq_net::TimeOverflow)?; + Ok(Timestamp::from_micros(micros)?) } fn publish( @@ -474,8 +487,10 @@ impl Producer { let codec_rate = self.encoder.codec_rate(); let channels = self.encoder.codec_channels() as usize; let source_frames = self.pending.len() / channels; - let start = Self::timestamp(epoch_us, self.frames_produced, codec_rate)?; - let end = Self::timestamp(epoch_us, self.frames_produced + source_frames as u64, codec_rate)?; + let delay = self.encoder.folded_delay(); + let start = Self::timestamp(epoch_us, self.frames_produced, delay, codec_rate)?; + // The source ends where it ends: priming only moves the packets carrying it. + let end = Self::timestamp(epoch_us, self.frames_produced + source_frames as u64, 0, codec_rate)?; let finish = self.encoder.drain(&self.pending)?; let discard_padding = finish.discard_padding(); let packets = finish.into_packets(); @@ -493,7 +508,7 @@ impl Producer { )?; } else { for packet in packets { - let timestamp = Self::timestamp(epoch_us, self.frames_produced, codec_rate)?; + let timestamp = Self::timestamp(epoch_us, self.frames_produced, delay, codec_rate)?; self.activity = packet.activity; Self::publish(&mut self.track, packet, timestamp)?; self.frames_produced += frame_size as u64; @@ -898,6 +913,51 @@ mod tests { assert_eq!(pts, vec![1_000_000]); } + /// AAC can't signal its encoder delay, so each packet is stamped that much + /// earlier and the first input sample still decodes at the epoch. + #[tokio::test] + async fn aac_folds_the_encoder_delay_into_timestamps() { + async fn pts(epoch_us: u64) -> Vec { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let consumer = broadcast.consume(); + + let input = Input::new(48_000, Layout::Mono); + let options = Options { + track: Some("audio".to_string()), + settings: Settings { + kind: crate::encode::Kind::Named(crate::encode::backend::stub::NAME.into()), + ..Settings::from_input(crate::encode::Codec::Aac, &input) + }, + ..Options::default() + }; + let mut producer = Producer::new(&mut broadcast, catalog, input, &options).unwrap(); + + let track = consumer + .track("audio") + .unwrap() + .subscribe(moq_net::track::Subscription::default().with_max_age(Duration::from_secs(1))) + .await + .unwrap(); + let mut reader = moq_mux::container::Consumer::new( + track, + moq_mux::container::legacy::Wire(moq_mux::container::Kind::Audio), + ); + + producer.write(&pcm_frame(&[0.1; 4 * 1024], epoch_us)).unwrap(); + let mut pts = Vec::new(); + for _ in 0..4 { + pts.push(reader.read().await.unwrap().expect("a packet").timestamp.as_micros()); + } + pts + } + + // 2112 frames of delay at 48 kHz is 44 ms, rounded down per packet. + assert_eq!(pts(1_000_000).await, vec![956_000, 977_333, 998_666, 1_020_000]); + // Priming before a zero epoch stamps at zero; the input still starts on time. + assert_eq!(pts(0).await, vec![0, 0, 0, 20_000]); + } + /// The encoder needs no correction for the resampler's own delay: it anchors /// the epoch to the first input timestamp and advances by emitted samples, /// while `Resampler::process` drops its startup silence rather than passing it diff --git a/rs/moq-audio/src/lib.rs b/rs/moq-audio/src/lib.rs index b8c7aa92428..92d1c04dfff 100644 --- a/rs/moq-audio/src/lib.rs +++ b/rs/moq-audio/src/lib.rs @@ -3,8 +3,9 @@ //! Counterpart to [`moq-video`](https://crates.io/crates/moq-video) for audio //! tracks, and shaped the same way. Sits on top of [`moq_mux`] and [`hang`] and //! adds the missing piece for native callers: Rust-native Opus and uncompressed -//! PCM codecs that turn raw samples into HANG audio tracks and back, plus AAC-LC -//! on the way in, which is what the gateways (RTMP, SRT, HLS, gstreamer) publish. +//! PCM codecs that turn raw samples into HANG audio tracks and back, plus AAC-LC, +//! which is what the gateways (RTMP, SRT, HLS, gstreamer) publish. AAC decodes in +//! software and encodes only through a platform encoder. //! //! - `capture` describes an audio source (`capture::Config`) and grabs buffers //! per platform: a microphone via cpal (CoreAudio / WASAPI / ALSA) everywhere, @@ -26,7 +27,7 @@ //! - [`decode`] subscribes to an encoded track and decodes it back to PCM. //! [`decode::Consumer`] is the mirror of [`encode::Producer`]. It reads AAC-LC //! too, behind the default-on `aac` feature, since a broadcast that came in -//! through a gateway is AAC rather than one of the two codecs we encode. +//! through a gateway is AAC rather than one of the codecs we encode in software. //! - `playback` plays decoded PCM out a speaker. `playback::Engine` owns the //! output device and mixes the `playback::Sink`s registered with it, so one //! device serves every track in a call. Requires the `playback` feature, so diff --git a/rs/moq-ffi/src/audio.rs b/rs/moq-ffi/src/audio.rs index 5e8bea78ce4..22c3ba47428 100644 --- a/rs/moq-ffi/src/audio.rs +++ b/rs/moq-ffi/src/audio.rs @@ -48,7 +48,7 @@ impl From for moq_audio::Format { /// Audio codec selection for the encoder. /// /// An immutable object so adding a codec later does not break callers -/// switching over a closed enum. Currently only Opus is available. +/// switching over a closed enum. #[derive(uniffi::Object)] pub struct MoqAudioCodec { inner: moq_audio::encode::Codec, @@ -63,6 +63,16 @@ impl MoqAudioCodec { inner: moq_audio::encode::Codec::Opus, }) } + + /// AAC-LC (`mp4a.40.2`) through the platform's encoder, at the input's rate + /// and layout. A host without one refuses it when the producer is built. + /// Its frames are 1024 samples, so leave `frame_duration_us` at 0. + #[uniffi::constructor] + pub fn aac() -> Arc { + Arc::new(Self { + inner: moq_audio::encode::Codec::Aac, + }) + } } impl MoqAudioCodec { @@ -96,7 +106,7 @@ pub struct MoqAudioEncoderOutput { pub bitrate: Option, /// Encoded frame duration in microseconds. Opus accepts exactly /// 2500/5000/10000/20000/40000/60000 us, and the default 20 ms matches the - /// JS publish path. + /// JS publish path. 0 takes the codec's own frame, which AAC needs. #[uniffi(default = 20000)] pub frame_duration_us: u32, } @@ -290,7 +300,16 @@ impl MoqBroadcastProducer { options.settings.layout = moq_audio::Layout::from_channels(channels)?; } options.settings.bitrate = output.bitrate.map(|bps| moq_net::bandwidth::Rate::from_bps(bps.into())); - options.settings.frame_duration = Duration::from_micros(output.frame_duration_us.into()); + if output.frame_duration_us != 0 { + options.settings.frame_duration = Duration::from_micros(output.frame_duration_us.into()); + } else if output.codec.codec() == moq_audio::encode::Codec::Aac { + // from_input sized this at the input rate. The codec rate may be the + // override above, and AAC's own frame is 1024 samples of that rate. + let mut rated = input.clone(); + rated.sample_rate = options.settings.sample_rate; + options.settings.frame_duration = + moq_audio::encode::Settings::from_input(moq_audio::encode::Codec::Aac, &rated).frame_duration; + } if let Some(bandwidth) = &bandwidth { options.bandwidth = bandwidth.allocator().clone(); } diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 265dee6162e..01f10f0e996 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -512,6 +512,60 @@ async fn raw_audio_frame_durations() { broadcast.finish().unwrap(); } +/// A frame duration of 0 takes the codec's own frame, and AAC, which encodes only +/// through a platform encoder, is refused where there is none. +#[cfg(feature = "audio")] +#[tokio::test] +async fn raw_audio_codec_default_frame() { + use crate::audio::*; + + let broadcast = MoqBroadcastProducer::new().unwrap(); + let input = || MoqAudioEncoderInput { + format: MoqAudioSampleFormat::F32, + sample_rate: 48_000, + channels: 2, + }; + let output = |codec| MoqAudioEncoderOutput { + codec, + sample_rate: None, + channels: None, + bitrate: None, + frame_duration_us: 0, + }; + + let opus = broadcast + .encode_audio("opus".into(), input(), output(MoqAudioCodec::opus()), None) + .unwrap(); + opus.finish().unwrap(); + + assert_eq!(MoqAudioCodec::aac().codec(), moq_audio::encode::Codec::Aac); + let aac = broadcast.encode_audio("aac".into(), input(), output(MoqAudioCodec::aac()), None); + let Err(MoqError::Audio(message)) = aac else { + panic!("no platform AAC encoder on this host"); + }; + assert!(message.contains("no aac audio encoder"), "{message}"); + + // 0 still means 1024 samples after an output-rate override, not the input rate. + let mut resampled = output(MoqAudioCodec::aac()); + resampled.sample_rate = Some(48_000); + let aac = broadcast.encode_audio( + "aac-rate".into(), + MoqAudioEncoderInput { + format: MoqAudioSampleFormat::F32, + sample_rate: 44_100, + channels: 2, + }, + resampled, + None, + ); + let Err(MoqError::Audio(message)) = aac else { + panic!("no platform AAC encoder on this host"); + }; + assert!(message.contains("no aac audio encoder"), "{message}"); + + broadcast.finish().unwrap(); +} + #[tokio::test] async fn raw_track_datagram_roundtrip() { let broadcast = MoqBroadcastProducer::new().unwrap(); diff --git a/swift/Sources/Moq/Aliases.swift b/swift/Sources/Moq/Aliases.swift index dcfef7b25ea..c864511cfff 100644 --- a/swift/Sources/Moq/Aliases.swift +++ b/swift/Sources/Moq/Aliases.swift @@ -52,7 +52,7 @@ public typealias VideoDecodedFrame = MoqVideoDecodedFrame public typealias AudioDecoderOutput = MoqFFI.MoqAudioDecoderOutput /// A raw PCM sample format, mirroring WebCodecs `AudioData.format`. public typealias AudioSampleFormat = MoqFFI.MoqAudioSampleFormat -/// Selects the audio encoder codec. Build one with `AudioCodec.opus()`. +/// Selects the audio encoder codec. Build one with `AudioCodec.opus()` or `AudioCodec.aac()`. public typealias AudioCodec = MoqFFI.MoqAudioCodec /// One raw video frame: pixels in the configured layout plus a presentation /// timestamp. diff --git a/swift/Sources/Moq/Broadcast.swift b/swift/Sources/Moq/Broadcast.swift index cf52ddba1b7..85487ed175e 100644 --- a/swift/Sources/Moq/Broadcast.swift +++ b/swift/Sources/Moq/Broadcast.swift @@ -273,7 +273,7 @@ public final class BroadcastProducer: Sendable { /// Open a raw-audio track. PCM written via `AudioProducer.write` is encoded /// inside the FFI boundary per `input`/`output`. Select the codec with - /// `AudioCodec.opus()` (currently the only constructor), placed in `output`. + /// `AudioCodec.opus()` or `AudioCodec.aac()`, placed in `output`. /// /// Pass `bandwidth` to reserve this track's bitrate against the session's /// allocator so a co-resident video encoder sizes itself against what is left. From 0e8a5e8fe9f304822d6e8d4f677c85eeb82572d3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 17:12:12 -0700 Subject: [PATCH 08/20] quest(audio-codecs): ADTS export refuses what it cannot label (#4202) Co-authored-by: Claude Opus 5.5 --- quest/m1/audio-codecs/README.md | 1 + quest/m1/audio-codecs/adts-refusals.md | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+) create mode 100644 quest/m1/audio-codecs/adts-refusals.md diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index ee3cca548fd..b298d3560f8 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -45,6 +45,7 @@ its own decode and encode quest so verification stays per host. - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder +- [ADTS refusals](/quest/m1/audio-codecs/adts-refusals.md) - the ADTS writer refuses channel counts and object types it cannot label instead of mislabeling them - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC ## Related diff --git a/quest/m1/audio-codecs/adts-refusals.md b/quest/m1/audio-codecs/adts-refusals.md new file mode 100644 index 00000000000..c7896e377d7 --- /dev/null +++ b/quest/m1/audio-codecs/adts-refusals.md @@ -0,0 +1,25 @@ +# [S] ADTS export refuses what it cannot label + +## Goal + +`moq-mux`'s ADTS writer never labels an AAC track with the wrong layout or +object type. A channel count ADTS cannot name is refused, not silently written +as stereo, and an explicit SBR or PS description is written with the object +type ADTS can carry or refused, not masked to two bits into a wrong profile. + +## Plan + +- `channel_config_from_count` falls back to stereo for counts it cannot + represent; refuse instead, the same way #4178 made channelConfiguration 11 to + 14 refuse. +- The ADTS header masks the object type to two bits, so an explicit HE-AAC + (object type 5) or HE-AACv2 (29) description is mislabeled. Decide per case: + signal the backward-compatible AAC-LC core (implicit SBR) when the description + allows it, otherwise refuse. +- Tests for both with real fixtures, checked against ffprobe. + +Public API: none. Wire: none; TS output changes only for inputs it mislabeled. + +## Related + +- [#4178](https://github.com/moq-dev/moq/pull/4178) - the PCE export that found these From 7824c89e37bf5e9aabe6fa136d3a26574cff936d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 21:00:55 -0700 Subject: [PATCH 09/20] feat(audio): decode surround Opus through the multistream decoder (#4177) Co-authored-by: Claude Opus 5.5 Co-authored-by: Grok 4.7 --- doc/lib/rs/moq-audio.md | 6 +- quest/m1/audio-codecs/README.md | 2 +- quest/m1/audio-codecs/opus-surround.md | 22 -- quest/m1/audio-codecs/ts-opus-surround.md | 20 ++ quest/m1/audio-opus-input.md | 4 - rs/moq-audio/src/decode/backend/libopus.rs | 341 +++++++++++++++++++-- rs/moq-audio/src/opus.rs | 199 +++++++++++- rs/moq-mux/src/catalog/msf/consumer.rs | 3 +- rs/moq-mux/src/codec/opus/import.rs | 5 +- rs/moq-mux/src/codec/opus/mod.rs | 206 ++++++++++++- 10 files changed, 743 insertions(+), 65 deletions(-) delete mode 100644 quest/m1/audio-codecs/opus-surround.md create mode 100644 quest/m1/audio-codecs/ts-opus-surround.md diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index c76dc1179e7..13078570e08 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -39,14 +39,16 @@ first, then software. `decode::Config::kind` forces one (`Kind::Software`, or | Backend | Decodes | Hosts | | --- | --- | --- | -| `libopus` | Opus, mono or stereo | all | +| `libopus` | Opus, mono or stereo, and surround up to 7.1 (channel mapping family 1) | all | | `pcm` | PCM | all | | `symphonia` | AAC-LC, mono or stereo (the default-on `aac` feature) | all | No platform decoder is wired in yet, so multichannel AAC and HE-AAC declared in its config are refused at construction on every host. HE-AAC signaled only in band plays as its half-rate LC core. Linux has no OS audio decoder, so it will -stay that way there. +stay that way there. Surround Opus is pure Rust, so it is the one multichannel +path every host has. Opus mapping families other than 0 and 1 (ambisonics, and +255's unpositioned channels) are refused, since they declare no speakers. `encode` selects the same way, through `encode::Settings::kind`, and `Encoder::name()` reports what opened. diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index b298d3560f8..71b47defcc5 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,9 +44,9 @@ its own decode and encode quest so verification stays per host. ## Quests - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers -- [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - mapping family 1 decodes on every host through the multistream decoder - [ADTS refusals](/quest/m1/audio-codecs/adts-refusals.md) - the ADTS writer refuses channel counts and object types it cannot label instead of mislabeling them - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC +- [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes ## Related diff --git a/quest/m1/audio-codecs/opus-surround.md b/quest/m1/audio-codecs/opus-surround.md deleted file mode 100644 index 059552dddef..00000000000 --- a/quest/m1/audio-codecs/opus-surround.md +++ /dev/null @@ -1,22 +0,0 @@ -# [S] Opus surround through the multistream decoder - -## Goal - -An Opus track with channel mapping family 1 (up to 7.1, RFC 7845 §5.1.1) -decodes on every host, delivered in the canonical `Layout` order. Pure Rust, -so this is the one multichannel path Linux gets. - -## Plan - -- `moq_mux::codec::opus::Config` parses the channel mapping (family, stream - count, coupled count, and the mapping table) instead of skipping it. Family - 0 stays mono/stereo; family 1 maps to a `Layout` by channel count; family - 255 and unknown families are refused, since they carry no speaker - assignment. -- The Opus decode backend opens `opus_multistream_decoder_create` from - `unsafe-libopus` when the family is 1, and reorders Vorbis order into the - canonical one on the way out. -- Encode stays family 0; `Config::encode` keeps refusing more than two - channels. -- Regression: a family-1 5.1 fixture decodes to six channels in canonical - order; family 255 is refused at construction. diff --git a/quest/m1/audio-codecs/ts-opus-surround.md b/quest/m1/audio-codecs/ts-opus-surround.md new file mode 100644 index 00000000000..5e6e6be4fc3 --- /dev/null +++ b/quest/m1/audio-codecs/ts-opus-surround.md @@ -0,0 +1,20 @@ +# [S] Surround Opus from MPEG-TS + +## Goal + +An MPEG-TS Opus stream with 3 to 8 channels imports with an OpusHead that +decodes, so it plays natively like the same stream from Matroska or FLV. + +## Plan + +- The TS importer knows only the `channel_config_code` from the Opus + extension descriptor and builds `opus::Config::new(48_000, channels)`, whose + `encode` refuses more than two channels. The catalog then carries no + description and `moq-audio` refuses the track. +- Codes 3 to 8 mean family 1 with the Vorbis default stream and coupled counts + and mapping (the Vorbis orders of RFC 7845 §5.1.1.2), so the + importer can synthesize the head. That needs a way to build a family 1 + `Config` and have `encode` emit its table; decide whether that is a + constructor on `opus::Mapping` or a TS-local head writer. +- Regression: a 5.1 TS fixture imports with a family 1 description and decodes + to six channels. diff --git a/quest/m1/audio-opus-input.md b/quest/m1/audio-opus-input.md index 9a6b0927b50..c98e60889aa 100644 --- a/quest/m1/audio-opus-input.md +++ b/quest/m1/audio-opus-input.md @@ -22,7 +22,3 @@ real Opus fixtures in CI. Verify timestamps as well as decoded sample counts. Public API and wire schema: unchanged. Update the existing codec documentation with supported/refused cases; header-parser changes stay focused on this path. - -## Related - -- [Opus surround](/quest/m1/audio-codecs/opus-surround.md) - adds supported mappings separately diff --git a/rs/moq-audio/src/decode/backend/libopus.rs b/rs/moq-audio/src/decode/backend/libopus.rs index 0fa9f78b69d..0d66970c62a 100644 --- a/rs/moq-audio/src/decode/backend/libopus.rs +++ b/rs/moq-audio/src/decode/backend/libopus.rs @@ -1,8 +1,11 @@ //! Opus through libopus, the software decoder for every host. +//! +//! Every stream goes through the multistream decoder, which is also how a +//! family 0 (mono/stereo) stream decodes: one stream, coupled when stereo. use unsafe_libopus::{ - OPUS_OK, OPUS_RESET_STATE, OpusDecoder, opus_decode_float, opus_decoder_create, opus_decoder_ctl_impl, - opus_decoder_destroy, varargs, + OPUS_OK, OPUS_RESET_STATE, OpusMSDecoder, opus_multistream_decode_float, opus_multistream_decoder_create, + opus_multistream_decoder_ctl_impl, opus_multistream_decoder_destroy, varargs, }; use super::Backend; @@ -14,10 +17,18 @@ pub(super) const NAME: &str = "libopus"; /// Opus packets cap at 120 ms (RFC 6716 §2.1.4). const MAX_FRAME_MS: usize = 120; +/// The family 0 streams: one stream, coupled when stereo, decoded in order. +const MONO: &[u8] = &[0]; +const STEREO: &[u8] = &[0, 1]; + pub(super) struct Libopus { - inner: *mut OpusDecoder, + inner: *mut OpusMSDecoder, sample_rate: u32, layout: Layout, + /// For each canonical channel, the Vorbis-order channel it comes from, when they differ. + reorder: Option<&'static [usize]>, + /// Opus streams in one packet. Family 0 is one; the rest come from the mapping. + streams: u8, pre_skip: usize, max_frame_size: usize, in_dtx: bool, @@ -27,34 +38,67 @@ pub(super) struct Libopus { unsafe impl Send for Libopus {} impl Libopus { - /// Parses the OpusHead `description` if present; falls back to the catalog's - /// declared sample rate / channel count. + /// Parses the OpusHead `description` when one is present. A missing description + /// falls back to the catalog's sample rate and channel count, which must then + /// be mono or stereo. A description that does not parse is refused: guessing + /// family 0 would decode those packets with the wrong stream layout. + /// + /// Channel mapping family 1 decodes up to 7.1 in the canonical [`Layout`] + /// order; every other family is refused, since none declares speakers. pub(super) fn open(catalog: &hang::catalog::AudioConfig) -> Result, Error> { - let (sample_rate, channel_count, pre_skip) = if let Some(desc) = &catalog.description { - let mut buf = desc.as_ref(); - match moq_mux::codec::opus::Config::parse(&mut buf) { - Ok(head) => (head.sample_rate, head.channel_count, head.pre_skip), - Err(_) => (catalog.sample_rate, catalog.channel_count, 0), - } - } else { - (catalog.sample_rate, catalog.channel_count, 0) + let head = match catalog.description.as_ref() { + Some(desc) => moq_mux::codec::opus::Config::parse(&mut desc.as_ref()) + .map_err(|err| Error::Unsupported(format!("opus description: {err}")))?, + None => moq_mux::codec::opus::Config::new(catalog.sample_rate, catalog.channel_count), }; + let (sample_rate, channel_count, pre_skip) = (head.sample_rate, head.channel_count, head.pre_skip); opus::validate_rate(sample_rate)?; - let channels = opus::validate_channels(channel_count)?; - let layout = Layout::from_channels(channel_count)?; + let (streams, coupled, table, layout, reorder) = match &head.mapping { + None => { + let table = match opus::validate_channels(channel_count)? { + 1 => MONO, + _ => STEREO, + }; + let layout = Layout::from_channels(channel_count)?; + (1, table.len() as i32 - 1, table, layout, None) + } + Some(mapping) if mapping.family() == 1 => { + let (layout, reorder) = vorbis(channel_count)?; + let (streams, coupled) = (mapping.streams() as i32, mapping.coupled() as i32); + (streams, coupled, mapping.table(), layout, reorder) + } + Some(mapping) => { + return Err(Error::Unsupported(format!( + "opus channel mapping family {} declares no speaker positions", + mapping.family() + ))); + } + }; let mut err = 0i32; - // SAFETY: out-pointer is valid; inner is checked for null below. - let inner = unsafe { opus_decoder_create(sample_rate as i32, channels, &mut err) }; + // SAFETY: `table` holds one entry per output channel, the count we pass, + // and the out-pointer is valid; inner is checked for null below. + let inner = unsafe { + opus_multistream_decoder_create( + sample_rate as i32, + table.len() as i32, + streams, + coupled, + table.as_ptr(), + &mut err, + ) + }; if err != OPUS_OK || inner.is_null() { - return Err(opus::error(err, "opus_decoder_create")); + return Err(opus::error(err, "opus_multistream_decoder_create")); } Ok(Box::new(Self { inner, sample_rate, layout, + reorder, + streams: streams as u8, // OpusHead counts pre-skip at 48 kHz whatever rate the decoder runs at. pre_skip: (pre_skip as usize * sample_rate as usize) / 48_000, max_frame_size: (sample_rate as usize * MAX_FRAME_MS) / 1000, @@ -63,17 +107,47 @@ impl Libopus { } } +/// The layout a family 1 stream of `channels` declares (RFC 7845 §5.1.1.2), and +/// how to reorder its Vorbis channel order into that layout's canonical one. +/// +/// Vorbis puts the center between the fronts and the LFE last. Its "rear" +/// pair in 5.0 and 5.1 is the surround pair, which is side in the canonical +/// layouts, while 7.1 has distinct side and rear pairs. +fn vorbis(channels: u32) -> Result<(Layout, Option<&'static [usize]>), Error> { + Ok(match channels { + 1 => (Layout::Mono, None), + 2 => (Layout::Stereo, None), + // L, C, R. + 3 => (Layout::ThreePointZero, Some(&[0, 2, 1])), + // FL, FR, RL, RR. + 4 => (Layout::Quad, None), + // FL, C, FR, RL, RR. + 5 => (Layout::FivePointZero, Some(&[0, 2, 1, 3, 4])), + // FL, C, FR, RL, RR, LFE. + 6 => (Layout::FivePointOne, Some(&[0, 2, 1, 5, 3, 4])), + // FL, C, FR, SL, SR, RC, LFE. + 7 => (Layout::SixPointOne, Some(&[0, 2, 1, 6, 5, 3, 4])), + // FL, C, FR, SL, SR, RL, RR, LFE. + 8 => (Layout::SevenPointOne, Some(&[0, 2, 1, 7, 5, 6, 3, 4])), + other => { + return Err(Error::Unsupported(format!( + "opus channel mapping family 1 has no {other}-channel layout" + ))); + } + }) +} + impl Backend for Libopus { /// Empty packets invoke packet-loss concealment. Loss during DTX remains /// classified as DTX, while loss during active audio remains active. fn decode(&mut self, packet: &[u8]) -> Result { let channels = self.layout.channels() as usize; let mut out = vec![0.0f32; self.max_frame_size * channels]; - // SAFETY: `inner` owns a live OpusDecoder; packet/out slices are bounded by + // SAFETY: `inner` owns a live OpusMSDecoder; packet/out slices are bounded by // the lengths we pass. let samples = unsafe { - opus_decode_float( - &mut *self.inner, + opus_multistream_decode_float( + self.inner, packet.as_ptr(), packet.len() as i32, out.as_mut_ptr(), @@ -86,14 +160,24 @@ impl Backend for Libopus { } out.truncate(samples as usize * channels); - let activity = opus::activity(packet, self.in_dtx); + if let Some(order) = self.reorder { + let mut vorbis = [0.0f32; 8]; + for frame in out.chunks_exact_mut(channels) { + vorbis[..channels].copy_from_slice(frame); + for (sample, &from) in frame.iter_mut().zip(order) { + *sample = vorbis[from]; + } + } + } + + let activity = opus::multistream_activity(packet, self.streams, self.in_dtx); self.in_dtx = activity.is_dtx(); Ok(Decoded { samples: out, activity }) } fn reset(&mut self) -> Result<(), Error> { // SAFETY: `inner` owns a live decoder and OPUS_RESET_STATE takes no arguments. - let rc = unsafe { opus_decoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; + let rc = unsafe { opus_multistream_decoder_ctl_impl(self.inner, OPUS_RESET_STATE, varargs![]) }; if rc != OPUS_OK { return Err(opus::error(rc, "OPUS_RESET_STATE")); } @@ -120,7 +204,214 @@ impl Backend for Libopus { impl Drop for Libopus { fn drop(&mut self) { - // SAFETY: `inner` is a live OpusDecoder that nothing else aliases. - unsafe { opus_decoder_destroy(self.inner) }; + // SAFETY: `inner` is a live OpusMSDecoder that nothing else aliases. + unsafe { opus_multistream_decoder_destroy(self.inner) }; + } +} + +#[cfg(test)] +mod tests { + use unsafe_libopus::{ + OPUS_APPLICATION_AUDIO, opus_multistream_encode_float, opus_multistream_encoder_destroy, + opus_multistream_surround_encoder_create, + }; + + use crate::Error; + use crate::decode::{Config, Decoder}; + use crate::layout::Speaker::{self, *}; + + /// Speakers in Vorbis channel order for each family 1 channel count (RFC 7845 §5.1.1.2). + const VORBIS: [&[Speaker]; 8] = [ + &[FrontCenter], + &[FrontLeft, FrontRight], + &[FrontLeft, FrontCenter, FrontRight], + &[FrontLeft, FrontRight, BackLeft, BackRight], + &[FrontLeft, FrontCenter, FrontRight, SideLeft, SideRight], + &[FrontLeft, FrontCenter, FrontRight, SideLeft, SideRight, Lfe], + &[FrontLeft, FrontCenter, FrontRight, SideLeft, SideRight, BackCenter, Lfe], + &[ + FrontLeft, + FrontCenter, + FrontRight, + SideLeft, + SideRight, + BackLeft, + BackRight, + Lfe, + ], + ]; + + const RATE: usize = 48_000; + const FRAME: usize = 960; + const PACKETS: usize = 15; + + /// A distinct tone per speaker, low for the LFE, which libopus band-limits. + fn tone(speaker: Speaker) -> f32 { + match speaker { + Lfe => 80.0, + other => 400.0 + 300.0 * other as u8 as f32, + } + } + + /// Encode `channels` of Vorbis-ordered tones as a family 1 stream, returning + /// its OpusHead and packets. + fn surround(channels: usize) -> (bytes::Bytes, Vec>) { + let speakers = VORBIS[channels - 1]; + let (mut streams, mut coupled, mut mapping) = (0i32, 0i32, [0u8; 8]); + let mut err = 0i32; + // SAFETY: every out-pointer is valid and `mapping` holds `channels` entries. + let encoder = unsafe { + opus_multistream_surround_encoder_create( + RATE as i32, + channels as i32, + 1, + &mut streams, + &mut coupled, + mapping.as_mut_ptr(), + OPUS_APPLICATION_AUDIO, + &mut err, + ) + }; + assert!(err == 0 && !encoder.is_null(), "encoder create failed: {err}"); + + let packets = (0..PACKETS) + .map(|packet| { + let mut pcm = Vec::with_capacity(FRAME * channels); + for i in packet * FRAME..(packet + 1) * FRAME { + for &speaker in speakers { + let phase = std::f32::consts::TAU * tone(speaker) * i as f32 / RATE as f32; + pcm.push(phase.sin() * 0.5); + } + } + let mut out = vec![0u8; 4000]; + // SAFETY: `encoder` is live, `pcm` holds FRAME frames, and `out` is as long as we say. + let len = unsafe { + opus_multistream_encode_float(encoder, pcm.as_ptr(), FRAME as i32, out.as_mut_ptr(), 4000) + }; + assert!(len > 0, "encode failed: {len}"); + out.truncate(len as usize); + out + }) + .collect(); + // SAFETY: `encoder` is live and not used again. + unsafe { opus_multistream_encoder_destroy(encoder) }; + + let mut head = moq_mux::codec::opus::Config::new(RATE as u32, 2) + .encode() + .unwrap() + .to_vec(); + head[9] = channels as u8; + head[18] = 1; + head.extend_from_slice(&[streams as u8, coupled as u8]); + head.extend_from_slice(&mapping[..channels]); + (head.into(), packets) + } + + fn catalog(head: bytes::Bytes, channels: u32) -> hang::catalog::AudioConfig { + let mut catalog = hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Opus, 48_000, channels); + catalog.description = Some(head); + catalog + } + + /// Energy of `samples` at `freq`, by the Goertzel recurrence. + fn power(samples: &[f32], freq: f32) -> f32 { + let coeff = 2.0 * (std::f32::consts::TAU * freq / RATE as f32).cos(); + let (mut s1, mut s2) = (0.0f32, 0.0f32); + for &x in samples { + let s = x + coeff * s1 - s2; + s2 = s1; + s1 = s; + } + s1 * s1 + s2 * s2 - coeff * s1 * s2 + } + + /// Every family 1 layout decodes with each canonical channel carrying its + /// own speaker's tone, which pins the Vorbis reorder independently of it. + #[test] + fn family_one_decodes_in_canonical_order() { + for channels in 1..=8 { + let (head, packets) = surround(channels); + let mut decoder = Decoder::new(&catalog(head, channels as u32), &Config::default()).unwrap(); + let layout = decoder.layout(); + // The layout has exactly the Vorbis speakers, which canonical order sorts. + let mut speakers = VORBIS[channels - 1].to_vec(); + speakers.sort_by_key(|&speaker| speaker as u8); + assert_eq!(layout.speakers().unwrap(), speakers, "{channels} channels"); + + let mut pcm = Vec::new(); + for packet in &packets { + pcm.extend(decoder.decode(packet).unwrap().samples); + } + // Skip the encoder's warmup. + let pcm = &pcm[pcm.len() / 2..]; + + let tones: Vec = VORBIS[channels - 1].iter().copied().map(tone).collect(); + for (index, &speaker) in layout.speakers().unwrap().iter().enumerate() { + let channel: Vec = pcm.iter().skip(index).step_by(channels).copied().collect(); + let loudest = tones + .iter() + .copied() + .max_by(|a, b| power(&channel, *a).total_cmp(&power(&channel, *b))) + .unwrap(); + assert_eq!(loudest, tone(speaker), "{channels} channels, {speaker:?} at {index}"); + } + } + } + + /// An all-DTX surround packet is one empty Opus packet per stream. Read as a + /// single stream, the later subpackets look like payload and the span is lost. + #[test] + fn surround_silence_is_dtx() { + let (head, packets) = surround(6); + let mut decoder = Decoder::new(&catalog(head.clone(), 6), &Config::default()).unwrap(); + let mid = decoder.decode(&packets[PACKETS / 2]).unwrap(); + assert!(mid.activity.is_active(), "a coded surround frame must stay active"); + + // Two coupled streams, then two mono, each a 20 ms empty frame. + let dtx = [0xfc, 0x00, 0xfc, 0x00, 0xf8, 0x00, 0xf8]; + let mut decoder = Decoder::new(&catalog(head, 6), &Config::default()).unwrap(); + let decoded = decoder.decode(&dtx).expect("empty multistream packet"); + assert!(decoded.activity.is_dtx(), "all-DTX surround packet read as active"); + assert_eq!(decoded.samples.len() % 6, 0); + } + + /// A description that is present but truncated used to be dropped, and a + /// stereo catalog then opened a family 0 decoder for a family 1 or 255 head. + #[test] + fn malformed_description_is_refused() { + for family in [1u8, 255] { + let mut head = moq_mux::codec::opus::Config::new(48_000, 2).encode().unwrap().to_vec(); + head[18] = family; + let err = Decoder::new(&catalog(head.into(), 2), &Config::default()) + .err() + .expect("refused"); + assert!( + matches!(&err, Error::Unsupported(message) if message.contains("opus description")), + "family {family}: {err}" + ); + } + + let plain = hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Opus, 48_000, 2); + assert!(Decoder::new(&plain, &Config::default()).is_ok()); + } + + /// Families other than 0 and 1 carry no speaker positions, so there is + /// nothing to put in a layout: refused, not passed through as discrete. + #[test] + fn other_families_are_refused() { + for (family, channels, table) in [(255u8, 2u8, &[2, 0, 0, 1][..]), (2, 4, &[4, 0, 0, 1, 2, 3])] { + let mut head = moq_mux::codec::opus::Config::new(48_000, 2).encode().unwrap().to_vec(); + head[9] = channels; + head[18] = family; + head.extend_from_slice(table); + + let err = Decoder::new(&catalog(head.into(), channels.into()), &Config::default()) + .err() + .expect("refused"); + assert!( + matches!(&err, Error::Unsupported(message) if message.contains("no speaker positions")), + "family {family}: {err}" + ); + } } } diff --git a/rs/moq-audio/src/opus.rs b/rs/moq-audio/src/opus.rs index d87a6055d3a..0dd53c4edcd 100644 --- a/rs/moq-audio/src/opus.rs +++ b/rs/moq-audio/src/opus.rs @@ -61,7 +61,7 @@ pub(crate) fn decode_error(code: i32) -> Error { if code == unsafe_libopus::OPUS_INVALID_PACKET { return Error::Decode(format!("libopus rejected the packet (code {code})")); } - error(code, "opus_decode_float") + error(code, "opus_multistream_decode_float") } /// Classify a packet, preserving DTX across packet loss. @@ -80,6 +80,171 @@ pub(crate) fn activity(packet: &[u8], in_dtx: bool) -> Activity { } } +/// Classify a multistream packet, preserving DTX across packet loss. +/// +/// Every stream but the last is self-delimited (RFC 6716 framing plus the length +/// libopus writes ahead of that stream's payload). One `opus_packet_parse` of the +/// whole buffer treats the later streams as payload, so an all-DTX surround +/// packet reads as active. DTX means every stream coded nothing. +pub(crate) fn multistream_activity(packet: &[u8], streams: u8, in_dtx: bool) -> Activity { + if packet.is_empty() { + return if in_dtx { Activity::Dtx } else { Activity::Active }; + } + if streams <= 1 { + return activity(packet, in_dtx); + } + if streams_carry_nothing(packet, streams) { + Activity::Dtx + } else { + Activity::Active + } +} + +/// Whether every stream in a multistream packet coded no audio. +fn streams_carry_nothing(mut packet: &[u8], streams: u8) -> bool { + for index in 0..streams { + let last = index + 1 == streams; + if last { + return carries_nothing(packet); + } + let Some((silent, offset)) = self_delimited(packet) else { + return false; + }; + if !silent { + return false; + } + packet = &packet[offset..]; + } + false +} + +/// One self-delimited Opus packet at the front of `data`: whether every frame is +/// empty, and how many bytes it occupies, including its trailing padding. +/// +/// `None` when the bytes are not that packet. The length of the last frame is +/// coded where libopus puts it, before the frame payloads. +fn self_delimited(data: &[u8]) -> Option<(bool, usize)> { + if data.is_empty() { + return None; + } + let framesize = unsafe { unsafe_libopus::opus_packet_get_samples_per_frame(data.as_ptr(), 48_000) }; + if framesize <= 0 { + return None; + } + + let toc = data[0]; + let mut pos = 1usize; + let mut len = data.len() - 1; + let mut last_size = len; + let mut cbr = false; + let mut pad = 0usize; + let mut sizes = [0i32; 48]; + + let count = match toc & 0x3 { + 0 => 1, + 1 => { + cbr = true; + 2 + } + 2 => { + let (bytes, size) = parse_size(data.get(pos..pos.checked_add(len)?)?)?; + len = len.checked_sub(bytes)?; + if size as usize > len { + return None; + } + sizes[0] = size; + pos += bytes; + last_size = len - size as usize; + 2 + } + _ => { + if len < 1 { + return None; + } + let ch = data[pos]; + pos += 1; + len -= 1; + let count = (ch & 0x3f) as usize; + if count == 0 || count > sizes.len() || framesize as usize * count > 5760 { + return None; + } + if ch & 0x40 != 0 { + loop { + if len == 0 { + return None; + } + let p = data[pos] as usize; + pos += 1; + len -= 1; + let tmp = if p == 255 { 254 } else { p }; + len = len.checked_sub(tmp)?; + pad += tmp; + if p != 255 { + break; + } + } + } + cbr = ch & 0x80 == 0; + if !cbr { + last_size = len; + for slot in &mut sizes[..count - 1] { + let (bytes, size) = parse_size(data.get(pos..pos.checked_add(len)?)?)?; + len = len.checked_sub(bytes)?; + if size as usize > len { + return None; + } + *slot = size; + pos += bytes; + last_size = last_size.checked_sub(bytes + size as usize)?; + } + } + count + } + }; + + let (bytes, size) = parse_size(data.get(pos..pos.checked_add(len)?)?)?; + len = len.checked_sub(bytes)?; + if size as usize > len { + return None; + } + sizes[count - 1] = size; + pos += bytes; + if cbr { + if size as usize * count > len { + return None; + } + for slot in &mut sizes[..count - 1] { + *slot = size; + } + } else if bytes + size as usize > last_size { + return None; + } + + for &frame in &sizes[..count] { + pos = pos.checked_add(frame as usize)?; + if pos > data.len() { + return None; + } + } + let offset = pad.checked_add(pos)?; + if offset == 0 || offset > data.len() { + return None; + } + let silent = sizes[..count].iter().all(|&frame| frame == 0); + Some((silent, offset)) +} + +/// The self-delimited length prefix (RFC 6716 §3.2.5), or `None` when it is cut off. +fn parse_size(data: &[u8]) -> Option<(usize, i32)> { + let first = *data.first()?; + if first < 252 { + Some((1, i32::from(first))) + } else { + let second = *data.get(1)?; + Some((2, 4 * i32::from(second) + i32::from(first))) + } +} + /// Whether the sender coded no audio at all for this packet: a bare TOC whose /// every frame is empty. /// @@ -168,6 +333,38 @@ mod tests { assert_eq!(activity(&[], false), Activity::Active); } + /// A surround packet is one self-delimited Opus packet per stream, then a + /// normal packet for the last. DTX is all of them empty; one coded frame is + /// audio. Parsing the buffer as a single stream would read the later + /// subpackets as that frame's payload. + #[test] + fn multistream_activity_requires_every_stream_empty() { + // 5.1: two coupled streams, then two mono. Code 0, empty frame. + let dtx = [0xfc, 0x00, 0xfc, 0x00, 0xf8, 0x00, 0xf8]; + assert_eq!(multistream_activity(&dtx, 4, false), Activity::Dtx); + // A coded frame on the last stream. + let last = [0xfc, 0x00, 0xfc, 0x00, 0xf8, 0x00, 0xf8, 0xaa]; + assert_eq!(multistream_activity(&last, 4, false), Activity::Active); + // A coded frame on an earlier stream. The length byte is the payload size. + let early = [0xfc, 0x01, 0xaa, 0xfc, 0x00, 0xf8, 0x00, 0xf8]; + assert_eq!(multistream_activity(&early, 4, false), Activity::Active); + // Two empty frames in one self-delimited code-1 packet, then a bare TOC. + let cbr = [0xf9, 0x00, 0xf8]; + assert_eq!(multistream_activity(&cbr, 2, false), Activity::Dtx); + let cbr_payload = [0xf9, 0x01, 0xaa, 0xbb, 0xf8]; + assert_eq!(multistream_activity(&cbr_payload, 2, false), Activity::Active); + // Code 3 packs the 60 ms case: three empty frames, self-delimited, then a bare TOC. + let code3 = [0xfb, 0x03, 0x00, 0xf8]; + assert_eq!(multistream_activity(&code3, 2, false), Activity::Dtx); + let code3_payload = [0xfb, 0x03, 0x01, 0xaa, 0xbb, 0xcc, 0xf8]; + assert_eq!(multistream_activity(&code3_payload, 2, false), Activity::Active); + // Loss still carries the previous classification, whatever the layout. + assert_eq!(multistream_activity(&[], 4, true), Activity::Dtx); + assert_eq!(multistream_activity(&[], 4, false), Activity::Active); + // One stream is an ordinary packet. + assert_eq!(multistream_activity(&[0xf8], 1, false), Activity::Dtx); + } + /// A silence run's periodic refresh is an ordinarily coded frame, and a /// speech onset can leave an earlier frame of the same packet empty. Neither /// is distinguishable by framing, so both read active rather than risking diff --git a/rs/moq-mux/src/catalog/msf/consumer.rs b/rs/moq-mux/src/catalog/msf/consumer.rs index 6df191bfffd..bbdeb41bef5 100644 --- a/rs/moq-mux/src/catalog/msf/consumer.rs +++ b/rs/moq-mux/src/catalog/msf/consumer.rs @@ -615,7 +615,8 @@ mod test { head.push(6); // channel_count (5.1) head.extend_from_slice(&0u16.to_le_bytes()); // pre_skip head.extend_from_slice(&24_000u32.to_le_bytes()); // sample_rate - head.extend_from_slice(&[0, 0, 0]); // output gain (i16) + channel mapping family (1 byte) + head.extend_from_slice(&[0, 0, 1]); // output gain (i16) + channel mapping family 1 + head.extend_from_slice(&[4, 2, 0, 4, 1, 2, 3, 5]); // streams, coupled, Vorbis 5.1 mapping let init_b64 = base64::engine::general_purpose::STANDARD.encode(&head); let mut track = audio_track("audio0", moq_msf::Packaging::Loc); diff --git a/rs/moq-mux/src/codec/opus/import.rs b/rs/moq-mux/src/codec/opus/import.rs index d32c33ae58e..69b4631c86f 100644 --- a/rs/moq-mux/src/codec/opus/import.rs +++ b/rs/moq-mux/src/codec/opus/import.rs @@ -114,7 +114,10 @@ impl Import { /// Build a catalog config from an OpusHead. Errors on a malformed or empty buffer. pub fn config(init: &[u8]) -> crate::Result { let mut buf = init; - Ok(Config::parse(&mut buf)?.into()) + let mut config: hang::catalog::AudioConfig = Config::parse(&mut buf)?.into(); + // Publish the head as given: re-encoding it would drop a channel mapping table. + config.description = Some(bytes::Bytes::copy_from_slice(init)); + Ok(config) } impl From for hang::catalog::AudioConfig { diff --git a/rs/moq-mux/src/codec/opus/mod.rs b/rs/moq-mux/src/codec/opus/mod.rs index bd1bba4a9ba..a73e9227a6c 100644 --- a/rs/moq-mux/src/codec/opus/mod.rs +++ b/rs/moq-mux/src/codec/opus/mod.rs @@ -23,10 +23,23 @@ pub enum Error { #[error("invalid OpusHead signature")] InvalidSignature, - /// [`Config::encode`] was asked to emit an OpusHead for a channel count other - /// than mono or stereo; channel mapping family 0 only covers 1 or 2 channels. - #[error("channel mapping family 0 only supports mono/stereo (got {0} channels)")] + /// The channel count is zero, or more than the channel mapping family allows: + /// family 0 covers mono/stereo and family 1 up to eight channels. + #[error("channel mapping family does not allow {0} channels")] UnsupportedChannelCount(u32), + + /// A nonzero channel mapping family without its complete mapping table. + #[error("OpusHead channel mapping table is truncated")] + MappingTableTooShort, + + /// The channel mapping table names no streams, more coupled streams than + /// streams, or a channel index past the decoded streams. + #[error("invalid OpusHead channel mapping table")] + InvalidMappingTable, + + /// [`Config::encode`] only emits channel mapping family 0. + #[error("cannot encode channel mapping family {0}")] + UnsupportedMappingFamily(u8), } pub type Result = std::result::Result; @@ -37,10 +50,85 @@ pub type Result = std::result::Result; pub struct Config { /// Original input sample rate in Hz. pub sample_rate: u32, - /// Number of encoded channels. + /// Number of output channels. pub channel_count: u32, /// Number of decoded 48 kHz samples to discard at stream start. pub pre_skip: u16, + /// The channel mapping table, or `None` for family 0 (mono/stereo, one stream). + pub mapping: Option, +} + +/// An OpusHead channel mapping table (RFC 7845 §5.1.1), present for every family but 0. +#[derive(Clone, Copy, PartialEq, Eq)] +pub struct Mapping { + family: u8, + streams: u8, + coupled: u8, + channels: u8, + // Sized for the most channels any family allows, so `Config` stays `Copy`. + table: [u8; 255], +} + +impl Mapping { + fn parse(buf: &mut T, family: u8, channels: u8) -> Result { + if buf.remaining() < 2 + channels as usize { + return Err(Error::MappingTableTooShort); + } + let streams = buf.get_u8(); + let coupled = buf.get_u8(); + if streams == 0 || coupled > streams || streams as u32 + coupled as u32 > 255 { + return Err(Error::InvalidMappingTable); + } + + let mut table = [0u8; 255]; + for entry in &mut table[..channels as usize] { + *entry = buf.get_u8(); + // 255 marks a silent channel. + if *entry != 255 && *entry >= streams + coupled { + return Err(Error::InvalidMappingTable); + } + } + + Ok(Self { + family, + streams, + coupled, + channels, + table, + }) + } + + /// The channel mapping family: 1 is the Vorbis speaker order, 2 and 3 are + /// ambisonics, and 255 is channels with no declared position. + pub fn family(&self) -> u8 { + self.family + } + + /// Number of Opus streams in each packet. + pub fn streams(&self) -> u8 { + self.streams + } + + /// How many of those streams are coupled (stereo); they come first. + pub fn coupled(&self) -> u8 { + self.coupled + } + + /// The decoded channel feeding each output channel, 255 for silence. + pub fn table(&self) -> &[u8] { + &self.table[..self.channels as usize] + } +} + +impl std::fmt::Debug for Mapping { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Mapping") + .field("family", &self.family) + .field("streams", &self.streams) + .field("coupled", &self.coupled) + .field("table", &self.table()) + .finish() + } } impl Config { @@ -50,6 +138,7 @@ impl Config { sample_rate, channel_count, pre_skip: 0, + mapping: None, } } @@ -61,8 +150,9 @@ impl Config { /// Parse an OpusHead buffer (RFC 7845 §5.1). /// - /// Verifies the magic signature; reads channel count, pre-skip, and sample - /// rate; ignores gain and channel mapping. Any trailing bytes are consumed. + /// Verifies the magic signature; reads channel count, pre-skip, sample rate, + /// and the channel mapping, refusing a channel count the family forbids or an + /// inconsistent mapping table; ignores gain. Any trailing bytes are consumed. pub fn parse(buf: &mut T) -> Result { if buf.remaining() < 19 { return Err(Error::HeadTooShort); @@ -76,8 +166,23 @@ impl Config { let channel_count = buf.get_u8() as u32; let pre_skip = buf.get_u16_le(); let sample_rate = buf.get_u32_le(); + buf.advance(2); // Skip gain until if/when we support it. + let family = buf.get_u8(); + + let max_channels = match family { + 0 => 2, + 1 => 8, + _ => 255, + }; + if channel_count == 0 || channel_count > max_channels { + return Err(Error::UnsupportedChannelCount(channel_count)); + } + + let mapping = match family { + 0 => None, + family => Some(Mapping::parse(buf, family, channel_count as u8)?), + }; - // Skip gain, channel mapping until if/when we support them. if buf.remaining() > 0 { buf.advance(buf.remaining()); } @@ -86,6 +191,7 @@ impl Config { sample_rate, channel_count, pre_skip, + mapping, }) } @@ -95,8 +201,11 @@ impl Config { /// Errors with [`Error::UnsupportedChannelCount`] unless `channel_count` is 1 /// or 2, since mapping family 0 is only defined for mono/stereo per RFC 7845 §5.1. /// Multi-channel streams need family 1 with a channel mapping table, which - /// this helper does not emit. + /// this helper does not emit, so any `mapping` is [`Error::UnsupportedMappingFamily`]. pub fn encode(&self) -> Result { + if let Some(mapping) = &self.mapping { + return Err(Error::UnsupportedMappingFamily(mapping.family)); + } if !(1..=2).contains(&self.channel_count) { return Err(Error::UnsupportedChannelCount(self.channel_count)); } @@ -190,4 +299,85 @@ mod tests { let err = Config::new(48_000, 6).encode().unwrap_err(); assert!(matches!(err, Error::UnsupportedChannelCount(6))); } + + /// A 19-byte head with the given channel count and mapping family. + fn head(channels: u8, family: u8) -> Vec { + let mut head = Config::new(48_000, 2).encode().unwrap().to_vec(); + head[9] = channels; + head[18] = family; + head + } + + #[test] + fn parses_a_mapping_table() { + // 5.1 in family 1: four streams, two coupled, Vorbis order. + let mut bytes = head(6, 1); + bytes.extend_from_slice(&[4, 2, 0, 4, 1, 2, 3, 5]); + let parsed = Config::parse(&mut bytes.as_slice()).unwrap(); + assert_eq!(parsed.channel_count, 6); + + let mapping = parsed.mapping.unwrap(); + assert_eq!(mapping.family(), 1); + assert_eq!(mapping.streams(), 4); + assert_eq!(mapping.coupled(), 2); + assert_eq!(mapping.table(), &[0, 4, 1, 2, 3, 5]); + + // Encode only emits family 0. + assert!(matches!(parsed.encode(), Err(Error::UnsupportedMappingFamily(1)))); + } + + #[test] + fn parse_rejects_channel_counts_the_family_does_not_allow() { + for (channels, family) in [(0, 0), (3, 0), (0, 1), (9, 1), (0, 255)] { + let mut bytes = head(channels, family); + bytes.extend_from_slice(&[1, 0]); + bytes.extend(std::iter::repeat_n(0, channels as usize)); + assert!( + matches!( + Config::parse(&mut bytes.as_slice()), + Err(Error::UnsupportedChannelCount(_)) + ), + "{channels} channels in family {family}" + ); + } + } + + #[test] + fn parse_rejects_a_bad_mapping_table() { + // Family 1 promising a table that is not there, or only half of it. + assert!(matches!( + Config::parse(&mut head(2, 1).as_slice()), + Err(Error::MappingTableTooShort) + )); + let mut short = head(2, 1); + short.extend_from_slice(&[1, 1, 0]); + assert!(matches!( + Config::parse(&mut short.as_slice()), + Err(Error::MappingTableTooShort) + )); + + for table in [ + // No streams. + [0, 0, 0, 1], + // More coupled streams than streams. + [1, 2, 0, 1], + // A channel index past the two decoded channels. + [1, 1, 0, 2], + ] { + let mut bytes = head(2, 1); + bytes.extend_from_slice(&table); + assert!( + matches!(Config::parse(&mut bytes.as_slice()), Err(Error::InvalidMappingTable)), + "{table:?}" + ); + } + + // 255 is a silent channel, not an index. + let mut silent = head(2, 1); + silent.extend_from_slice(&[1, 0, 0, 255]); + assert_eq!( + Config::parse(&mut silent.as_slice()).unwrap().mapping.unwrap().table(), + &[0, 255] + ); + } } From 42582a8cc75c01ec433a2486c7b645118e7cdec4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 26 Sep 2026 18:10:45 -0700 Subject: [PATCH 10/20] fix(mux): ADTS export refuses what it cannot label (#4283) Co-authored-by: Claude Opus 5.5 --- doc/bin/cli.md | 5 + quest/m1/audio-codecs/README.md | 4 +- quest/m1/audio-codecs/aac-encode-refusals.md | 24 +++ .../audio-codecs/aac-parse-truncated-sbr.md | 21 +++ quest/m1/audio-codecs/adts-refusals.md | 25 --- .../m1/audio-codecs/he-aac-catalog-output.md | 29 +++ rs/moq-mux/src/codec/aac/mod.rs | 168 ++++++++++++------ rs/moq-mux/src/container/ts/adts.rs | 38 +++- rs/moq-mux/src/container/ts/export.rs | 52 +++--- rs/moq-mux/src/container/ts/export_test.rs | 85 +++++++++ .../src/container/ts/test_data/he_aac.flv | Bin 0 -> 3110 bytes .../src/container/ts/test_data/he_aac_v2.flv | Bin 0 -> 3297 bytes 12 files changed, 336 insertions(+), 115 deletions(-) create mode 100644 quest/m1/audio-codecs/aac-encode-refusals.md create mode 100644 quest/m1/audio-codecs/aac-parse-truncated-sbr.md delete mode 100644 quest/m1/audio-codecs/adts-refusals.md create mode 100644 quest/m1/audio-codecs/he-aac-catalog-output.md create mode 100644 rs/moq-mux/src/container/ts/test_data/he_aac.flv create mode 100644 rs/moq-mux/src/container/ts/test_data/he_aac_v2.flv diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 723b80a3e46..76b01f12a76 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -66,6 +66,11 @@ discarding the old mux buffer. The first new clock packet signals the break and stdout pacing re-anchors. Every rendition joins the new program generation; no track is fenced across the marker. +MPEG-TS export frames AAC as ADTS, which labels only the AAC Main, LC, SSR, +and LTP profiles. HE-AAC and HE-AACv2 go out as their AAC-LC core, and decoders +find the SBR and PS in band, as ffmpeg's ADTS output does. A track whose +profile or channel layout ADTS cannot label is refused rather than mislabeled. + A constant-rate MPEG-TS source records its multiplex rate in the catalog (`mpegts.muxRate`, measured off the PCR clock, null stuffing included), and `export ts` pads its output with null packets back to that rate so an IRD or diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 71b47defcc5..762769dc5ae 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -44,9 +44,11 @@ its own decode and encode quest so verification stays per host. ## Quests - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers -- [ADTS refusals](/quest/m1/audio-codecs/adts-refusals.md) - the ADTS writer refuses channel counts and object types it cannot label instead of mislabeling them +- [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC - [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes +- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - `Config::encode` refuses channel counts it cannot name instead of writing stereo +- [AAC parse truncated SBR](/quest/m1/audio-codecs/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core ## Related diff --git a/quest/m1/audio-codecs/aac-encode-refusals.md b/quest/m1/audio-codecs/aac-encode-refusals.md new file mode 100644 index 00000000000..9a891bb69f1 --- /dev/null +++ b/quest/m1/audio-codecs/aac-encode-refusals.md @@ -0,0 +1,24 @@ +# [S] AAC config encode refuses unnameable channel counts + +## Goal + +`moq_mux::codec::aac::Config::encode` never writes an AudioSpecificConfig that +names the wrong layout. A channel count no channelConfiguration names is +refused, not logged and written as stereo, the way the ADTS writer refuses it. + +## Plan + +- `encode` returns `Bytes` today, so refusing is a published API break: this + retargets to `dev`. Decide whether it returns a `Result` or whether a + constructor validates the count up front so encoding cannot fail. +- A count a program config element could describe (7, or more than 8) is + still refused unless the caller supplies a layout; guessing speaker + positions is what this removes. +- Callers in `moq-audio` (encoder config, description synthesis) already + validate first; check they keep their error messages. + +Public API: `Config::encode` (or its constructor) changes. Wire: none. + +## Related + +- [#4283](https://github.com/moq-dev/moq/pull/4283) - the ADTS export refusals this mirrors diff --git a/quest/m1/audio-codecs/aac-parse-truncated-sbr.md b/quest/m1/audio-codecs/aac-parse-truncated-sbr.md new file mode 100644 index 00000000000..57f9bf04e69 --- /dev/null +++ b/quest/m1/audio-codecs/aac-parse-truncated-sbr.md @@ -0,0 +1,21 @@ +# [XS] AAC config parse refuses truncated SBR and PS + +## Goal + +`Config::parse` refuses an AudioSpecificConfig that signals SBR or PS +(object type 5 or 29) but ends before its extension rate and core object type, +instead of accepting it as if the core were known. + +## Plan + +- The export path already requires these fields; `parse` stays lenient only + because `Config::encode` writes a two-byte config for those object types. + Once encode refuses or completes them, the leniency has no producer. +- Check the `moq-audio` tests that feed a two-byte HE-AAC config; they should + keep asserting a refusal, just a different one. + +Public API: none beyond a new error case. Wire: none. + +## Required + +- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - encode stops producing truncated SBR and PS configs diff --git a/quest/m1/audio-codecs/adts-refusals.md b/quest/m1/audio-codecs/adts-refusals.md deleted file mode 100644 index c7896e377d7..00000000000 --- a/quest/m1/audio-codecs/adts-refusals.md +++ /dev/null @@ -1,25 +0,0 @@ -# [S] ADTS export refuses what it cannot label - -## Goal - -`moq-mux`'s ADTS writer never labels an AAC track with the wrong layout or -object type. A channel count ADTS cannot name is refused, not silently written -as stereo, and an explicit SBR or PS description is written with the object -type ADTS can carry or refused, not masked to two bits into a wrong profile. - -## Plan - -- `channel_config_from_count` falls back to stereo for counts it cannot - represent; refuse instead, the same way #4178 made channelConfiguration 11 to - 14 refuse. -- The ADTS header masks the object type to two bits, so an explicit HE-AAC - (object type 5) or HE-AACv2 (29) description is mislabeled. Decide per case: - signal the backward-compatible AAC-LC core (implicit SBR) when the description - allows it, otherwise refuse. -- Tests for both with real fixtures, checked against ffprobe. - -Public API: none. Wire: none; TS output changes only for inputs it mislabeled. - -## Related - -- [#4178](https://github.com/moq-dev/moq/pull/4178) - the PCE export that found these diff --git a/quest/m1/audio-codecs/he-aac-catalog-output.md b/quest/m1/audio-codecs/he-aac-catalog-output.md new file mode 100644 index 00000000000..9032995d6e9 --- /dev/null +++ b/quest/m1/audio-codecs/he-aac-catalog-output.md @@ -0,0 +1,29 @@ +# [S] HE-AAC catalog entries describe the output + +## Goal + +A catalog entry for HE-AAC or HE-AACv2 names what the stream plays as, for +example 48 kHz stereo, not its AAC-LC core (24 kHz, and mono for v2). Players +and gateways that size buffers or pick a decoder from the catalog see the real +output. + +## Plan + +- `Config::parse` reports the core rate and channel count because those lead + an explicit SBR or PS AudioSpecificConfig. The FLV and MKV importers copy + them into the catalog; fMP4 takes the sample entry instead, so the importers + disagree for the same stream. +- Derive the output from the config: the extension rate under SBR, two + channels under PS. Implicit SBR, found only in band, cannot be known from the + config; keep the documented half-rate behavior for it. +- Consumers that rebuild a config from catalog fields (description synthesis in + `moq-audio`, the MSF path) must not then mistake the output rate for the core. +- Regression: the `fdkaacenc` HE-AAC and HE-AACv2 FLV fixtures under + `rs/moq-mux/src/container/ts/test_data` import as 48 kHz stereo, matching + ffprobe. + +Public API: possibly `Config` fields. Wire: catalog values change for HE-AAC. + +## Related + +- [#4283](https://github.com/moq-dev/moq/pull/4283) - found while labeling HE-AAC for ADTS diff --git a/rs/moq-mux/src/codec/aac/mod.rs b/rs/moq-mux/src/codec/aac/mod.rs index 5ccb24a1103..ad46c1773b9 100644 --- a/rs/moq-mux/src/codec/aac/mod.rs +++ b/rs/moq-mux/src/codec/aac/mod.rs @@ -73,7 +73,11 @@ impl Config { let mut reader = BitReader::new(buf); let (object_type, sample_rate, channel_config) = read_header(&mut reader)?; let channel_count = match channel_config { - 0 => program_config(&mut reader)?, + 0 => { + let core = read_core(&mut reader, object_type)?; + read_general_audio(&mut reader, core)?; + program_config(&mut reader)? + } _ => channel_count_from_config(channel_config)?, }; @@ -135,9 +139,8 @@ impl Config { } } -/// Read an AudioSpecificConfig up to its channels: the audioObjectType, sample rate, and -/// channelConfiguration. For a channelConfiguration of 0 the reader stops at the program config -/// element. +/// Read an AudioSpecificConfig's leading fields: the audioObjectType, sample rate, and +/// channelConfiguration. fn read_header(reader: &mut BitReader) -> Result<(u8, u32, u8)> { if reader.buf.remaining() < 2 { return Err(Error::ConfigTooShort); @@ -156,35 +159,40 @@ fn read_header(reader: &mut BitReader) -> Result<(u8, u32, u8)> { // channelConfiguration: 4 bits, immediately after the (possibly explicit) rate. let channel_config = reader.read(4, Error::IncompleteConfig)? as u8; - if channel_config != 0 { - return Ok((object_type, sample_rate, channel_config)); - } + Ok((object_type, sample_rate, channel_config)) +} - // Explicit SBR and PS name their core object type after an extension rate; the - // GASpecificConfig carrying the program config element follows that core type. - let mut core = object_type; - if matches!(object_type, 5 | 29) { - if reader.read(4, Error::IncompleteConfig)? == 15 { - reader.read(24, Error::IncompleteConfig)?; - } - core = read_object_type(reader)?; - if core == 22 { - // extensionChannelConfiguration, only for ER BSAC. - reader.read(4, Error::IncompleteConfig)?; - } +/// Read the core audioObjectType, which explicit SBR and PS name after an extension rate. Any +/// other object type is its own core, and the leading sample rate is always the core's. +fn read_core(reader: &mut BitReader, object_type: u8) -> Result { + if !matches!(object_type, 5 | 29) { + return Ok(object_type); } + if reader.read(4, Error::IncompleteConfig)? == 15 { + reader.read(24, Error::IncompleteConfig)?; + } + let core = read_object_type(reader)?; + if core == 22 { + // extensionChannelConfiguration, only for ER BSAC. + reader.read(4, Error::IncompleteConfig)?; + } + Ok(core) +} + +/// Read a GASpecificConfig up to the program config element that a channelConfiguration of 0 +/// puts next, refusing a `core` object type whose specific config is something else. +fn read_general_audio(reader: &mut BitReader, core: u8) -> Result<()> { if !GENERAL_AUDIO.contains(&core) { return Err(Error::ProgramConfigUnsupported(core)); } - // GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit - // coreCoderDelay), and extensionFlag precede the element. + // frameLengthFlag, dependsOnCoreCoder (then a 14-bit coreCoderDelay), and extensionFlag. reader.read(1, Error::IncompleteConfig)?; if reader.read(1, Error::IncompleteConfig)? == 1 { reader.read(14, Error::IncompleteConfig)?; } reader.read(1, Error::IncompleteConfig)?; - Ok((object_type, sample_rate, 0)) + Ok(()) } /// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does. @@ -222,25 +230,46 @@ pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8, Ok(Bytes::from(out.bytes)) } -/// Split an AudioSpecificConfig into the channel signaling of a stream that carries it per frame, -/// as ADTS does; the inverse of [`in_band_config`]. -/// -/// Returns the channelConfiguration and, when that is 0, the program config element framed to -/// lead a raw data block: its element ID, then the element aligned to the block. -pub(crate) fn in_band_channels(asc: &[u8]) -> Result<(u8, Option)> { +/// The fields a stream that signals its config per frame, as ADTS does, carries in each header. +#[derive(Clone, Debug, PartialEq)] +pub(crate) struct InBand { + /// The core audioObjectType. Explicit SBR and PS name it under their own, and a decoder finds + /// them again in band, as it would in a stream that never signaled them (implicit signaling). + pub object_type: u8, + /// The core's sample rate, which SBR doubles on output. + pub sample_rate: u32, + pub channel_config: u8, + /// When `channel_config` is 0, the program config element framed to lead a raw data block: + /// its element ID, then the element aligned to the block. + pub program_config: Option, +} + +/// Split an AudioSpecificConfig into what a stream that carries it per frame signals, as ADTS +/// does; the inverse of [`in_band_config`]. +pub(crate) fn in_band(asc: &[u8]) -> Result { let mut asc = asc; let mut reader = BitReader::new(&mut asc); - let (_, _, channel_config) = read_header(&mut reader)?; - if channel_config != 0 { - return Ok((channel_config, None)); - } + let (object_type, sample_rate, channel_config) = read_header(&mut reader)?; + let core = read_core(&mut reader, object_type)?; + + let program_config = match channel_config { + 0 => { + read_general_audio(&mut reader, core)?; + let mut out = BitWriter::default(); + out.write(3, ID_PCE); + reader.record = Some(out); + program_config(&mut reader)?; + Some(Bytes::from(reader.record.take().expect("recording set above").bytes)) + } + _ => None, + }; - let mut out = BitWriter::default(); - out.write(3, ID_PCE); - reader.record = Some(out); - program_config(&mut reader)?; - let out = reader.record.take().expect("recording set above"); - Ok((0, Some(Bytes::from(out.bytes)))) + Ok(InBand { + object_type: core, + sample_rate, + channel_config, + program_config, + }) } /// The raw data block element ID of a program config element (ISO 14496-3 Table 4.85). @@ -554,10 +583,9 @@ mod tests { assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, 7); } - #[test] - fn parses_program_config_element_behind_explicit_sbr() { - // audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension - // rate, then the core type (LC) whose GASpecificConfig carries the element. + /// audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension rate, + /// then the core type (LC) whose GASpecificConfig carries a front and back pair. + fn sbr_pce_asc() -> Vec { let mut out = BitWriter::default(); out.write(5, 5); out.write(4, 6); @@ -566,7 +594,12 @@ mod tests { out.write(5, 2); out.write(3, 0); write_pce(&mut out, &[true], &[], &[true], 0); - let cfg = Config::parse(&mut out.bytes.as_slice()).unwrap(); + out.bytes + } + + #[test] + fn parses_program_config_element_behind_explicit_sbr() { + let cfg = Config::parse(&mut sbr_pce_asc().as_slice()).unwrap(); assert_eq!(cfg.profile, 5); assert_eq!(cfg.channel_count, 4); } @@ -638,16 +671,18 @@ mod tests { assert_eq!(rest, [0xFF], "the element moves out of the block"); // And back: the element leads the block again, byte for byte. - assert_eq!(in_band_channels(&asc).unwrap(), (0, Some(Bytes::from(pce)))); + let in_band = in_band(&asc).unwrap(); + assert_eq!(in_band.channel_config, 0); + assert_eq!(in_band.program_config, Some(Bytes::from(pce))); } #[test] - fn in_band_channels_of_ffmpeg_program_config_element() { + fn in_band_of_ffmpeg_program_config_element() { // Round trip ffmpeg's own element through a raw data block and back into a config. The // trailing SBR sync extension is not part of the element, so it drops. - let (config, pce) = in_band_channels(&FFMPEG_QUAD_ASC).unwrap(); - assert_eq!(config, 0); - let pce = pce.unwrap(); + let in_band = in_band(&FFMPEG_QUAD_ASC).unwrap(); + assert_eq!(in_band.channel_config, 0); + let pce = in_band.program_config.unwrap(); let mut block = pce.as_ref(); let asc = in_band_config(2, 48_000, 0, &mut block).unwrap(); assert!(block.is_empty(), "the element is all that was framed"); @@ -655,9 +690,40 @@ mod tests { } #[test] - fn in_band_channels_without_a_program_config_element() { - assert_eq!(in_band_channels(&[0x11, 0x90]).unwrap(), (2, None)); - assert_eq!(in_band_channels(&[0x11, 0xE0]).unwrap(), (12, None)); + fn in_band_without_a_program_config_element() { + let stereo = in_band(&[0x11, 0x90]).unwrap(); + assert_eq!((stereo.channel_config, stereo.program_config), (2, None)); + assert_eq!(in_band(&[0x11, 0xE0]).unwrap().channel_config, 12); + } + + #[test] + fn in_band_of_explicit_sbr_is_its_lc_core() { + // GStreamer 1.28 `fdkaacenc` at 48 kHz stereo: HE-AAC is SBR over a 24 kHz stereo LC core, + // and HE-AACv2 is PS over a mono one. + for (asc, channel_config) in [([0x2B, 0x11, 0x88, 0x00], 2), ([0xEB, 0x09, 0x88, 0x00], 1)] { + let expected = InBand { + object_type: 2, + sample_rate: 24_000, + channel_config, + program_config: None, + }; + assert_eq!(in_band(&asc).unwrap(), expected); + } + + // A config naming SBR but stopping before its core has no core to name. + assert!(matches!(in_band(&[0x2A, 0x10]), Err(Error::IncompleteConfig))); + } + + #[test] + fn in_band_of_explicit_sbr_keeps_its_program_config_element() { + let in_band = in_band(&sbr_pce_asc()).unwrap(); + assert_eq!(in_band.object_type, 2); + assert_eq!(in_band.sample_rate, 24_000); + assert_eq!(in_band.channel_config, 0); + let pce = in_band.program_config.expect("a program config element"); + let mut block = pce.as_ref(); + let asc = in_band_config(2, 24_000, 0, &mut block).unwrap(); + assert_eq!(Config::parse(&mut asc.as_ref()).unwrap().channel_count, 4); } #[test] diff --git a/rs/moq-mux/src/container/ts/adts.rs b/rs/moq-mux/src/container/ts/adts.rs index ac1aec68232..632947bdce4 100644 --- a/rs/moq-mux/src/container/ts/adts.rs +++ b/rs/moq-mux/src/container/ts/adts.rs @@ -66,8 +66,12 @@ pub(super) fn write_header( channel_config: u8, raw_len: usize, ) -> anyhow::Result<[u8; 7]> { - // ADTS `profile` is the 2-bit audioObjectType - 1. - let profile = object_type.saturating_sub(1) & 0x03; + // ADTS `profile` is the 2-bit audioObjectType - 1, so only Main, LC, SSR, and LTP fit. + anyhow::ensure!( + (1..=4).contains(&object_type), + "audioObjectType {object_type} not representable in ADTS" + ); + let profile = object_type - 1; let freq_index = freq_index_from_rate(sample_rate)?; // ADTS has 3 bits for it; the higher configurations only fit an AudioSpecificConfig. anyhow::ensure!( @@ -101,12 +105,13 @@ fn freq_index_from_rate(sample_rate: u32) -> anyhow::Result { .with_context(|| format!("sample rate {sample_rate} not representable in ADTS")) } -/// Map a channel count to an AAC `channel_config` (ISO 14496-3 Table 1.19). -pub(super) fn channel_config_from_count(channel_count: u32) -> u8 { +/// Map a channel count to the ADTS `channel_config` (ISO 14496-3 Table 1.19) naming it, refusing a +/// count that none does rather than guess a layout. +pub(super) fn channel_config_from_count(channel_count: u32) -> anyhow::Result { match channel_count { - 1..=6 => channel_count as u8, - 8 => 7, - _ => 2, + 1..=6 => Ok(channel_count as u8), + 8 => Ok(7), + _ => anyhow::bail!("{channel_count} channels have no ADTS channelConfiguration"), } } @@ -134,6 +139,25 @@ mod tests { assert!(Header::parse(&header).is_err()); } + #[test] + fn write_refuses_object_types_outside_the_profile_field() { + for object_type in [0, 5, 29] { + assert!(write_header(object_type, 48_000, 2, 10).is_err()); + } + for object_type in 1..=4 { + let header = write_header(object_type, 48_000, 2, 10).unwrap(); + assert_eq!(Header::parse(&header).unwrap().object_type, object_type); + } + } + + #[test] + fn channel_counts_without_a_configuration_are_refused() { + assert_eq!(channel_config_from_count(8).unwrap(), 7); + for count in [0, 7, 9, 24] { + assert!(channel_config_from_count(count).is_err()); + } + } + #[test] fn frame_len_for_5_1() { let header = write_header(2, 44_100, 6, 512).unwrap(); diff --git a/rs/moq-mux/src/container/ts/export.rs b/rs/moq-mux/src/container/ts/export.rs index c53a7c141f5..07ac0bc49fb 100644 --- a/rs/moq-mux/src/container/ts/export.rs +++ b/rs/moq-mux/src/container/ts/export.rs @@ -37,7 +37,7 @@ use moq_net::Timestamp; use crate::catalog::hang::Catalog; use crate::catalog::{CatalogFormat, Stream}; -use crate::codec::annexb; +use crate::codec::{aac, annexb}; use crate::container::{ExportSource, Frame}; use super::adts; @@ -257,12 +257,7 @@ enum Kind { /// AAC, framed as ADTS. A `channel_config` of 0 defers the layout to a program config /// element, which leads the next raw data block written and is then taken. A catalog update /// rebuilds the kind and so repeats it once, which a decoder tuning in mid-stream welcomes. - Aac { - object_type: u8, - sample_rate: u32, - channel_config: u8, - program_config: Option, - }, + Aac(aac::InBand), /// Opus (private stream_type 0x06). Each frame is one Opus packet, prefixed with /// the Opus-in-TS access-unit control header and announced with the 'Opus' /// registration plus DVB extension descriptor. @@ -1185,7 +1180,7 @@ impl Export { tracks.iter().find(|t| { matches!( t.kind, - Kind::Aac { .. } | Kind::Opus { .. } | Kind::Mp2 { .. } | Kind::Ac3 | Kind::Eac3 + Kind::Aac(_) | Kind::Opus { .. } | Kind::Mp2 { .. } | Kind::Ac3 | Kind::Eac3 ) }) }) @@ -1197,7 +1192,7 @@ impl Export { .map(|t| { let stream_type = match &t.kind { Kind::Video(stream_type) => *stream_type, - Kind::Aac { .. } => StreamType::AdtsAac, + Kind::Aac(_) => StreamType::AdtsAac, // Opus rides private-data PES; the registration + extension descriptors // below tell the demuxer it's Opus. Kind::Opus { .. } => StreamType::from_u8(0x06).map_err(anyhow::Error::msg)?, @@ -1348,8 +1343,8 @@ impl Export { let track = self.tracks.get_mut(name).context("missing track")?; let pid = track.pid; let kind = track.kind.clone(); - if let Kind::Aac { program_config, .. } = &mut track.kind { - program_config.take(); + if let Kind::Aac(aac) = &mut track.kind { + aac.program_config.take(); } let is_video = matches!(kind, Kind::Video(_)); let timestamp = frame.timestamp; @@ -1360,15 +1355,10 @@ impl Export { // verbatim streams carry no PES payload; the section is written separately below. let es_payload = match &kind { Kind::Video(stream_type) => Some(video_es_payload(*stream_type, track.source.description(), &frame)?), - Kind::Aac { - object_type, - sample_rate, - channel_config, - program_config, - } => { - let pce = program_config.as_deref().unwrap_or_default(); + Kind::Aac(aac) => { + let pce = aac.program_config.as_deref().unwrap_or_default(); let raw_len = pce.len() + frame.payload.len(); - let header = adts::write_header(*object_type, *sample_rate, *channel_config, raw_len)?; + let header = adts::write_header(aac.object_type, aac.sample_rate, aac.channel_config, raw_len)?; let mut framed = Vec::with_capacity(header.len() + raw_len); framed.extend_from_slice(&header); framed.extend_from_slice(pce); @@ -2097,18 +2087,18 @@ fn video_es_payload(stream_type: StreamType, description: Option<&Bytes>, frame: fn audio_kind(config: &AudioConfig, name: &str) -> anyhow::Result { ensure_raw(&config.container, "audio", name)?; match &config.codec { - AudioCodec::AAC(aac) => { - // The description names the layout exactly; without one, the count is all there is. - let (channel_config, program_config) = match &config.description { - Some(asc) => crate::codec::aac::in_band_channels(asc)?, - None => (adts::channel_config_from_count(config.channel_count), None), - }; - Ok(Kind::Aac { - object_type: aac.profile, - sample_rate: config.sample_rate, - channel_config, - program_config, - }) + AudioCodec::AAC(codec) => { + // The description is exact, and names the LC core under explicit SBR or PS. Without + // one, the catalog is all there is. + Ok(Kind::Aac(match &config.description { + Some(asc) => aac::in_band(asc)?, + None => aac::InBand { + object_type: codec.profile, + sample_rate: config.sample_rate, + channel_config: adts::channel_config_from_count(config.channel_count)?, + program_config: None, + }, + })) } AudioCodec::Mp2 => Ok(Kind::Mp2 { sample_rate: config.sample_rate, diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 00d9294308c..f8847cffc2c 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -1484,6 +1484,91 @@ async fn aac_program_config_roundtrip() { assert_eq!(roundtripped, ingested, "the element is written once, not per frame"); } +/// GStreamer 1.28 `fdkaacenc` output, 48 kHz stereo, remuxed to FLV by ffmpeg 9.0.1. Its +/// AudioSpecificConfig signals SBR (and PS for v2) explicitly: object type 5 or 29 over an LC core +/// at 24 kHz, stereo for v1 and mono for v2. ADTS has two bits for the object type, so export +/// labels the LC core at its own rate and layout and leaves SBR and PS to implicit signaling, the +/// header ffmpeg's ADTS muxer writes too. ffprobe reads the result back as HE-AAC or HE-AACv2 at +/// 48 kHz stereo, like the source. +#[tokio::test(start_paused = true)] +async fn aac_explicit_sbr_exports_its_lc_core() { + let fixtures: [(&[u8], u8); 2] = [ + (include_bytes!("test_data/he_aac.flv"), 2), + (include_bytes!("test_data/he_aac_v2.flv"), 1), + ]; + for (data, channel_config) in fixtures { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + let mut import = crate::container::flv::Import::new(broadcast, catalog.reserve()); + import.decode(data).unwrap(); + import.finish().unwrap(); + + let snapshot = catalog.snapshot(); + let (name, _) = snapshot.audio.renditions.iter().next().expect("an AAC track"); + let ingested = read_frames(&consumer, name, Kind::Audio).await; + assert!(!ingested.is_empty(), "no AAC frames"); + + let ts = drain(consumer).await; + assert_packet_aligned(&ts); + + let (header, block) = first_adts_frame(&ts); + assert_eq!(header.object_type, 2, "the LC core, not a masked SBR or PS"); + assert_eq!(header.sample_rate, 24_000, "the core rate, not the output rate"); + assert_eq!(header.channel_config, channel_config); + assert_eq!(block, ingested[0], "the raw data block is untouched"); + } +} + +/// Without a description, the catalog is all ADTS has to label a track with. A channel count no +/// channelConfiguration names is refused rather than written as stereo, and HE-AAC, whose core +/// rate and layout only a description names, is refused rather than masked to AAC Main. +#[tokio::test(start_paused = true)] +async fn aac_export_refuses_what_adts_cannot_label() { + for (profile, channel_count, refusal) in [(2, 7, "7 channels"), (5, 2, "audioObjectType 5")] { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut catalog = crate::catalog::Producer::new(&mut broadcast, crate::catalog::Config::default()).unwrap(); + + let track = broadcast + .create_track( + broadcast.unique_name(".aac"), + hang::container::track_info(hang::catalog::PRIORITY.audio), + ) + .unwrap(); + let mut cfg = AudioConfig::new(AAC { profile }, 48_000, channel_count); + cfg.container = Container::Legacy; + catalog + .modify() + .unwrap() + .audio + .renditions + .insert(track.name().to_string(), cfg); + + let mut producer = Producer::new(track, HangContainer::Legacy(crate::container::Kind::Data)); + producer + .write(Frame { + timestamp: Timestamp::from_millis(0).unwrap(), + duration: None, + payload: Bytes::from_static(&[0x01, 0x02]), + keyframe: true, + }) + .unwrap(); + producer.finish().unwrap(); + + let mut exporter = Export::new(crate::source::announced(&consumer)).await.unwrap(); + let err = loop { + match tokio::time::timeout(Duration::from_secs(1), exporter.next()).await { + Ok(Ok(Some(_))) => continue, + Ok(Ok(None)) => panic!("export completed; expected a refusal naming {refusal}"), + Ok(Err(e)) => break e, + Err(_) => panic!("export neither errored nor completed"), + } + }; + assert!(err.to_string().contains(refusal), "expected {refusal}, got: {err}"); + } +} + /// The ffmpeg E-AC-3 fixture must survive TS -> MoQ -> TS byte-for-byte in an /// audio-only program; the PMT re-announces ATSC 0x87 with the 'EAC3' /// registration descriptor. diff --git a/rs/moq-mux/src/container/ts/test_data/he_aac.flv b/rs/moq-mux/src/container/ts/test_data/he_aac.flv new file mode 100644 index 0000000000000000000000000000000000000000..5bf778ce3c02107160e456ee1dca1f0b63c0e1b5 GIT binary patch literal 3110 zcmd5;c~BEq9DW-z5D+90P*g0&0wP|KQILWm96}Y3P!$9f2_$BM=8*dmQP4q-;ZhML zU~L>xK`Mw^AP_-X7!fRrNWl=qyFo5f5eVtqBv?D`Uv^;nZQkzNcl_S>e);y>b=%~H zU;zNyuz*kNY*l7@$WBqqDel zI0W`E0L9ToIT`3H+e2t<-T{WH=N0dtJt3@UIESR_0Gc78j9>;AL4dAub|9C|2xQR# z#Y{b52AdYd4c;3NOykfo0L$jBwg!4^76-P4GlD}PF$_RRfDp)K^B~WF0}R#J5DdVh zgJD_43gNP0u?Dz(tOE=cfx=Dd1z-*k0Qah?zAphw2}x+302_-Y8xbOLJusvqZhc}v zR5s4k1xUQ!K!D+m{}-wtyIYWYBLGWLq9q8?qbXirp-^BYQQ_z8FK-v?wC@RPeOoqB z;5c@0c8=qS+_C@V$a;GDuNmt?aoM{u7B{fNH8s-y>NtZmM?%-E*CuCU=z17mY>83d zp_0HS8@2B0@SWS5iqs)&ifl z*h}ZawqX;+LhIUn!l?b)9?|cKa*lL7(_N%VF#{LsfP=n%Z)pWBzx)vf<1L| z(`OUz6+8m_seNDpW??S&+lrWH>uB=KYSpFcc{4ta&rYd0H8(3m?BAfI)A3yGEB5}IvRqR{tlwrEI zO|SxG4S@&`IG>&#xPs>Wa8tHa zF069=s>>3)X-h>kxuIii^DpMlx|iXH%I&D4^<~VmefKlzTFq-%Gn4^f46Zl+;xs@4 zmr_gSGnbiA8J{-Jh(K=XY$+LkdjvkHXW!J7{FJAW@!ZaJ^j59&x)$@-L^<8e%_2Or)2vFu0l$+xyq+OjCAU$9H<*r{-P+zwc1|?S(U$_nL7T zMJkmJ)6%)QPc*hPXCy_svK0F+*vIV|Qg3>MhznS!R@R>dNri1zV zBU=N<-P6X~3~?)-iS6#(yp7%9$=uAh3GH~uE=CXa&A(i_pS%M88ea%&dSfo*w9lvC zJ_`$S5}!+fUkK&Lfruy}N*14D-}uY}MR2>n!|g@`%qL?yDT?i*`IF*#k&1j0ej74e sek{HgxzjrlDRScKv{q%ghGO*PVD`6gE`GJ@yIKUqe*ejeP$I|v1S3oeTL1t6 literal 0 HcmV?d00001 diff --git a/rs/moq-mux/src/container/ts/test_data/he_aac_v2.flv b/rs/moq-mux/src/container/ts/test_data/he_aac_v2.flv new file mode 100644 index 0000000000000000000000000000000000000000..1df313b4099b5f084b889f928d0e7a24784f352c GIT binary patch literal 3297 zcmd5;dpOf;9Djc^8Z}Kxky-T*>-|z3c&+lFB;)Y-W0Lrj{ zX#fnOH3SBzu!7byLTPriP#O*nr~n+D6G97RvVwqhce8GNw~sBr(>Qb{iw>8-83v$O zQ)*CSFjYFvrUeEEFk~}5!P1F;%&?ieppA)a0w}RV86gZ7f`BPfYhMO~R*3hO5Lyr&1F(U?q@@5K$P9vGTt)~R62kz*1gQD40)rvXEddPKS~dm{(8aKr zGTE#^Se63ZR%QT0M&Ls2_yI5m2!OsSjra4xYvlyAtp*1#$_EjIqANJHb#NXc9l@Rl zO2bhq9zP(wAi-itJr96IDA#;Q-Bxglry7O!6ykdA5Q+6Lr0@FU(MP$LK<+I7Y(&{Y z?t~-h?QCv_^W`SpGHt#5F+4})<^RfTjK~sKmgJT>jdc!1uuh0ncC^_sUaMB|c}b}6QA z^=F&boo2sjkL|fvndIQ9TJ6+Serx1TdfUWlAW@U z^FU-qCSNVy)Of$H?$0+PqSxMe*w|x9<{Y;ivCtCnHc#>J89Fs;OFXH+yu-65R0Z`2 zGpGy3h@${lz)lZzH6H*0IsskP__(Cd9=GDTXIoXc*!o1&!MXgyV8nbUk8G7q9c8-za6E(vhcg%Q)@cUe|egtXAcZuZ$v5dt41` zA>qt_LOV2%m*u+<-3rYa67KL$in{I84UTUmE1L&La*UClM48MWa+x7O>MLN_6$sHc)~CI1XsfPGx}$l2@9)iTdhT?Au4VTta^F_;C-z*$ zT*~JNQ9RCaKg+PY`Cz3Y>H#tf%of@$WKJ}A@GLa!c>wt$GmyKfCDu^6LtI%NY5C3_ zsi#t@+j5hSfZ6X)M_3gv=yIjL#Na)y`3rMqxX-TeeN}IOa>mtw)-oOE?}nOj?l5Hp zuqjU&l4Sx;2T8D@5sjPks`+t?JSlyBeIG^Z-Y`ri#pW3h8X}1E%kM~dqHs}F@`b2; zuSy=5E1nt?x@D8glPoH_m6`ylB-tLGD5On-XN;$eTr^gaW@wm_6AbwG4|b1+^CGKw zKW`EwCW_WONl;(0{7wbk#LjLd=czSBE}y42yeH^J2(kt7T8(y?6A2E(^;myU86GEsCz38#c@e8Nm4+bg{8Y~Vwckih$@h?&uPTF) ztBmwLGfnNr3^i-xCl9Mf2kqni{QPcDi8xNnQdn1a0LL?Kx>4aBH_)zx!j4Rk@V9Gx zjrMEXiH-J`mxes-B7l*r%o5V#{arqawwj&~!!kOZ)yoh=AWT=-@lx;(WN$2beDn${ z(7Ajlp8VH>&BK22qvtk`I24FG3njugR4?<6((6SfME&zc4YE&Zle{IX3$kw|X&G%R zYPfo~E(12k`vuvD9PLK*&pQyL0f9fsT4q1F+%vjzmyXxI0_%VKHI+Fsd)K636E4|lK%z_QapPA literal 0 HcmV?d00001 From 437159c8e128c498572335a2a726c8d532b0f96c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 26 Sep 2026 18:47:12 -0700 Subject: [PATCH 11/20] feat(ts): import surround Opus with a family 1 OpusHead (#4282) Co-authored-by: Claude Opus 5.5 --- doc/lib/rs/moq-mux.md | 2 +- quest/m1/audio-codecs/README.md | 4 +- quest/m1/audio-codecs/gst-opus-surround.md | 21 ++ .../m1/audio-codecs/ts-opus-channel-codes.md | 27 +++ .../audio-codecs/ts-opus-export-refusals.md | 23 +++ quest/m1/audio-codecs/ts-opus-surround.md | 20 -- rs/moq-audio/tests/import.rs | 64 +++++++ rs/moq-mux/src/codec/opus/import.rs | 2 +- rs/moq-mux/src/codec/opus/mapping.rs | 179 ++++++++++++++++++ rs/moq-mux/src/codec/opus/mod.rs | 154 ++++++--------- rs/moq-mux/src/container/ts/import.rs | 23 ++- rs/moq-mux/src/container/ts/import_test.rs | 23 +++ .../src/container/ts/test_data/opus_5_1.ts | Bin 0 -> 5452 bytes 13 files changed, 412 insertions(+), 130 deletions(-) create mode 100644 quest/m1/audio-codecs/gst-opus-surround.md create mode 100644 quest/m1/audio-codecs/ts-opus-channel-codes.md create mode 100644 quest/m1/audio-codecs/ts-opus-export-refusals.md delete mode 100644 quest/m1/audio-codecs/ts-opus-surround.md create mode 100644 rs/moq-audio/tests/import.rs create mode 100644 rs/moq-mux/src/codec/opus/mapping.rs create mode 100644 rs/moq-mux/src/container/ts/test_data/opus_5_1.ts diff --git a/doc/lib/rs/moq-mux.md b/doc/lib/rs/moq-mux.md index a3fd8d42e88..4d9f6b115f5 100644 --- a/doc/lib/rs/moq-mux.md +++ b/doc/lib/rs/moq-mux.md @@ -14,7 +14,7 @@ Turns existing container formats into hang broadcasts and back. This is what | Format | Import | Export | Notes | | --- | --- | --- | --- | | fMP4 / CMAF | yes | yes | Passthrough as `cmaf` or repackaged as `legacy`. | -| MPEG-TS | yes | yes | H.264/H.265; AAC, MP2, AC-3, E-AC-3; SCTE-35 and subtitle PIDs carried as tracks; service tables round-trip; signalled timebase discontinuities preserved; paced export. | +| MPEG-TS | yes | yes | H.264/H.265; AAC, MP2, AC-3, E-AC-3, Opus up to 7.1; SCTE-35 and subtitle PIDs carried as tracks; service tables round-trip; signalled timebase discontinuities preserved; paced export. | | FLV / RTMP | yes | yes | Legacy H.264 + AAC + MP3, plus enhanced-RTMP HEVC, AV1, VP9, Opus, AC-3, E-AC-3, and multitrack. | | Matroska / WebM | yes | yes | | | Annex-B (H.264, H.265) | yes | yes | Parameter sets extracted to the catalog or re-injected per keyframe. | diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 762769dc5ae..6dc3b3e0d16 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -45,10 +45,12 @@ its own decode and encode quest so verification stays per host. - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core +- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the TS exporter refuses Opus heads its channel code cannot describe instead of mislabeling them +- [TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) - TS Opus with a channel code of 0x81 or above imports with its real head or is refused, never guessed as stereo - [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC -- [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes - [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - `Config::encode` refuses channel counts it cannot name instead of writing stereo - [AAC parse truncated SBR](/quest/m1/audio-codecs/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core +- [GStreamer surround Opus](/quest/m1/audio-codecs/gst-opus-surround.md) - the moq-gst sink publishes 3 to 8 channel Opus with the OpusHead its caps describe ## Related diff --git a/quest/m1/audio-codecs/gst-opus-surround.md b/quest/m1/audio-codecs/gst-opus-surround.md new file mode 100644 index 00000000000..8b7a195ebcf --- /dev/null +++ b/quest/m1/audio-codecs/gst-opus-surround.md @@ -0,0 +1,21 @@ +# [S] Surround Opus from GStreamer + +## Goal + +`moq-gst`'s sink publishes a 3 to 8 channel `audio/x-opus` stream with the +OpusHead its caps describe, so surround Opus from `opusenc` plays natively like +the same stream from Matroska or MPEG-TS. + +## Plan + +- The sink refuses more than two channels because it builds the head from + `channels` and `rate` alone. Multichannel Opus caps also carry + `channel-mapping-family`, `stream-count`, `coupled-count`, and + `channel-mapping`; build the mapping with `opus::Mapping::new`, which + validates them, and refuse caps that omit or contradict them. +- Prefer a `streamheader` OpusHead when the caps carry one, if that path is + simpler and covers the same streams. +- Regression: an `opusenc` 5.1 pipeline publishes a family 1 description that + `moq-audio` decodes to six channels. + +Public API: none. Wire: none. diff --git a/quest/m1/audio-codecs/ts-opus-channel-codes.md b/quest/m1/audio-codecs/ts-opus-channel-codes.md new file mode 100644 index 00000000000..2fbce343d24 --- /dev/null +++ b/quest/m1/audio-codecs/ts-opus-channel-codes.md @@ -0,0 +1,27 @@ +# [S] TS Opus import refuses or parses extended channel codes + +## Goal + +An MPEG-TS Opus stream whose extension descriptor carries a +`channel_config_code` of 0x81 or above imports with its real OpusHead or is +refused. It never decodes as a guessed stereo stream, which misreads every +multistream packet. + +## Plan + +- The importer falls back to stereo for codes it does not know. 0x81 is + specified as an explicitly coded layout (channel count, mapping family, stream + counts, and table in the descriptor), and ffmpeg's muxer also writes + `0x80 | channels` for an alternate table its own demuxer does not read. Check + the Opus-in-TS spec and ffmpeg before deciding which codes to parse. +- Build parsed layouts with the public `opus::Mapping::new`, which validates + the table; refuse any code that is reserved or does not parse, per stream, + without failing the rest of the program. +- Fixtures from a real muxer where one exists; a hand-built descriptor + otherwise, noted as such. + +Public API: none. Wire: none. + +## Related + +- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the exporter side of the same descriptor diff --git a/quest/m1/audio-codecs/ts-opus-export-refusals.md b/quest/m1/audio-codecs/ts-opus-export-refusals.md new file mode 100644 index 00000000000..c501cb032f5 --- /dev/null +++ b/quest/m1/audio-codecs/ts-opus-export-refusals.md @@ -0,0 +1,23 @@ +# [S] TS Opus export refuses what it cannot label + +## Goal + +`moq-mux`'s MPEG-TS exporter never labels an Opus track with a channel +configuration its packets do not have. A track whose OpusHead the Opus +extension descriptor can describe exports as today; anything else is refused, +not written with a guessed `channel_config_code`. + +## Plan + +- The exporter derives `channel_config_code` from the catalog channel count + alone, clamped to 1..=8, so a family 255 or ambisonics head, a family 1 head + with a non-Vorbis table, or more than eight channels exports mislabeled. + Decide from the parsed description: family 0, and family 1 with the Vorbis + default mapping, keep the plain code; refuse the rest, or write the explicit + layout if the import side of + [TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) settles + one both sides agree on. +- A track with no description stays mono or stereo only. +- Tests with real heads, checked against ffprobe where ffmpeg reads the result. + +Public API: none. Wire: none; TS output changes only for inputs it mislabeled. diff --git a/quest/m1/audio-codecs/ts-opus-surround.md b/quest/m1/audio-codecs/ts-opus-surround.md deleted file mode 100644 index 5e6e6be4fc3..00000000000 --- a/quest/m1/audio-codecs/ts-opus-surround.md +++ /dev/null @@ -1,20 +0,0 @@ -# [S] Surround Opus from MPEG-TS - -## Goal - -An MPEG-TS Opus stream with 3 to 8 channels imports with an OpusHead that -decodes, so it plays natively like the same stream from Matroska or FLV. - -## Plan - -- The TS importer knows only the `channel_config_code` from the Opus - extension descriptor and builds `opus::Config::new(48_000, channels)`, whose - `encode` refuses more than two channels. The catalog then carries no - description and `moq-audio` refuses the track. -- Codes 3 to 8 mean family 1 with the Vorbis default stream and coupled counts - and mapping (the Vorbis orders of RFC 7845 §5.1.1.2), so the - importer can synthesize the head. That needs a way to build a family 1 - `Config` and have `encode` emit its table; decide whether that is a - constructor on `opus::Mapping` or a TS-local head writer. -- Regression: a 5.1 TS fixture imports with a family 1 description and decodes - to six channels. diff --git a/rs/moq-audio/tests/import.rs b/rs/moq-audio/tests/import.rs new file mode 100644 index 00000000000..010c1976c4e --- /dev/null +++ b/rs/moq-audio/tests/import.rs @@ -0,0 +1,64 @@ +//! Container imports that must decode end to end: a stream `moq-mux` imports +//! plays through [`decode::Consumer`] like the same stream from any container. + +use std::time::Duration; + +use moq_audio::{Format, Layout, decode}; + +/// MPEG-TS names surround Opus only by channel count, so the importer has to +/// synthesize the OpusHead mapping table or the decoder refuses the track. +#[tokio::test] +async fn ts_surround_opus_decodes() { + // A 440 Hz center channel in 5.1; see `import_opus_surround_catalog` in moq-mux. + let data = include_bytes!("../../moq-mux/src/container/ts/test_data/opus_5_1.ts"); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let broadcast_consumer = broadcast.consume(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let mut import = moq_mux::container::ts::Import::new(broadcast, catalog.reserve()); + import.decode(&data[..]).unwrap(); + import.finish().unwrap(); + + let snapshot = catalog.snapshot(); + let (name, config) = snapshot.audio.renditions.iter().next().expect("an Opus track"); + + let mut options = decode::Options::default(); + options.output.format = Format::F32; + // The whole file is imported before anything decodes it. + options.max_age = Duration::from_secs(30); + let mut consumer = decode::Consumer::new(&broadcast_consumer, config, name, options) + .await + .unwrap(); + assert_eq!(consumer.layout(), Layout::FivePointOne); + + // Canonical 5.1 order: FL, FR, FC, LFE, SL, SR. + let mut energy = [0.0f64; 6]; + let mut frames = 0; + while let Some(frame) = tokio::time::timeout(Duration::from_secs(5), consumer.read()) + .await + .expect("decoded frame timed out") + .unwrap() + { + let (samples, rest) = frame.data.as_chunks::<4>(); + assert!(rest.is_empty()); + for chunk in samples.as_chunks::<6>().0 { + for (channel, sample) in energy.iter_mut().zip(chunk) { + *channel += f32::from_le_bytes(*sample).powi(2) as f64; + } + frames += 1; + } + } + + assert!(frames > 12_000, "expected most of 0.5 s at 48 kHz, got {frames} frames"); + let center = energy[2]; + assert!( + center / frames as f64 > 0.001, + "center should carry the tone: {energy:?}" + ); + for (channel, &other) in energy.iter().enumerate().filter(|&(channel, _)| channel != 2) { + assert!( + other < center / 100.0, + "channel {channel} should be near silent: {energy:?}" + ); + } +} diff --git a/rs/moq-mux/src/codec/opus/import.rs b/rs/moq-mux/src/codec/opus/import.rs index 69b4631c86f..829bef3736c 100644 --- a/rs/moq-mux/src/codec/opus/import.rs +++ b/rs/moq-mux/src/codec/opus/import.rs @@ -115,7 +115,7 @@ impl Import { pub fn config(init: &[u8]) -> crate::Result { let mut buf = init; let mut config: hang::catalog::AudioConfig = Config::parse(&mut buf)?.into(); - // Publish the head as given: re-encoding it would drop a channel mapping table. + // Publish the head as given: re-encoding it would drop the output gain. config.description = Some(bytes::Bytes::copy_from_slice(init)); Ok(config) } diff --git a/rs/moq-mux/src/codec/opus/mapping.rs b/rs/moq-mux/src/codec/opus/mapping.rs new file mode 100644 index 00000000000..4157dd5303f --- /dev/null +++ b/rs/moq-mux/src/codec/opus/mapping.rs @@ -0,0 +1,179 @@ +//! The OpusHead channel mapping table (RFC 7845 §5.1.1). + +use bytes::Buf; + +use super::{Error, Result}; + +/// An OpusHead channel mapping table (RFC 7845 §5.1.1), present for every family but 0. +#[derive(Clone, Copy, PartialEq, Eq)] +pub struct Mapping { + family: u8, + streams: u8, + coupled: u8, + channels: u8, + // Sized for the most channels any family allows, so `opus::Config` stays `Copy`. + table: [u8; 255], +} + +/// The fields of a channel mapping table, checked into a [`Mapping`] by [`Mapping::new`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Config<'a> { + /// The channel mapping family; 0 has no table and is refused. + pub family: u8, + /// Number of Opus streams in each packet. + pub streams: u8, + /// How many of those streams are coupled (stereo); they come first. + pub coupled: u8, + /// The decoded channel feeding each output channel, 255 for silence. + pub table: &'a [u8], +} + +/// The family 1 stream counts and table for 1 to 8 channels, as libopus and +/// RFC 7845 §5.1.1.2 lay out the Vorbis channel orders. +const VORBIS: [(u8, u8, &[u8]); 8] = [ + (1, 0, &[0]), + (1, 1, &[0, 1]), + (2, 1, &[0, 2, 1]), + (2, 2, &[0, 1, 2, 3]), + (3, 2, &[0, 4, 1, 2, 3]), + (4, 2, &[0, 4, 1, 2, 3, 5]), + (4, 3, &[0, 4, 1, 2, 3, 5, 6]), + (5, 3, &[0, 6, 1, 2, 3, 4, 5, 7]), +]; + +impl Mapping { + /// The family 1 mapping a surround encoder uses for `channels` in Vorbis + /// order, for sources that name only a channel count. + pub(crate) fn vorbis(channels: u8) -> Result { + let (streams, coupled, table) = *channels + .checked_sub(1) + .and_then(|index| VORBIS.get(index as usize)) + .ok_or(Error::UnsupportedChannelCount(channels as u32))?; + Self::new(Config { + family: 1, + streams, + coupled, + table, + }) + } + + /// Check `config` into a mapping with one output channel per table entry. + /// + /// Refuses family 0, a channel count the family does not allow, and a table + /// that names no streams, more coupled streams than streams, or a channel + /// past the decoded ones. + pub fn new(config: Config<'_>) -> Result { + let Config { + family, + streams, + coupled, + table, + } = config; + let max_channels = match family { + 0 => return Err(Error::UnsupportedMappingFamily(0)), + 1 => 8, + _ => 255, + }; + if table.is_empty() || table.len() > max_channels { + return Err(Error::UnsupportedChannelCount(table.len() as u32)); + } + + let decoded = streams as u32 + coupled as u32; + if streams == 0 || coupled > streams || decoded > 255 { + return Err(Error::InvalidMappingTable); + } + if table.iter().any(|&entry| entry != 255 && entry as u32 >= decoded) { + return Err(Error::InvalidMappingTable); + } + + let mut padded = [0u8; 255]; + padded[..table.len()].copy_from_slice(table); + Ok(Self { + family, + streams, + coupled, + channels: table.len() as u8, + table: padded, + }) + } + + pub(super) fn parse(buf: &mut T, family: u8, channels: u8) -> Result { + if buf.remaining() < 2 + channels as usize { + return Err(Error::MappingTableTooShort); + } + let streams = buf.get_u8(); + let coupled = buf.get_u8(); + let mut table = [0u8; 255]; + buf.copy_to_slice(&mut table[..channels as usize]); + Self::new(Config { + family, + streams, + coupled, + table: &table[..channels as usize], + }) + } + + /// The channel mapping family: 1 is the Vorbis speaker order, 2 and 3 are + /// ambisonics, and 255 is channels with no declared position. + pub fn family(&self) -> u8 { + self.family + } + + /// Number of Opus streams in each packet. + pub fn streams(&self) -> u8 { + self.streams + } + + /// How many of those streams are coupled (stereo); they come first. + pub fn coupled(&self) -> u8 { + self.coupled + } + + /// The decoded channel feeding each output channel, 255 for silence. + pub fn table(&self) -> &[u8] { + &self.table[..self.channels as usize] + } +} + +impl std::fmt::Debug for Mapping { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Mapping") + .field("family", &self.family) + .field("streams", &self.streams) + .field("coupled", &self.coupled) + .field("table", &self.table()) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn new_validates_the_table() { + fn new(family: u8, streams: u8, coupled: u8, table: &[u8]) -> Result { + Mapping::new(Config { + family, + streams, + coupled, + table, + }) + } + + let mapping = new(255, 2, 0, &[0, 1, 255]).unwrap(); + assert_eq!(mapping.family(), 255); + assert_eq!((mapping.streams(), mapping.coupled()), (2, 0)); + assert_eq!(mapping.table(), &[0, 1, 255]); + + assert!(matches!(new(0, 1, 1, &[0, 1]), Err(Error::UnsupportedMappingFamily(0)))); + assert!(matches!(new(1, 1, 0, &[]), Err(Error::UnsupportedChannelCount(0)))); + assert!(matches!(new(1, 9, 0, &[0; 9]), Err(Error::UnsupportedChannelCount(9)))); + for (streams, coupled, table) in [(0, 0, &[0][..]), (1, 2, &[0]), (1, 1, &[2]), (200, 100, &[0])] { + assert!( + matches!(new(1, streams, coupled, table), Err(Error::InvalidMappingTable)), + "{streams} streams, {coupled} coupled, {table:?}" + ); + } + } +} diff --git a/rs/moq-mux/src/codec/opus/mod.rs b/rs/moq-mux/src/codec/opus/mod.rs index a73e9227a6c..749cce7217c 100644 --- a/rs/moq-mux/src/codec/opus/mod.rs +++ b/rs/moq-mux/src/codec/opus/mod.rs @@ -4,8 +4,10 @@ //! publishes raw Opus frames (no Ogg framing) to a moq broadcast. mod import; +pub mod mapping; pub use import::*; +pub use mapping::Mapping; use bytes::{Buf, Bytes}; @@ -37,8 +39,8 @@ pub enum Error { #[error("invalid OpusHead channel mapping table")] InvalidMappingTable, - /// [`Config::encode`] only emits channel mapping family 0. - #[error("cannot encode channel mapping family {0}")] + /// [`Mapping::new`] was given family 0, which has no mapping table. + #[error("channel mapping family {0} has no mapping table")] UnsupportedMappingFamily(u8), } @@ -58,79 +60,6 @@ pub struct Config { pub mapping: Option, } -/// An OpusHead channel mapping table (RFC 7845 §5.1.1), present for every family but 0. -#[derive(Clone, Copy, PartialEq, Eq)] -pub struct Mapping { - family: u8, - streams: u8, - coupled: u8, - channels: u8, - // Sized for the most channels any family allows, so `Config` stays `Copy`. - table: [u8; 255], -} - -impl Mapping { - fn parse(buf: &mut T, family: u8, channels: u8) -> Result { - if buf.remaining() < 2 + channels as usize { - return Err(Error::MappingTableTooShort); - } - let streams = buf.get_u8(); - let coupled = buf.get_u8(); - if streams == 0 || coupled > streams || streams as u32 + coupled as u32 > 255 { - return Err(Error::InvalidMappingTable); - } - - let mut table = [0u8; 255]; - for entry in &mut table[..channels as usize] { - *entry = buf.get_u8(); - // 255 marks a silent channel. - if *entry != 255 && *entry >= streams + coupled { - return Err(Error::InvalidMappingTable); - } - } - - Ok(Self { - family, - streams, - coupled, - channels, - table, - }) - } - - /// The channel mapping family: 1 is the Vorbis speaker order, 2 and 3 are - /// ambisonics, and 255 is channels with no declared position. - pub fn family(&self) -> u8 { - self.family - } - - /// Number of Opus streams in each packet. - pub fn streams(&self) -> u8 { - self.streams - } - - /// How many of those streams are coupled (stereo); they come first. - pub fn coupled(&self) -> u8 { - self.coupled - } - - /// The decoded channel feeding each output channel, 255 for silence. - pub fn table(&self) -> &[u8] { - &self.table[..self.channels as usize] - } -} - -impl std::fmt::Debug for Mapping { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("Mapping") - .field("family", &self.family) - .field("streams", &self.streams) - .field("coupled", &self.coupled) - .field("table", &self.table()) - .finish() - } -} - impl Config { /// Build a mono/stereo Opus config with no pre-skip. pub fn new(sample_rate: u32, channel_count: u32) -> Self { @@ -169,17 +98,9 @@ impl Config { buf.advance(2); // Skip gain until if/when we support it. let family = buf.get_u8(); - let max_channels = match family { - 0 => 2, - 1 => 8, - _ => 255, - }; - if channel_count == 0 || channel_count > max_channels { - return Err(Error::UnsupportedChannelCount(channel_count)); - } - let mapping = match family { - 0 => None, + 0 if (1..=2).contains(&channel_count) => None, + 0 => return Err(Error::UnsupportedChannelCount(channel_count)), family => Some(Mapping::parse(buf, family, channel_count as u8)?), }; @@ -195,28 +116,35 @@ impl Config { }) } - /// Encode the minimal OpusHead packet (19 bytes; channel mapping family - /// 0 and zero gain). + /// Encode an OpusHead packet (RFC 7845 §5.1) with zero gain, followed by + /// the channel mapping table when there is one. /// - /// Errors with [`Error::UnsupportedChannelCount`] unless `channel_count` is 1 - /// or 2, since mapping family 0 is only defined for mono/stereo per RFC 7845 §5.1. - /// Multi-channel streams need family 1 with a channel mapping table, which - /// this helper does not emit, so any `mapping` is [`Error::UnsupportedMappingFamily`]. + /// Errors with [`Error::UnsupportedChannelCount`] when `channel_count` is not + /// 1 or 2 without a `mapping` (family 0 is only defined for mono/stereo), or + /// is not the mapping's own channel count. pub fn encode(&self) -> Result { - if let Some(mapping) = &self.mapping { - return Err(Error::UnsupportedMappingFamily(mapping.family)); - } - if !(1..=2).contains(&self.channel_count) { + let valid = match &self.mapping { + None => (1..=2).contains(&self.channel_count), + Some(mapping) => self.channel_count == mapping.table().len() as u32, + }; + if !valid { return Err(Error::UnsupportedChannelCount(self.channel_count)); } - let mut head = Vec::with_capacity(19); + + let mut head = Vec::with_capacity(21 + self.channel_count as usize); head.extend_from_slice(b"OpusHead"); head.push(1); // version head.push(self.channel_count as u8); head.extend_from_slice(&self.pre_skip.to_le_bytes()); head.extend_from_slice(&self.sample_rate.to_le_bytes()); head.extend_from_slice(&0i16.to_le_bytes()); // output gain - head.push(0); // channel mapping family (0 = mono/stereo) + match &self.mapping { + None => head.push(0), + Some(mapping) => { + head.extend_from_slice(&[mapping.family(), mapping.streams(), mapping.coupled()]); + head.extend_from_slice(mapping.table()); + } + } Ok(Bytes::from(head)) } } @@ -322,8 +250,36 @@ mod tests { assert_eq!(mapping.coupled(), 2); assert_eq!(mapping.table(), &[0, 4, 1, 2, 3, 5]); - // Encode only emits family 0. - assert!(matches!(parsed.encode(), Err(Error::UnsupportedMappingFamily(1)))); + // The table survives a re-encode. + assert_eq!(parsed.encode().unwrap(), bytes); + } + + #[test] + fn encodes_the_vorbis_mappings() { + for channels in 1..=8u8 { + let mut config = Config::new(48_000, channels as u32); + config.mapping = Some(Mapping::vorbis(channels).unwrap()); + let head = config.encode().unwrap(); + assert_eq!(head.len(), 21 + channels as usize, "{channels} channels"); + assert_eq!( + Config::parse(&mut head.as_ref()).unwrap(), + config, + "{channels} channels" + ); + } + + // 5.1 is the table libopus and ffmpeg write. + let five_one = Mapping::vorbis(6).unwrap(); + assert_eq!((five_one.streams(), five_one.coupled()), (4, 2)); + assert_eq!(five_one.table(), &[0, 4, 1, 2, 3, 5]); + + assert!(matches!(Mapping::vorbis(0), Err(Error::UnsupportedChannelCount(0)))); + assert!(matches!(Mapping::vorbis(9), Err(Error::UnsupportedChannelCount(9)))); + + // The channel count must agree with the table. + let mut config = Config::new(48_000, 5); + config.mapping = Some(five_one); + assert!(matches!(config.encode(), Err(Error::UnsupportedChannelCount(5)))); } #[test] diff --git a/rs/moq-mux/src/container/ts/import.rs b/rs/moq-mux/src/container/ts/import.rs index 057d998aa1e..c8ac4d5090a 100644 --- a/rs/moq-mux/src/container/ts/import.rs +++ b/rs/moq-mux/src/container/ts/import.rs @@ -497,11 +497,10 @@ impl Import { // by an 'Opus' registration descriptor. Channels and the (always 48 kHz) rate // come from the descriptors, so the importer is built up front. StreamType::Mpeg2PacketizedData if registration_format(descriptors) == Some(*b"Opus") => { - let channel_count = opus_channel_count(descriptors).unwrap_or(2); + let config = opus_config(descriptors)?; let track = self .broadcast .unique_track(".opus", self.catalog.track_info(hang::catalog::PRIORITY.audio))?; - let config = opus::Config::new(48_000, channel_count); let mut config: hang::catalog::AudioConfig = config.into(); config.container = self.container.clone(); Stream::Opus(Box::new(OpusStream { @@ -2342,14 +2341,15 @@ fn registration_format(descriptors: &[mpeg2ts::ts::Descriptor]) -> Option<[u8; 4 .and_then(|s| s.try_into().ok()) } -/// The Opus channel count from the DVB extension descriptor (tag 0x7f, ext tag 0x80). +/// The OpusHead implied by the DVB extension descriptor (tag 0x7f, ext tag 0x80). /// /// `channel_config_code` follows the Opus-in-TS mapping (and ffmpeg's demuxer): 0 is dual -/// mono (decoded as stereo), 1..=8 is the channel count directly. Higher codes (0x81 -/// explicitly-coded layouts, reserved values) aren't supported, so they fall back to the -/// caller's default rather than being read as a raw 129..=255 count. -fn opus_channel_count(descriptors: &[mpeg2ts::ts::Descriptor]) -> Option { - descriptors +/// mono (decoded as stereo), 1..=8 is the channel count directly, with family 0 for mono and +/// stereo and the Vorbis family 1 mapping above that. Higher codes (0x81 explicitly-coded +/// layouts, reserved values) aren't supported, so they fall back to stereo rather than being +/// read as a raw 129..=255 count. +fn opus_config(descriptors: &[mpeg2ts::ts::Descriptor]) -> crate::Result { + let channels = descriptors .iter() .find(|d| d.tag == 0x7f && d.data.first() == Some(&0x80)) .and_then(|d| d.data.get(1)) @@ -2358,6 +2358,13 @@ fn opus_channel_count(descriptors: &[mpeg2ts::ts::Descriptor]) -> Option { 1..=8 => Some(cc as u32), _ => None, }) + .unwrap_or(2); + + let mut config = opus::Config::new(48_000, channels); + if channels > 2 { + config.mapping = Some(opus::Mapping::vorbis(channels as u8)?); + } + Ok(config) } /// Parse one Opus-in-TS access-unit control header, returning `(header_len, payload_size)`. diff --git a/rs/moq-mux/src/container/ts/import_test.rs b/rs/moq-mux/src/container/ts/import_test.rs index d0c48534ac6..46f8b2c792a 100644 --- a/rs/moq-mux/src/container/ts/import_test.rs +++ b/rs/moq-mux/src/container/ts/import_test.rs @@ -200,6 +200,29 @@ fn import_opus_catalog() { assert_eq!(audio.channel_count, 2); } +/// `opus_5_1.ts` is a 440 Hz center channel in 5.1, which ffmpeg's libopus +/// encodes as family 1 and its muxer labels `channel_config_code` 6: +/// `ffmpeg -f lavfi -i sine=frequency=440:sample_rate=48000:duration=0.5 +/// -ac 6 -c:a libopus -b:a 128k -f mpegts opus_5_1.ts`. The descriptor names only +/// the channel count, so the importer must synthesize the Vorbis mapping table +/// or the track has no OpusHead a decoder accepts. +#[test] +fn import_opus_surround_catalog() { + let data = include_bytes!("test_data/opus_5_1.ts"); + let catalog = import_ts(data); + + assert_eq!(catalog.audio.renditions.len(), 1, "expected one Opus track"); + let audio = catalog.audio.renditions.values().next().unwrap(); + assert_eq!(audio.channel_count, 6); + + let head = crate::codec::opus::Config::parse(&mut audio.description.as_deref().expect("an OpusHead")).unwrap(); + assert_eq!(head.channel_count, 6); + let mapping = head.mapping.expect("a family 1 mapping"); + assert_eq!(mapping.family(), 1); + assert_eq!((mapping.streams(), mapping.coupled()), (4, 2)); + assert_eq!(mapping.table(), &[0, 4, 1, 2, 3, 5]); +} + /// Opus frames from real ffmpeg output must decode: a non-empty run of Opus packets, /// each a plausible size (the control header was stripped, not left in the payload). #[tokio::test(start_paused = true)] diff --git a/rs/moq-mux/src/container/ts/test_data/opus_5_1.ts b/rs/moq-mux/src/container/ts/test_data/opus_5_1.ts new file mode 100644 index 0000000000000000000000000000000000000000..14ad7cb9c04960dcb68052e1a3177667cb971d93 GIT binary patch literal 5452 zcmdUzc{G&k1IOP{_PtEm%FTRI-LqG}PcXW;EyC)31B(xqsZ>HRrtZ&%E#V`9AOS`8@BO7j7sj0vJ;j z08kqMtbx`55IsuFVjGCd+&m!A%bVBE>(@}9lU`~N^hJ}7wSU(KZU~3~K#d@?I{=~r zO0`?-{#_M+*amLH=+03i&FuaLpaNWsFFww#sKAh53`Z2#&9TmfA71=_{sC?TsBzl> zK)668005w7PLqR209ig4x!K}$#w@OmJ) z?M;KG)nKuZy<5CkoO72%M=x&c(%LG9f?^M6obL@9r(==U<>Dn0Bkx_DM!udQXm~A_ z79j*j9nf(pe8xR#a1bDBp1LB_2MatzBWIQB9*aJr5XgUfF!P|aFv^H_wV9a;XQx7O zbt55o`eCTq`Szii(FV}sbmiBX+IJ(~Uy)Pv;aF^Xa}+pOsuic~R~9<%9^K57zxZ;T zrZRMq^M5Cq&Ah7aoo~Ol+8QJ=+?gIrG$?b76blfavgG1!`1J|;ZrH)_aBun72ljrJ za_xg2-hS3ekeOt`=&pVi_-7ZNeIH<~`+(56E3vq%8SkxLWv`_hZk_fM)u<}@LiYlw zzc|`zY4pXMm{oe!(V6Qc3u~Pv#S*{Ii1+IY#c)-06T2$9iCxuLclBDqhO4nmS9y2W zVkk)V;u_+TC~0?Z-!|0fA9U#(D!H&gq*T97ED^snS*qWW>sD)&X+h$a?5!Rxp2iKo zq%IGi(si>vc(&7*EXDVc1V%CewPWX*QH zhX7W*(>LvhqL%Nxpg`EmDe`3^Q!dQ;gGhlqN^xk&PS57*5V>*f)w+$WAWL?TJ}by< z<}sN6e%icL<7|lFkZ|6OUE=9PA|$?ZX(%_x9<|Mah%pqq|ndKkVvX+8N%iA6vfi;qr=ySi-?yQ;nJYM$nXtL7|My+9bsAvSC3IhHaE0}v4PsevVK z144a;@n`OZ`mViY68mGh0-7owyIW2)M)0@2IReAiEFlm)I9B0o~W`^z!|^x4hSSj`|Bb$MPIT%_RL#aS{> zIAmL8pZ^Bsxd-ygpDtUj30~8P^t<=cwBp%3P#-r&8M#e0P< zh5}`;s4AQHfa`k{M*G5wm)cdKEnn*p(epmlPVwO9PQ()gNkucUk4lYLCJsjIi#1s| zsLml_IaO7z5&DusMKV2VN2JRNmhT&-zN@>U`nkh9dnaGs7m0dds$~IR&^VQu;lj>9 zZNPq1>hno~wEp|p5CX!H!Mg-E#pdk=o(1B$8t8O!~ zJBSbLQfEiyW|{fU?0oB5!pFE{dWla;O;iJ~FuFs7|HRe5^c?vcC+XOlO6V2%Ai-@e z*FkHu_#R%{|K!mZILh~==NhVbnavYN>H|To!7xoR`s1`h8W#NW-1*|GBzMJq4G{IH zf%RE2{zr;!VpqjBv8%9kSKC}RTpeM#I#0k*e#cqOFY||dE#6aH`Z~^K6{hf<589Qv zGvDe=Df|@8_t)1H8|dtYaDYS*2e?$B*6?3 zBwoDqVEj9Zt0yI;Q<8LM^lypbo+=ersXeexDQf%24O`T~h0 z9=AYRc*<}F?4N$Z;9c69<5OY?4eQhO@dN5%%guqEg4bWY{Pf)jHOOvlbMvONCmMVd zxI33sGMR*&9g%s*fh;oWcw*J`}1kmID zMhUIhKW(3{d4_gPN?D|~dQv!r(On<<_g&ShJZxQ*`zQjtLYGiqUKvLPIjh%A_lod1 zrhGT|>Phj|Y+t*pgq#J-0P%69J^S699f!3c1$`kNSlcr|N$6~fe^u7hN)NwB(xbc; zVwBJp{|~+JH=+voIVI(c6^AX`Pwy#{B@gb?E2VD$ literal 0 HcmV?d00001 From 80072f73b75305429e7f7220915d32b0381ac422 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 28 Sep 2026 19:27:26 -0700 Subject: [PATCH 12/20] chore(quest): start named-codecs From 1af5bd0853fe841c3fecbc55ef9d269d995b2326 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 28 Sep 2026 19:33:09 -0700 Subject: [PATCH 13/20] fix(audio): Kind::Named keeps its codec names Co-Authored-By: Claude Sonnet 5.5 --- doc/lib/rs/moq-audio.md | 9 +- quest/m1/audio-codecs/README.md | 1 - quest/m1/audio-codecs/named-codecs.md | 27 ------ rs/moq-audio/src/decode/backend/mod.rs | 124 +++++++++++-------------- rs/moq-audio/src/decode/decoder.rs | 23 ++++- rs/moq-audio/src/encode/backend/mod.rs | 117 ++++++++++++----------- rs/moq-audio/src/encode/encoder.rs | 11 +-- rs/moq-audio/src/encode/producer.rs | 5 +- 8 files changed, 144 insertions(+), 173 deletions(-) delete mode 100644 quest/m1/audio-codecs/named-codecs.md diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 40e062bb04e..ba5a7256a3b 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -34,8 +34,10 @@ policy. Decoding likewise separates low-level `decode::Config`, PCM | `aec` | Acoustic echo cancellation (a port of WebRTC's), so a laptop with no headset doesn't feed itself back | `decode` picks a backend per track the way `moq-video` does: a platform decoder -first, then software. `decode::Config::kind` forces one (`Kind::Software`, or -`Kind::Named` with a name below), and `Decoder::name()` reports what opened. +first, then software. `decode::Config::kind` narrows it: `Kind::Software` skips +the platform tier, and `Kind::Named` requires the track to be a codec, by its +name (`"opus"`, `"pcm"`, or `"aac"`), and refuses any other. Backends are never +named; `Decoder::name()` reports which one opened. | Backend | Decodes | Hosts | | --- | --- | --- | @@ -57,7 +59,8 @@ catalog with neither. A description that is present but malformed (truncated, wrong signature, a new major version, or a channel count its mapping family forbids) is refused rather than falling back to the catalog's fields. -`encode` selects the same way, through `encode::Settings::kind`, and +`encode` selects the same way, through `encode::Settings::kind`, where +`Kind::Named` requires `Settings::codec` to match the codec name, and `Encoder::name()` reports what opened. | Backend | Encodes | Hosts | diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index 74b7a30195c..cc301af29ce 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -43,7 +43,6 @@ its own decode and encode quest so verification stays per host. ## Required -- [Named codecs](/quest/m1/audio-codecs/named-codecs.md) - `decode::Kind::Named` keeps the published codec names and picks backends internally; must land before the line merges - [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core - [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the TS exporter refuses Opus heads its channel code cannot describe instead of mislabeling them diff --git a/quest/m1/audio-codecs/named-codecs.md b/quest/m1/audio-codecs/named-codecs.md deleted file mode 100644 index 4bba08d514e..00000000000 --- a/quest/m1/audio-codecs/named-codecs.md +++ /dev/null @@ -1,27 +0,0 @@ -# [XS] Kind::Named keeps its codec names - -## Goal - -`moq_audio::decode::Kind::Named` accepts what published moq-audio 0.1.6 -accepts, the codec names (`"opus"`, `"aac"`, `"pcm"`), so the line can merge -to `main` without breaking callers. [#4131](https://github.com/moq-dev/moq/pull/4131) -switched it to backend names (`"libopus"`, `"symphonia"`, `"pcm"`) on the line -branch, which turns every existing `Named("opus")` or `Named("aac")` into -`Error::Unsupported` after an upgrade. This must land before the line PR -[#4081](https://github.com/moq-dev/moq/pull/4081) merges to `main`. - -## Plan - -- Decided: keep codec names; backends are picked internally. Which backend - decodes AAC (AudioToolbox, Media Foundation, symphonia) is the library's - choice through `Auto` and `Software`, not a string a caller has to know and - that changes per platform. This reverses #4131's reasoning on purpose: the - published contract wins over matching moq-video's backend naming. -- Make `encode::Kind` read the same way. It is unpublished, so drop `Named` - there or give it the same codec meaning, whichever leaves the two enums - mirrored; don't keep backend names on one side only. -- Tests that forced a backend by name move to a crate-private seam. -- Docs and the `Kind` doc comments list the codec names. - -Public API: `decode::Kind::Named` returns to its published meaning, so the -line stays additive on `main`. Wire: none. diff --git a/rs/moq-audio/src/decode/backend/mod.rs b/rs/moq-audio/src/decode/backend/mod.rs index f39bb4c170c..6b25cbab5d8 100644 --- a/rs/moq-audio/src/decode/backend/mod.rs +++ b/rs/moq-audio/src/decode/backend/mod.rs @@ -6,7 +6,8 @@ //! //! [`open`] tries the platform decoders before the software ones, skipping any //! that does not advertise the catalog codec, and refuses when none opens the -//! track. The software tier is libopus for Opus, a passthrough for PCM, and +//! track. [`Kind::Named`] names the codec, not a backend, so which backend +//! decodes it stays the library's choice. The software tier is libopus for Opus, a passthrough for PCM, and //! symphonia for AAC-LC mono and stereo (behind the `aac` feature). No platform //! decoder is wired in yet. @@ -42,7 +43,8 @@ pub(crate) trait Backend: Send { 0 } - /// The stable lowercase name [`Kind::Named`] selects this backend by. + /// The stable lowercase name of this backend, as [`Decoder::name`](super::Decoder::name) + /// reports it. Callers never select by it. fn name(&self) -> &str; } @@ -80,9 +82,28 @@ const SOFTWARE: &[Candidate] = &[ }, ]; +/// The name [`Kind::Named`] uses for a catalog codec, `None` for one this crate has no name for. +fn codec_name(codec: &AudioCodec) -> Option<&'static str> { + match codec { + AudioCodec::Opus => Some("opus"), + AudioCodec::Pcm => Some("pcm"), + AudioCodec::AAC(_) => Some("aac"), + _ => None, + } +} + /// Open the first backend that advertises the catalog codec and accepts the track. pub(crate) fn open(catalog: &AudioConfig, config: &Config) -> Result, Error> { - select(catalog, &config.kind, candidates(&config.kind, PLATFORM, SOFTWARE)) + if let Kind::Named(requested) = &config.kind + && codec_name(&catalog.codec) != Some(requested) + { + return Err(Error::Unsupported(format!( + "audio decoder {requested:?} is unavailable for {}", + catalog.codec + ))); + } + + select(catalog, candidates(&config.kind, PLATFORM, SOFTWARE)) } /// The candidates `kind` allows, in the order to try them. @@ -91,13 +112,13 @@ pub(crate) fn open(catalog: &AudioConfig, config: &Config) -> Result(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { match kind { - Kind::Auto => platform.iter().chain(software).collect(), Kind::Software => software.iter().collect(), - Kind::Named(name) => platform.iter().chain(software).filter(|c| c.name == name).collect(), + // `open` has already checked that a name matches the codec. + Kind::Auto | Kind::Named(_) => platform.iter().chain(software).collect(), } } -fn select(catalog: &AudioConfig, kind: &Kind, candidates: Vec<&Candidate>) -> Result, Error> { +fn select(catalog: &AudioConfig, candidates: Vec<&Candidate>) -> Result, Error> { let codec = &catalog.codec; let mut refused = Vec::new(); @@ -122,21 +143,7 @@ fn select(catalog: &AudioConfig, kind: &Kind, candidates: Vec<&Candidate>) -> Re return Err(Error::Unsupported(reasons.join(", "))); } - match kind { - Kind::Named(name) => { - let available: Vec<&str> = PLATFORM - .iter() - .chain(SOFTWARE) - .filter(|c| (c.supports)(codec)) - .map(|c| c.name) - .collect(); - Err(Error::Unsupported(format!( - "no audio decoder named {name:?} for {codec} (this build has: {})", - available.join(", ") - ))) - } - _ => Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))), - } + Err(Error::Unsupported(format!("unsupported audio codec: {codec}"))) } #[cfg(test)] @@ -204,7 +211,7 @@ mod tests { } fn pick(kind: Kind, platform: &[Candidate], software: &[Candidate]) -> Result { - let backend = select(&opus(), &kind, candidates(&kind, platform, software))?; + let backend = select(&opus(), candidates(&kind, platform, software))?; Ok(backend.name().to_owned()) } @@ -226,22 +233,11 @@ mod tests { assert_eq!(name, "software"); } + /// A codec name is not a backend selector: it tries the same tiers as `Auto`. #[test] - fn named_forces_one() { - let name = pick( - Kind::Named("software".into()), - &[PLATFORM_STUB], - &[PCM_ONLY, SOFTWARE_STUB], - ) - .unwrap(); - assert_eq!(name, "software"); - } - - /// A named backend that refuses the track is the answer: nothing else is tried. - #[test] - fn named_refusal_does_not_fall_back() { - let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); - assert!(err.to_string().contains("not this track"), "{err}"); + fn named_codec_picks_like_auto() { + let name = pick(Kind::Named("opus".into()), &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); } #[test] @@ -259,45 +255,33 @@ mod tests { ); } - /// An unknown name says what this build has for the codec instead. - #[test] - fn unknown_name_lists_the_alternatives() { - let err = open( - &opus(), - &Config { - kind: Kind::Named("opus".into()), - }, - ) - .err() - .expect("no backend is named after its codec"); - let message = err.to_string(); - assert!( - message.contains("\"opus\"") && message.contains(libopus::NAME), - "{message}" - ); + fn named(name: &str) -> Config { + Config { + kind: Kind::Named(name.into()), + } } - /// Asking for a real backend that does not decode the codec is refused, not - /// quietly swapped for one that does. + /// The codec names published moq-audio accepts still open their codec. #[test] - fn named_backend_for_another_codec_is_refused() { - let config = Config { - kind: Kind::Named(pcm::NAME.into()), - }; - assert!(matches!(open(&opus(), &config), Err(Error::Unsupported(_)))); + fn codec_names_open_their_codec() { + let pcm = AudioConfig::new(AudioCodec::Pcm, 48_000, 2); + assert_eq!(open(&pcm, &named("pcm")).unwrap().name(), pcm::NAME); + assert_eq!(open(&opus(), &named("opus")).unwrap().name(), libopus::NAME); } + /// A name for another codec is refused, not quietly swapped for one that decodes. #[test] - fn software_backends_open_by_name() { - let pcm = AudioConfig::new(AudioCodec::Pcm, 48_000, 2); - let config = Config { - kind: Kind::Named(pcm::NAME.into()), - }; - assert_eq!(open(&pcm, &config).unwrap().name(), pcm::NAME); + fn another_codecs_name_is_refused() { + assert!(matches!(open(&opus(), &named("pcm")), Err(Error::Unsupported(_)))); + } - let config = Config { - kind: Kind::Named(libopus::NAME.into()), - }; - assert_eq!(open(&opus(), &config).unwrap().name(), libopus::NAME); + /// Backend names are internal: asking for one is refused like any unknown name. + #[test] + fn backend_names_are_refused() { + let message = open(&opus(), &named(libopus::NAME)) + .err() + .expect("libopus is a backend, not a codec") + .to_string(); + assert!(message.contains("\"libopus\"") && message.contains("opus"), "{message}"); } } diff --git a/rs/moq-audio/src/decode/decoder.rs b/rs/moq-audio/src/decode/decoder.rs index 2a16214a072..ded1fb53dbb 100644 --- a/rs/moq-audio/src/decode/decoder.rs +++ b/rs/moq-audio/src/decode/decoder.rs @@ -17,8 +17,9 @@ pub enum Kind { Auto, /// Require a software backend. Software, - /// Require a backend by its stable lowercase name: `"libopus"`, `"pcm"`, or - /// `"symphonia"`. + /// Require the track to be this codec, by its lowercase name: `"opus"`, + /// `"pcm"`, or `"aac"`. Any other codec is refused. The library still picks + /// the backend, platform first. Named(String), } @@ -371,6 +372,24 @@ pub(crate) mod tests { )); } + /// Published moq-audio accepts the codec's name, not a backend's. + #[cfg(feature = "aac")] + #[test] + fn named_aac_decodes_aac() { + let config = Config { + kind: Kind::Named("aac".into()), + }; + assert!(Decoder::new(&aac_catalog(), &config).is_ok()); + + let config = Config { + kind: Kind::Named("opus".into()), + }; + assert!(matches!( + Decoder::new(&aac_catalog(), &config), + Err(Error::Unsupported(_)) + )); + } + #[test] fn refuses_unavailable_backend() { let catalog = hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Pcm, 48_000, 2); diff --git a/rs/moq-audio/src/encode/backend/mod.rs b/rs/moq-audio/src/encode/backend/mod.rs index 1879d49f37e..1e12ec9d933 100644 --- a/rs/moq-audio/src/encode/backend/mod.rs +++ b/rs/moq-audio/src/encode/backend/mod.rs @@ -8,7 +8,9 @@ //! read back from the backend. //! //! [`open`] tries the platform encoders before the software ones, skipping any -//! that does not emit the codec, and refuses when none opens. The software tier +//! that does not emit the codec, and refuses when none opens. [`Kind::Named`] +//! names the codec, not a backend, so which backend encodes it stays the +//! library's choice. The software tier //! is libopus for Opus and a passthrough for PCM. AAC has no software encoder, //! so it is only as available as the platform's, and no platform encoder is //! wired in yet. @@ -54,7 +56,8 @@ pub(crate) trait Backend: Send { /// rate: Opus lookahead, AAC encoder delay. fn delay(&self) -> usize; - /// The stable lowercase name [`Kind::Named`] selects this backend by. + /// The stable lowercase name of this backend, as [`Encoder::name`](super::Encoder::name) + /// reports it. Callers never select by it. fn name(&self) -> &str; } @@ -81,21 +84,30 @@ const SOFTWARE: &[Candidate] = &[ }, ]; -/// Test-only backends, in neither tier so `Auto` and `Software` never pick one: -/// they exist to be asked for by name. +/// Test-only backends, tried after every real one so a host with a real encoder +/// never opens one. They stand in for a codec this host has no encoder for. #[cfg(test)] -const NAMED_ONLY: &[Candidate] = &[Candidate { +const FALLBACK: &[Candidate] = &[Candidate { name: stub::NAME, codecs: &[Codec::Aac], open: stub::Stub::open, }]; -#[cfg(not(test))] -const NAMED_ONLY: &[Candidate] = &[]; - /// Open the first backend that emits the codec and accepts the settings. pub(crate) fn open(settings: &Settings) -> Result, Error> { - select(settings, candidates(&settings.kind, PLATFORM, SOFTWARE)) + if let Kind::Named(requested) = &settings.kind + && requested != settings.codec.as_str() + { + return Err(Error::Unsupported(format!( + "audio encoder {requested:?} is unavailable for {}", + settings.codec + ))); + } + + let tried = candidates(&settings.kind, PLATFORM, SOFTWARE); + #[cfg(test)] + let tried = [tried, FALLBACK.iter().collect()].concat(); + select(settings, tried) } /// The candidates `kind` allows, in the order to try them. @@ -104,14 +116,9 @@ pub(crate) fn open(settings: &Settings) -> Result, Error> { /// this host compiles in. fn candidates<'a>(kind: &Kind, platform: &'a [Candidate], software: &'a [Candidate]) -> Vec<&'a Candidate> { match kind { - Kind::Auto => platform.iter().chain(software).collect(), Kind::Software => software.iter().collect(), - Kind::Named(name) => platform - .iter() - .chain(software) - .chain(NAMED_ONLY) - .filter(|c| c.name == name) - .collect(), + // `open` has already checked that a name matches the codec. + Kind::Auto | Kind::Named(_) => platform.iter().chain(software).collect(), } } @@ -150,9 +157,8 @@ fn select(settings: &Settings, candidates: Vec<&Candidate>) -> Result available.join(", "), }; Err(Error::Unsupported(match &settings.kind { - Kind::Named(name) => format!("no audio encoder named {name:?} for {codec} (this build has: {available})"), Kind::Software => format!("no software {codec} audio encoder (this build has: {available})"), - Kind::Auto => format!("no {codec} audio encoder (this build has: {available})"), + Kind::Auto | Kind::Named(_) => format!("no {codec} audio encoder (this build has: {available})"), })) } @@ -245,22 +251,11 @@ mod tests { assert_eq!(name, "software"); } + /// A codec name is not a backend selector: it tries the same tiers as `Auto`. #[test] - fn named_forces_one() { - let name = pick( - Kind::Named("software".into()), - &[PLATFORM_STUB], - &[PCM_ONLY, SOFTWARE_STUB], - ) - .unwrap(); - assert_eq!(name, "software"); - } - - /// A named backend that refuses the settings is the answer: nothing else is tried. - #[test] - fn named_refusal_does_not_fall_back() { - let err = pick(Kind::Named("refusing".into()), &[REFUSING], &[SOFTWARE_STUB]).unwrap_err(); - assert!(err.to_string().contains("not these settings"), "{err}"); + fn named_codec_picks_like_auto() { + let name = pick(Kind::Named("aac".into()), &[PLATFORM_STUB], &[SOFTWARE_STUB]).unwrap(); + assert_eq!(name, "platform"); } #[test] @@ -283,41 +278,43 @@ mod tests { /// gates this to the hosts without one. #[test] fn aac_without_a_platform_encoder_is_refused() { - let err = open(&aac(Kind::Auto)).err().expect("no AAC encoder on this host"); + // The real tiers only: `open` adds the test stub. + let err = select(&aac(Kind::Auto), candidates(&Kind::Auto, PLATFORM, SOFTWARE)) + .err() + .expect("no AAC encoder on this host"); let message = err.to_string(); assert!(message.contains("aac") && message.contains("none"), "{message}"); } - /// An unknown name says what this build has for the codec instead. - #[test] - fn unknown_name_lists_the_alternatives() { - let settings = Settings { - kind: Kind::Named("opus".into()), + fn named(codec: Codec, name: &str) -> Settings { + Settings { + codec, + kind: Kind::Named(name.into()), ..Settings::default() - }; - let message = open(&settings) - .err() - .expect("no backend is named after its codec") - .to_string(); - assert!( - message.contains("\"opus\"") && message.contains(libopus::NAME), - "{message}" - ); + } } + /// A codec's own name opens that codec, like `Auto`. #[test] - fn software_backends_open_by_name() { - let settings = Settings { - kind: Kind::Named(libopus::NAME.into()), - ..Settings::default() - }; - assert_eq!(open(&settings).unwrap().name(), libopus::NAME); + fn codec_names_open_their_codec() { + assert_eq!(open(&named(Codec::Opus, "opus")).unwrap().name(), libopus::NAME); + assert_eq!(open(&named(Codec::Pcm, "pcm")).unwrap().name(), pcm::NAME); + assert_eq!(open(&named(Codec::Aac, "aac")).unwrap().name(), stub::NAME); + } - let settings = Settings { - codec: Codec::Pcm, - kind: Kind::Named(pcm::NAME.into()), - ..Settings::default() - }; - assert_eq!(open(&settings).unwrap().name(), pcm::NAME); + /// A name for another codec is refused, not quietly swapped for the settings' codec. + #[test] + fn another_codecs_name_is_refused() { + assert!(matches!(open(&named(Codec::Opus, "pcm")), Err(Error::Unsupported(_)))); + } + + /// Backend names are internal: asking for one is refused like any unknown name. + #[test] + fn backend_names_are_refused() { + let message = open(&named(Codec::Opus, libopus::NAME)) + .err() + .expect("libopus is a backend, not a codec") + .to_string(); + assert!(message.contains("\"libopus\"") && message.contains("opus"), "{message}"); } } diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index 5f908c8b686..9421c00db7d 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -80,7 +80,9 @@ pub enum Kind { Auto, /// Require a software backend. Software, - /// Require a backend by its stable lowercase name: `"libopus"` or `"pcm"`. + /// Require [`Settings::codec`] to be this codec, by its lowercase name: + /// `"opus"`, `"pcm"`, or `"aac"`. Any other codec is refused. The library + /// still picks the backend, platform first. Named(String), } @@ -839,12 +841,9 @@ mod tests { assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); } - /// AAC settings routed to the test stub, since this host has no AAC encoder. + /// AAC settings, which open the test stub since this host has no AAC encoder. fn aac(layout: Layout) -> Settings { - Settings { - kind: Kind::Named(backend::stub::NAME.into()), - ..Settings::from_input(Codec::Aac, &Input::new(48_000, layout)) - } + Settings::from_input(Codec::Aac, &Input::new(48_000, layout)) } /// The ASC is synthesized from the settings, so it exists before any packet. diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index afb0b737221..31670ade686 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -925,10 +925,7 @@ mod tests { let input = Input::new(48_000, Layout::Mono); let options = Options { track: Some("audio".to_string()), - settings: Settings { - kind: crate::encode::Kind::Named(crate::encode::backend::stub::NAME.into()), - ..Settings::from_input(crate::encode::Codec::Aac, &input) - }, + settings: Settings::from_input(crate::encode::Codec::Aac, &input), ..Options::default() }; let mut producer = Producer::new(&mut broadcast, catalog, input, &options).unwrap(); From a76fd04093b546fe43cf2938df5cf0c72787d449 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 29 Sep 2026 10:59:34 -0700 Subject: [PATCH 14/20] test(audio): install the AAC encoder stub per test The cfg(test) stub no longer rides along after every real candidate. A test opts in with stub::install(), which stands it in for the platform tier on that thread, so Software and uninstalled tests see the real tiers and refusals. Co-Authored-By: Claude Opus 5.5 --- rs/moq-audio/src/encode/backend/mod.rs | 34 +++++++++++++++++------- rs/moq-audio/src/encode/backend/stub.rs | 29 +++++++++++++++++++- rs/moq-audio/src/encode/encoder.rs | 35 ++++++++++++++++--------- rs/moq-audio/src/encode/producer.rs | 2 ++ 4 files changed, 77 insertions(+), 23 deletions(-) diff --git a/rs/moq-audio/src/encode/backend/mod.rs b/rs/moq-audio/src/encode/backend/mod.rs index 1e12ec9d933..ba277b8eba0 100644 --- a/rs/moq-audio/src/encode/backend/mod.rs +++ b/rs/moq-audio/src/encode/backend/mod.rs @@ -84,10 +84,10 @@ const SOFTWARE: &[Candidate] = &[ }, ]; -/// Test-only backends, tried after every real one so a host with a real encoder -/// never opens one. They stand in for a codec this host has no encoder for. +/// The platform tier on a test thread that called [`stub::install`], so AAC tests +/// open the same backend on every host and every other test sees the real tiers. #[cfg(test)] -const FALLBACK: &[Candidate] = &[Candidate { +const STUB_PLATFORM: &[Candidate] = &[Candidate { name: stub::NAME, codecs: &[Codec::Aac], open: stub::Stub::open, @@ -104,10 +104,15 @@ pub(crate) fn open(settings: &Settings) -> Result, Error> { ))); } - let tried = candidates(&settings.kind, PLATFORM, SOFTWARE); #[cfg(test)] - let tried = [tried, FALLBACK.iter().collect()].concat(); - select(settings, tried) + let platform = match stub::installed() { + true => STUB_PLATFORM, + false => PLATFORM, + }; + #[cfg(not(test))] + let platform = PLATFORM; + + select(settings, candidates(&settings.kind, platform, SOFTWARE)) } /// The candidates `kind` allows, in the order to try them. @@ -278,14 +283,22 @@ mod tests { /// gates this to the hosts without one. #[test] fn aac_without_a_platform_encoder_is_refused() { - // The real tiers only: `open` adds the test stub. - let err = select(&aac(Kind::Auto), candidates(&Kind::Auto, PLATFORM, SOFTWARE)) - .err() - .expect("no AAC encoder on this host"); + let err = open(&aac(Kind::Auto)).err().expect("no AAC encoder on this host"); let message = err.to_string(); assert!(message.contains("aac") && message.contains("none"), "{message}"); } + /// The stub stands in for the platform tier only, so `Software` still refuses + /// AAC with it installed. + #[test] + fn stub_is_not_software() { + let _stub = stub::install(); + assert_eq!(open(&aac(Kind::Auto)).unwrap().name(), stub::NAME); + + let err = open(&aac(Kind::Software)).err().expect("no software AAC encoder"); + assert!(err.to_string().contains("no software aac"), "{err}"); + } + fn named(codec: Codec, name: &str) -> Settings { Settings { codec, @@ -297,6 +310,7 @@ mod tests { /// A codec's own name opens that codec, like `Auto`. #[test] fn codec_names_open_their_codec() { + let _stub = stub::install(); assert_eq!(open(&named(Codec::Opus, "opus")).unwrap().name(), libopus::NAME); assert_eq!(open(&named(Codec::Pcm, "pcm")).unwrap().name(), pcm::NAME); assert_eq!(open(&named(Codec::Aac, "aac")).unwrap().name(), stub::NAME); diff --git a/rs/moq-audio/src/encode/backend/stub.rs b/rs/moq-audio/src/encode/backend/stub.rs index c581332c156..a64cf664689 100644 --- a/rs/moq-audio/src/encode/backend/stub.rs +++ b/rs/moq-audio/src/encode/backend/stub.rs @@ -1,5 +1,8 @@ //! A stand-in AAC encoder, so the AAC front end is testable on a host with no -//! platform encoder. Selectable only by name, and only in tests. +//! platform encoder. A test opts in with [`install`]; every other test sees the +//! real tiers. + +use std::cell::Cell; use bytes::Bytes; @@ -12,6 +15,30 @@ pub(crate) const NAME: &str = "stub"; /// The AudioToolbox AAC-LC encoder delay, which is what a real backend reports. pub(crate) const DELAY: usize = 2112; +thread_local! { + static INSTALLED: Cell = const { Cell::new(false) }; +} + +/// Stand the stub in for the platform tier on this thread until the guard drops. +pub(crate) fn install() -> Installed { + INSTALLED.set(true); + Installed(()) +} + +/// Whether this thread has the stub installed. +pub(super) fn installed() -> bool { + INSTALLED.get() +} + +/// Uninstalls the stub on drop. +pub(crate) struct Installed(()); + +impl Drop for Installed { + fn drop(&mut self) { + INSTALLED.set(false); + } +} + /// Emits each frame's index as its payload, and cannot retune. pub(crate) struct Stub { bitrate: u64, diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index 9421c00db7d..95872c644dc 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -841,15 +841,29 @@ mod tests { assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); } - /// AAC settings, which open the test stub since this host has no AAC encoder. fn aac(layout: Layout) -> Settings { Settings::from_input(Codec::Aac, &Input::new(48_000, layout)) } + /// Open `settings` on the test stub, since this host has no AAC encoder. + fn stub(settings: &Settings) -> Result { + let _stub = backend::stub::install(); + Encoder::new(settings) + } + + /// Without the stub, the public constructor sees the real tiers and refuses. + #[test] + fn aac_is_refused_without_an_encoder() { + let err = Encoder::new(&aac(Layout::Stereo)) + .err() + .expect("no AAC encoder on this host"); + assert!(matches!(err, Error::Unsupported(_)), "{err}"); + } + /// The ASC is synthesized from the settings, so it exists before any packet. #[test] fn aac_catalog_carries_the_synthesized_asc() { - let enc = Encoder::new(&aac(Layout::Stereo)).unwrap(); + let enc = stub(&aac(Layout::Stereo)).unwrap(); assert_eq!(enc.name(), backend::stub::NAME); assert_eq!(enc.frame_size(), 1024); @@ -874,7 +888,7 @@ mod tests { (Layout::FivePointOne, 6), (Layout::SevenPointOne, 7), ] { - let catalog = Encoder::new(&aac(layout)).unwrap().catalog(); + let catalog = stub(&aac(layout)).unwrap().catalog(); let description = catalog.description.unwrap(); assert_eq!(description[1] >> 3 & 0xF, config, "{layout:?}"); assert_eq!(catalog.channel_count, layout.channels(), "{layout:?}"); @@ -886,10 +900,7 @@ mod tests { Layout::SixPointOne, Layout::Discrete(2), ] { - assert!( - matches!(Encoder::new(&aac(layout)), Err(Error::Unsupported(_))), - "{layout:?}" - ); + assert!(matches!(stub(&aac(layout)), Err(Error::Unsupported(_))), "{layout:?}"); } } @@ -901,13 +912,13 @@ mod tests { frame_duration: Duration::from_micros(21_333), ..aac(Layout::Stereo) }; - assert_eq!(Encoder::new(&settings).unwrap().frame_size(), 1024); + assert_eq!(stub(&settings).unwrap().frame_size(), 1024); let settings = Settings { frame_duration: Duration::from_millis(20), ..aac(Layout::Stereo) }; - let err = Encoder::new(&settings).err().expect("20 ms is 960 samples"); + let err = stub(&settings).err().expect("20 ms is 960 samples"); assert!(err.to_string().contains("1024"), "{err}"); } @@ -917,13 +928,13 @@ mod tests { dtx: true, ..aac(Layout::Stereo) }; - assert!(matches!(Encoder::new(&settings), Err(Error::Unsupported(_)))); + assert!(matches!(stub(&settings), Err(Error::Unsupported(_)))); } /// A backend that can't retune keeps its opening rate. #[test] fn fixed_rate_backend_keeps_its_opening_rate() { - let mut enc = Encoder::new(&Settings { + let mut enc = stub(&Settings { bitrate: Some(moq_net::bandwidth::Rate::from_bps(96_000)), ..aac(Layout::Stereo) }) @@ -939,7 +950,7 @@ mod tests { /// The drain pushes the encoder delay out through whole silent frames. #[test] fn aac_finish_drains_the_encoder_delay() { - let mut enc = Encoder::new(&aac(Layout::Mono)).unwrap(); + let mut enc = stub(&aac(Layout::Mono)).unwrap(); assert_eq!(enc.folded_delay(), backend::stub::DELAY); enc.encode(&[0.0; 1024]).unwrap(); diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index 31670ade686..742f5636793 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -928,7 +928,9 @@ mod tests { settings: Settings::from_input(crate::encode::Codec::Aac, &input), ..Options::default() }; + let stub = crate::encode::backend::stub::install(); let mut producer = Producer::new(&mut broadcast, catalog, input, &options).unwrap(); + drop(stub); let track = consumer .track("audio") From c6675bd1fda19a348b1dedd3a4e069196c671e05 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 07:21:24 -0700 Subject: [PATCH 15/20] quest(audio-codecs): make the remaining children standalone AudioToolbox decode and encode need a macOS host, and two quests break published APIs on dev, which a main-bound line cannot. Move every remaining child to m1 so the line can land, and split the uniffi frame_duration_us default change into its own dev quest. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 9 +++++++ .../{audio-codecs => }/aac-encode-refusals.md | 0 .../aac-parse-truncated-sbr.md | 2 +- quest/m1/audio-codecs/README.md | 26 +++++++++---------- ...oolbox.md => audio-decode-audiotoolbox.md} | 0 ...oolbox.md => audio-encode-audiotoolbox.md} | 10 +++---- quest/m1/ffi-frame-duration-default.md | 22 ++++++++++++++++ .../{audio-codecs => }/gst-opus-surround.md | 0 .../he-aac-catalog-output.md | 0 .../ts-opus-channel-codes.md | 2 +- .../ts-opus-export-refusals.md | 2 +- 11 files changed, 50 insertions(+), 23 deletions(-) rename quest/m1/{audio-codecs => }/aac-encode-refusals.md (100%) rename quest/m1/{audio-codecs => }/aac-parse-truncated-sbr.md (84%) rename quest/m1/{audio-codecs/decode-audiotoolbox.md => audio-decode-audiotoolbox.md} (100%) rename quest/m1/{audio-codecs/encode-audiotoolbox.md => audio-encode-audiotoolbox.md} (72%) create mode 100644 quest/m1/ffi-frame-duration-default.md rename quest/m1/{audio-codecs => }/gst-opus-surround.md (100%) rename quest/m1/{audio-codecs => }/he-aac-catalog-output.md (100%) rename quest/m1/{audio-codecs => }/ts-opus-channel-codes.md (89%) rename quest/m1/{audio-codecs => }/ts-opus-export-refusals.md (92%) diff --git a/quest/m1/README.md b/quest/m1/README.md index be62e0315bf..b9da45b8ccf 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -90,6 +90,15 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [OBS native codecs](/quest/m1/obs-moq-video/README.md) - replace FFmpeg video and audio decoding with moq-video and moq-audio, deliver GPU frames, and use native audio/video encoders - [Opus concealment](/quest/m1/opus-conceal.md) - a lost Opus packet conceals the last packet's length, not 120 ms - [Audio codecs](/quest/m1/audio-codecs/README.md) - platform audio codecs, explicit unsupported cases, and channel layouts up to 7.1 +- [AudioToolbox decode](/quest/m1/audio-decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers +- [HE-AAC catalog output](/quest/m1/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core +- [TS Opus export refusals](/quest/m1/ts-opus-export-refusals.md) - the TS exporter refuses Opus heads its channel code cannot describe instead of mislabeling them +- [TS Opus channel codes](/quest/m1/ts-opus-channel-codes.md) - TS Opus with a channel code of 0x81 or above imports with its real head or is refused, never guessed as stereo +- [AudioToolbox encode](/quest/m1/audio-encode-audiotoolbox.md) - macOS and iOS encode AAC-LC +- [AAC encode refusals](/quest/m1/aac-encode-refusals.md) - on dev, `Config::encode` refuses channel counts it cannot name instead of writing stereo +- [AAC parse truncated SBR](/quest/m1/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core +- [GStreamer surround Opus](/quest/m1/gst-opus-surround.md) - the moq-gst sink publishes 3 to 8 channel Opus with the OpusHead its caps describe +- [FFI frame duration default](/quest/m1/ffi-frame-duration-default.md) - on dev, the binding audio encoder takes the codec's own frame by default, so `aac()` needs no explicit 0 - [Opus catalog rate](/quest/m1/opus-catalog-rate.md) - MKV Opus import publishes the 48 kHz codec rate in the catalog, not the OpusHead input rate - [mp4-atom dOps mapping](/quest/m1/mp4-atom-dops-mapping.md) - a released mp4-atom reads and writes any `dOps` channel mapping family and table - [CMAF surround Opus](/quest/m1/cmaf-opus-surround.md) - fMP4 import and export carry an Opus channel mapping table diff --git a/quest/m1/audio-codecs/aac-encode-refusals.md b/quest/m1/aac-encode-refusals.md similarity index 100% rename from quest/m1/audio-codecs/aac-encode-refusals.md rename to quest/m1/aac-encode-refusals.md diff --git a/quest/m1/audio-codecs/aac-parse-truncated-sbr.md b/quest/m1/aac-parse-truncated-sbr.md similarity index 84% rename from quest/m1/audio-codecs/aac-parse-truncated-sbr.md rename to quest/m1/aac-parse-truncated-sbr.md index 57f9bf04e69..3a6d94fc831 100644 --- a/quest/m1/audio-codecs/aac-parse-truncated-sbr.md +++ b/quest/m1/aac-parse-truncated-sbr.md @@ -18,4 +18,4 @@ Public API: none beyond a new error case. Wire: none. ## Required -- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - encode stops producing truncated SBR and PS configs +- [AAC encode refusals](/quest/m1/aac-encode-refusals.md) - encode stops producing truncated SBR and PS configs diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md index cc301af29ce..2442de5c235 100644 --- a/quest/m1/audio-codecs/README.md +++ b/quest/m1/audio-codecs/README.md @@ -1,4 +1,4 @@ -# Audio codecs +# [M] Audio codecs ## Goal @@ -37,23 +37,21 @@ Encode mirrors decode: an `encode::backend` seam, `Codec::Aac` meaning AAC-LC at the input's layout, and platform encoders behind it. Opus encode stays mono/stereo. -The core configuration and layout contracts land in main. These quests implement -surround and backend dispatch on that contract; each platform then lands as -its own decode and encode quest so verification stays per host. +The core configuration and layout contracts land in main. The line lands the +decode and encode seams, surround layouts, surround Opus, and named codecs. +Platform backends and the small refusal fixes found along the way are +standalone quests (split out on 2026-09-30), so the line does not wait on a +macOS host or on `dev`. -## Required - -- [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers -- [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core -- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the TS exporter refuses Opus heads its channel code cannot describe instead of mislabeling them -- [TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) - TS Opus with a channel code of 0x81 or above imports with its real head or is refused, never guessed as stereo -- [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC -- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - `Config::encode` refuses channel counts it cannot name instead of writing stereo -- [AAC parse truncated SBR](/quest/m1/audio-codecs/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core -- [GStreamer surround Opus](/quest/m1/audio-codecs/gst-opus-surround.md) - the moq-gst sink publishes 3 to 8 channel Opus with the OpusHead its caps describe +Remaining work: merge main in, port +[#4442](https://github.com/moq-dev/moq/pull/4442)'s one-frame Opus +concealment from `decode/decoder.rs` into `decode/backend/libopus.rs`, taking +the concealed duration from the multistream packet, and get CI green. ## Related +- [AudioToolbox decode](/quest/m1/audio-decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers +- [AudioToolbox encode](/quest/m1/audio-encode-audiotoolbox.md) - macOS and iOS encode AAC-LC - [OBS native codecs](/quest/m1/obs-moq-video/README.md) - the OBS source and encoder adapters consume this through moq-ffi; #3498 narrowed OBS to what moq-audio decodes today - [Runtime QA hosts](/quest/m2/runtime-qa-hosts.md) - Windows and Android verification needs a host; the Windows and macOS CI gates only compile - [Dart codec parity](/quest/m1/dart-codecs.md) - Dart gains these once it builds with the `audio` feature diff --git a/quest/m1/audio-codecs/decode-audiotoolbox.md b/quest/m1/audio-decode-audiotoolbox.md similarity index 100% rename from quest/m1/audio-codecs/decode-audiotoolbox.md rename to quest/m1/audio-decode-audiotoolbox.md diff --git a/quest/m1/audio-codecs/encode-audiotoolbox.md b/quest/m1/audio-encode-audiotoolbox.md similarity index 72% rename from quest/m1/audio-codecs/encode-audiotoolbox.md rename to quest/m1/audio-encode-audiotoolbox.md index 30569545924..dfd0ddf478c 100644 --- a/quest/m1/audio-codecs/encode-audiotoolbox.md +++ b/quest/m1/audio-encode-audiotoolbox.md @@ -23,12 +23,10 @@ the encode seam as the platform candidate on macOS and iOS. - Regression: a stereo and a 5.1 encode round-trip through the AudioToolbox decoder and through symphonia (stereo only), with timestamps continuous across the priming. -- Decided in [#4183](https://github.com/moq-dev/moq/pull/4183): the uniffi - `MoqAudioEncoderOutput::frame_duration_us` default moves from 20000 to 0 - (the codec's own frame) with this quest, so `aac()` works without an - explicit 0. Changing a published binding default is a break, so that - change targets `dev`. +- Until [FFI frame duration default](/quest/m1/ffi-frame-duration-default.md) + lands on `dev`, binding callers pass `frame_duration_us: 0` with `aac()`. + Split out on 2026-09-30 so this quest stays on `main`. ## Required -- [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - the round-trip regression decodes through it +- [AudioToolbox decode](/quest/m1/audio-decode-audiotoolbox.md) - the round-trip regression decodes through it diff --git a/quest/m1/ffi-frame-duration-default.md b/quest/m1/ffi-frame-duration-default.md new file mode 100644 index 00000000000..56496f8c79b --- /dev/null +++ b/quest/m1/ffi-frame-duration-default.md @@ -0,0 +1,22 @@ +# [XS] FFI audio encoder takes the codec's own frame by default + +## Goal + +On `dev`, `MoqAudioEncoderOutput::frame_duration_us` defaults to 0, the +codec's own frame, so `MoqAudioCodec::aac()` works without an explicit 0 in +every binding. Opus still encodes 20 ms frames by default. + +## Plan + +- Decided in [#4183](https://github.com/moq-dev/moq/pull/4183): the uniffi + default moves from 20000 to 0. Changing a published binding default is a + break, so this targets `dev`. Split from AudioToolbox encode on 2026-09-30, + since a child of a `main`-bound line cannot land on `dev`. +- `rs/moq-ffi/src/audio.rs` already treats 0 as the codec's frame; the + `default_frame_duration_matches_moq_audio` test that pins 20000 to + moq-audio's default goes away with the literal. +- Update the wrappers and docs that restate 20000: the Python test asserting + the default, `py/moq-rs/README.md`, and `doc/lib/{py,swift,kt,go,dart}`. + libmoq already reads 0 as the default. + +Public API: breaking default in moq-ffi and every binding. Wire: none. diff --git a/quest/m1/audio-codecs/gst-opus-surround.md b/quest/m1/gst-opus-surround.md similarity index 100% rename from quest/m1/audio-codecs/gst-opus-surround.md rename to quest/m1/gst-opus-surround.md diff --git a/quest/m1/audio-codecs/he-aac-catalog-output.md b/quest/m1/he-aac-catalog-output.md similarity index 100% rename from quest/m1/audio-codecs/he-aac-catalog-output.md rename to quest/m1/he-aac-catalog-output.md diff --git a/quest/m1/audio-codecs/ts-opus-channel-codes.md b/quest/m1/ts-opus-channel-codes.md similarity index 89% rename from quest/m1/audio-codecs/ts-opus-channel-codes.md rename to quest/m1/ts-opus-channel-codes.md index 2fbce343d24..221cc675592 100644 --- a/quest/m1/audio-codecs/ts-opus-channel-codes.md +++ b/quest/m1/ts-opus-channel-codes.md @@ -24,4 +24,4 @@ Public API: none. Wire: none. ## Related -- [TS Opus export refusals](/quest/m1/audio-codecs/ts-opus-export-refusals.md) - the exporter side of the same descriptor +- [TS Opus export refusals](/quest/m1/ts-opus-export-refusals.md) - the exporter side of the same descriptor diff --git a/quest/m1/audio-codecs/ts-opus-export-refusals.md b/quest/m1/ts-opus-export-refusals.md similarity index 92% rename from quest/m1/audio-codecs/ts-opus-export-refusals.md rename to quest/m1/ts-opus-export-refusals.md index c501cb032f5..b376124fbf3 100644 --- a/quest/m1/audio-codecs/ts-opus-export-refusals.md +++ b/quest/m1/ts-opus-export-refusals.md @@ -15,7 +15,7 @@ not written with a guessed `channel_config_code`. Decide from the parsed description: family 0, and family 1 with the Vorbis default mapping, keep the plain code; refuse the rest, or write the explicit layout if the import side of - [TS Opus channel codes](/quest/m1/audio-codecs/ts-opus-channel-codes.md) settles + [TS Opus channel codes](/quest/m1/ts-opus-channel-codes.md) settles one both sides agree on. - A track with no description stays mono or stereo only. - Tests with real heads, checked against ffprobe where ffmpeg reads the result. From de599eabb8f38a1fecae927157787b645e7ec4fc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 07:25:18 -0700 Subject: [PATCH 16/20] quest(audio-codecs): finish the line Delete the README, whose remaining work landed with the main merge, and point every reference at the standalone quests or the code on the base. Co-Authored-By: Claude Opus 5.5 --- ...bandwidth-grant-in-moq-audio-instead-of.md | 14 ++--- quest/m1/README.md | 1 - quest/m1/audio-codecs/README.md | 60 ------------------- quest/m1/cmaf-opus-surround.md | 1 - quest/m1/obs-moq-video/README.md | 3 +- quest/m1/opus-mapping-family.md | 8 +-- 6 files changed, 10 insertions(+), 77 deletions(-) delete mode 100644 quest/m1/audio-codecs/README.md diff --git a/quest/m1/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md b/quest/m1/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md index 946a68a0a6c..9aaee72e13b 100644 --- a/quest/m1/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md +++ b/quest/m1/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md @@ -33,20 +33,18 @@ grant; `Options::bandwidth` documents that their own loop; the capture driver's `_reservation` goes away. Public entry points for a manual ceiling stay `Producer::set_bitrate` and `Encoder::set_bitrate`. -- The audio-codecs line branch moves the Opus encoder behind a backend seam: - the rate setter and its floor live in - `rs/moq-audio/src/encode/backend/libopus.rs` there, and - `Encoder::set_bitrate` dispatches through the backend. Target whichever - shape is on the base when this starts. -- Floor: `set_opus_bitrate` refuses anything outside +- The Opus encoder sits behind the encode backend seam: the rate setter and + its floor live in `rs/moq-audio/src/encode/backend/libopus.rs`, and + `Encoder::set_bitrate` dispatches through the backend. +- Floor: the libopus backend's `set_bitrate` refuses anything outside `opus::bitrate_floor(codec_rate, frame_size).max(500)` to - `300_000 * channels` (`encoder.rs`, `rs/moq-audio/src/opus.rs`). `Policy::min` defaults to a tenth of + `300_000 * channels` (`backend/libopus.rs`, `rs/moq-audio/src/opus.rs`). `Policy::min` defaults to a tenth of the ceiling (`rs/moq-mux/src/rate.rs`); for Opus it is the codec floor, so a grant below it clamps there and never errors. The reservation's ceiling is the configured bitrate; only the policy target moves. - PCM: `pcm::bitrate(sample_rate, channels)` is `pub(crate)` (`rs/moq-audio/src/pcm.rs`), `Settings::bitrate` is refused for it - and so is `set_bitrate` (`encoder.rs`). A PCM + and so is `set_bitrate` (`encoder.rs`, `backend/pcm.rs`). A PCM Producer reserves its fixed rate and runs no policy. That is the same reserve-only usage passthrough imports use, so nothing new is added for it. - Priority is unchanged: `PRIORITY` puts audio at 80 and video at 60 diff --git a/quest/m1/README.md b/quest/m1/README.md index 58ca91593cf..4c120ae9c18 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -72,7 +72,6 @@ transport, benchmark tooling); worktrees isolate commits, not semantics. - [C++ through moq-ffi](/quest/m1/cpp/README.md) - generated C++ over moq-ffi with futures and expected-style errors, shipped as a tarball, vcpkg, and Conan, and adopted by the OBS plugin - [Generated C bindings](/quest/m1/c/README.md) - C generated from moq-ffi ships as `moq-c` 0.8.0 and replaces the hand-written libmoq - [OBS native codecs](/quest/m1/obs-moq-video/README.md) - replace FFmpeg video and audio decoding with moq-video and moq-audio, deliver GPU frames, and use native audio/video encoders -- [Audio codecs](/quest/m1/audio-codecs/README.md) - platform audio codecs, explicit unsupported cases, and channel layouts up to 7.1 - [AudioToolbox decode](/quest/m1/audio-decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [HE-AAC catalog output](/quest/m1/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core - [TS Opus export refusals](/quest/m1/ts-opus-export-refusals.md) - the TS exporter refuses Opus heads its channel code cannot describe instead of mislabeling them diff --git a/quest/m1/audio-codecs/README.md b/quest/m1/audio-codecs/README.md deleted file mode 100644 index 2879ff54598..00000000000 --- a/quest/m1/audio-codecs/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# [M] Audio codecs - -## Goal - -A broadcast that plays in the browser plays natively. Today the browser -decodes whatever the platform offers (HE-AAC, 5.1 AAC, AC-3, surround Opus) -while `moq-audio` decodes Opus and AAC-LC in mono or stereo, so the same -stream plays on moq.dev and fails in OBS, `moq play`, and every binding. The -native side grows the backend seam `moq-video` already has, uses the OS codec -where one exists, keeps symphonia as the pure-Rust fallback, and carries more -than two channels end to end. Encoding follows the same seam so a native -publisher can produce AAC. - -## Plan - -Platform first, exactly like video: AudioToolbox on macOS and iOS, Media -Foundation on Windows, MediaCodec on Android, and symphonia (AAC-LC -mono/stereo) as the software fallback that openh264 is for H.264. Linux has no -OS audio decoder, so multichannel AAC stays refused there, stated in the docs -and rejected at construction. HE-AAC signaled only in band (implicit SBR, as -over MPEG-TS) plays as its half-rate LC core on symphonia; detecting it needs a -full element walk, so the docs state it instead of refusing it. A platform backend claims every catalog -codec its framework opens, so AC-3, E-AC-3, MP3, and FLAC ride along on the -hosts that have them; each still needs a fixture before the backend advertises -it. - -Use the extensible `Layout` contract settled in main, with well-known layouts in one -canonical order, derived from the codec description (AAC channelConfiguration, -OpusHead mapping) so the catalog and wire do not change. Every decoder reorders -from its codec's native order into that one, so the mixer, the FFI, and OBS -never guess where the LFE is. Playback downmixes to whatever the output device -opened. Preserve the existing arbitrary-channel PCM passthrough through an -unspecified discrete layout; refuse spatial remixing when speaker positions -are unknown instead of guessing them. - -Encode mirrors decode: an `encode::backend` seam, `Codec::Aac` meaning AAC-LC -at the input's layout, and platform encoders behind it. Opus encode stays -mono/stereo. - -The core configuration and layout contracts land in main. The line lands the -decode and encode seams, surround layouts, surround Opus, and named codecs. -Platform backends and the small refusal fixes found along the way are -standalone quests (split out on 2026-09-30), so the line does not wait on a -macOS host or on `dev`. - -Remaining work: keep main merged in and get CI green. The one-frame Opus -concealment from [#4442](https://github.com/moq-dev/moq/pull/4442) lives in -`decode/backend/libopus.rs` on this line. - -## Related - -- [AudioToolbox decode](/quest/m1/audio-decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers -- [AudioToolbox encode](/quest/m1/audio-encode-audiotoolbox.md) - macOS and iOS encode AAC-LC -- [OBS native codecs](/quest/m1/obs-moq-video/README.md) - the OBS source and encoder adapters consume this through moq-ffi; #3498 narrowed OBS to what moq-audio decodes today -- [Runtime QA hosts](/quest/m2/runtime-qa-hosts.md) - Windows and Android verification needs a host; the Windows and macOS CI gates only compile -- [Dart codec parity](/quest/m1/dart-codecs.md) - Dart gains these once it builds with the `audio` feature -- [Media Foundation decode](/quest/m2/audio-decode-mediafoundation.md) - Windows decodes HE-AAC, multichannel AAC, and what else the MFTs offer -- [Media Foundation encode](/quest/m2/audio-encode-mediafoundation.md) - Windows encodes AAC-LC -- [MediaCodec decode](/quest/m2/audio-decode-mediacodec.md) - Android decodes HE-AAC, multichannel AAC, and what else the device offers -- [MediaCodec encode](/quest/m2/audio-encode-mediacodec.md) - Android encodes AAC-LC diff --git a/quest/m1/cmaf-opus-surround.md b/quest/m1/cmaf-opus-surround.md index da3ae24b669..ab6b86e93c6 100644 --- a/quest/m1/cmaf-opus-surround.md +++ b/quest/m1/cmaf-opus-surround.md @@ -19,4 +19,3 @@ its table intact. ## Required - [mp4-atom dOps mapping](/quest/m1/mp4-atom-dops-mapping.md) - `Dops` carries the mapping family and table -- [Audio codecs](/quest/m1/audio-codecs/README.md) - `opus::Mapping` and a `Config::encode` that writes any family's table diff --git a/quest/m1/obs-moq-video/README.md b/quest/m1/obs-moq-video/README.md index 1e41ad84c8f..b2b3478da4d 100644 --- a/quest/m1/obs-moq-video/README.md +++ b/quest/m1/obs-moq-video/README.md @@ -38,4 +38,5 @@ The quests separate portable decoding, platform GPU delivery, audio, and publish - [Linux GPU input](/quest/m2/obs-linux-gpu.md) - allocation-export feasibility and its dependent implementation are deferred - [VAAPI encode and decode](/quest/m4/video-vaapi.md) - owns Linux backend decode/import capabilities; reconcile its older dependency assumptions against current code - [Video hardware validation](/quest/m3/video-hardware.md) - physical hardware evidence is required for each claimed GPU path -- [Audio codecs](/quest/m1/audio-codecs/README.md) - HE-AAC, multichannel, and native AAC encode reach the OBS source and encoder adapters through moq-ffi +- [AudioToolbox decode](/quest/m1/audio-decode-audiotoolbox.md) - HE-AAC and multichannel AAC reach the OBS source on macOS through moq-ffi +- [AudioToolbox encode](/quest/m1/audio-encode-audiotoolbox.md) - native AAC encode reaches the OBS encoder adapter on macOS through moq-ffi diff --git a/quest/m1/opus-mapping-family.md b/quest/m1/opus-mapping-family.md index f2dec437cd4..9bd243a1ddf 100644 --- a/quest/m1/opus-mapping-family.md +++ b/quest/m1/opus-mapping-family.md @@ -2,14 +2,10 @@ ## Goal -On dev, `moq_mux::codec::opus::Config` stores the channel mapping family in one place: the audio-codecs line's `mapping: Option`. The `mapping_family: u8` field that #4130 shipped in moq-mux 0.10.8 is removed, and every reader takes the family from `mapping`. +On dev, `moq_mux::codec::opus::Config` stores the channel mapping family in one place: `mapping: Option`. The `mapping_family: u8` field that #4130 shipped in moq-mux 0.10.8 is removed, and every reader takes the family from `mapping`. ## Plan -Decided 2026-09-28 while merging main into the audio-codecs line: that sync keeps both fields so the line stays additive on main, and `encode` refuses a `mapping_family` that disagrees with `mapping`. Removing the duplicate breaks the published moq-mux API, so it lands on dev once the line has merged. Update main's fMP4 and MSF paths that read `mapping_family`, plus the TS importer, which sets both fields. +Decided 2026-09-28 while merging main into the audio-codecs line: that sync keeps both fields so the line stays additive on main, and `encode` refuses a `mapping_family` that disagrees with `mapping`. Removing the duplicate breaks the published moq-mux API, so it lands on dev. Update main's fMP4 and MSF paths that read `mapping_family`, plus the TS importer, which sets both fields. Public API: breaking in moq-mux (field removed). Wire: none. - -## Required - -- [Audio codecs](/quest/m1/audio-codecs/README.md) - the line that introduces `mapping` From 94404331705da7a21329e9f7225130ed0c29fbf5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 07:38:46 -0700 Subject: [PATCH 17/20] fix(audio): keep Decoder and Encoder unwind safe Boxing a dyn Backend dropped UnwindSafe and RefUnwindSafe from both published types, which cargo-semver-checks reports as a major break. Require them of every backend. Co-Authored-By: Claude Opus 5.5 --- rs/moq-audio/src/decode/backend/mod.rs | 4 +++- rs/moq-audio/src/decode/decoder.rs | 7 +++++++ rs/moq-audio/src/encode/backend/mod.rs | 4 +++- rs/moq-audio/src/encode/encoder.rs | 7 +++++++ 4 files changed, 20 insertions(+), 2 deletions(-) diff --git a/rs/moq-audio/src/decode/backend/mod.rs b/rs/moq-audio/src/decode/backend/mod.rs index 6b25cbab5d8..45d76b3e6f2 100644 --- a/rs/moq-audio/src/decode/backend/mod.rs +++ b/rs/moq-audio/src/decode/backend/mod.rs @@ -23,7 +23,9 @@ mod pcm; mod symphonia; /// An opened decoder: packets in, interleaved `f32` PCM out. -pub(crate) trait Backend: Send { +/// +/// Unwind safe because the published `Decoder` that boxes it is. +pub(crate) trait Backend: Send + std::panic::UnwindSafe + std::panic::RefUnwindSafe { /// Decode one packet into interleaved samples at [`sample_rate`](Self::sample_rate) /// and [`layout`](Self::layout), untrimmed. fn decode(&mut self, packet: &[u8]) -> Result; diff --git a/rs/moq-audio/src/decode/decoder.rs b/rs/moq-audio/src/decode/decoder.rs index b8db58bb14a..44154aad4f0 100644 --- a/rs/moq-audio/src/decode/decoder.rs +++ b/rs/moq-audio/src/decode/decoder.rs @@ -159,6 +159,13 @@ pub(crate) mod tests { } #[cfg(feature = "aac")] + /// The published `Decoder` was unwind safe before it boxed a backend; keep it so. + #[test] + fn decoder_is_unwind_safe() { + fn assert_unwind_safe() {} + assert_unwind_safe::(); + } + #[test] fn aac_decodes_a_sine() { let mut decoder = Decoder::new(&aac_catalog(), &Config::default()).unwrap(); diff --git a/rs/moq-audio/src/encode/backend/mod.rs b/rs/moq-audio/src/encode/backend/mod.rs index ba277b8eba0..97f4dbdc025 100644 --- a/rs/moq-audio/src/encode/backend/mod.rs +++ b/rs/moq-audio/src/encode/backend/mod.rs @@ -34,7 +34,9 @@ pub(crate) mod stub; /// counting frames. A codec that pipelines output (MediaCodec) needs a `flush` /// and a zero-or-more return, which changes `Encoder::encode` too, so that lands /// with the first backend that needs it rather than as an always-empty method. -pub(crate) trait Backend: Send { +/// +/// Unwind safe because the published `Encoder` that boxes it is. +pub(crate) trait Backend: Send + std::panic::UnwindSafe + std::panic::RefUnwindSafe { /// Encode exactly one frame of the codec's frame size. fn encode(&mut self, pcm: &[f32]) -> Result; diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index 95872c644dc..5eefb000113 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -569,6 +569,13 @@ mod tests { out } + /// The published `Encoder` was unwind safe before it boxed a backend; keep it so. + #[test] + fn encoder_is_unwind_safe() { + fn assert_unwind_safe() {} + assert_unwind_safe::(); + } + #[test] fn opus_encode_then_decode_keeps_signal_close() { let mut enc = Encoder::new(&Settings { From 7ba1f06f3f77dca235349505ff74fd2618e07537 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 08:36:10 -0700 Subject: [PATCH 18/20] fix(audio): keep the AAC sine test behind the aac feature Co-Authored-By: Claude Opus 5.5 --- rs/moq-audio/src/decode/decoder.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-audio/src/decode/decoder.rs b/rs/moq-audio/src/decode/decoder.rs index 44154aad4f0..7b600582fd6 100644 --- a/rs/moq-audio/src/decode/decoder.rs +++ b/rs/moq-audio/src/decode/decoder.rs @@ -158,7 +158,6 @@ pub(crate) mod tests { catalog } - #[cfg(feature = "aac")] /// The published `Decoder` was unwind safe before it boxed a backend; keep it so. #[test] fn decoder_is_unwind_safe() { @@ -166,6 +165,7 @@ pub(crate) mod tests { assert_unwind_safe::(); } + #[cfg(feature = "aac")] #[test] fn aac_decodes_a_sine() { let mut decoder = Decoder::new(&aac_catalog(), &Config::default()).unwrap(); From d0fbb3672f279d4b4c8f5b66558a90770dd4bab9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 08:38:03 -0700 Subject: [PATCH 19/20] fix(audio): refuse PCM layouts its channel count cannot name A PCM catalog carries only a count, so 3.0 and 4.0 decoded as 2.1 and quad, putting the center in the LFE. Co-Authored-By: Claude Opus 5.5 --- rs/moq-audio/src/encode/backend/pcm.rs | 11 ++++++++++- rs/moq-audio/src/encode/encoder.rs | 25 +++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/rs/moq-audio/src/encode/backend/pcm.rs b/rs/moq-audio/src/encode/backend/pcm.rs index 04ff1f16e24..62a2382a536 100644 --- a/rs/moq-audio/src/encode/backend/pcm.rs +++ b/rs/moq-audio/src/encode/backend/pcm.rs @@ -2,7 +2,7 @@ use super::Backend; use crate::encode::{Encoded, Settings}; -use crate::{Error, pcm}; +use crate::{Error, Layout, pcm}; pub(super) const NAME: &str = "pcm"; @@ -11,7 +11,16 @@ pub(super) struct Pcm { } impl Pcm { + /// The catalog carries only a channel count, so a named layout that count + /// would decode as another one is refused rather than relabeled. pub(super) fn open(settings: &Settings) -> Result, Error> { + let layout = settings.layout; + if !matches!(layout, Layout::Discrete(_)) && Layout::from_channels(layout.channels())? != layout { + return Err(Error::Unsupported(format!( + "pcm carries only a channel count, which decodes as {:?}, not {layout:?}", + Layout::from_channels(layout.channels())? + ))); + } let bitrate = pcm::bitrate(settings.sample_rate, settings.layout.channels())?; Ok(Box::new(Self { bitrate })) } diff --git a/rs/moq-audio/src/encode/encoder.rs b/rs/moq-audio/src/encode/encoder.rs index 5eefb000113..5ba2e3b0747 100644 --- a/rs/moq-audio/src/encode/encoder.rs +++ b/rs/moq-audio/src/encode/encoder.rs @@ -767,6 +767,31 @@ mod tests { assert_eq!(output.samples, input); } + /// A PCM catalog names only a count, and 3 or 4 channels decode as 2.1 or + /// quad, so 3.0 and 4.0 are refused; discrete channels still pass through. + #[test] + fn pcm_refuses_layouts_its_count_cannot_name() { + for layout in [Layout::ThreePointZero, Layout::FourPointZero] { + let err = Encoder::new(&Settings { + codec: Codec::Pcm, + ..Settings::new(48_000, layout) + }); + assert!(matches!(err, Err(Error::Unsupported(_))), "{layout:?}"); + } + + for layout in [Layout::TwoPointOne, Layout::Quad, Layout::Discrete(3)] { + let mut enc = Encoder::new(&Settings { + codec: Codec::Pcm, + ..Settings::new(48_000, layout) + }) + .unwrap(); + let mut dec = Decoder::new(&enc.catalog(), &DecodeConfig::default()).unwrap(); + let input = sine(440.0, enc.codec_rate(), enc.codec_channels(), enc.frame_size()); + let output = dec.decode(&enc.encode(&input).unwrap().payload).unwrap(); + assert_eq!(output.samples, input, "{layout:?}"); + } + } + #[test] fn pcm_catalog_declares_fixed_bitrate() { let enc = Encoder::new(&Settings { From 8ec0c927490d54ca659f33e91af64e3865031631 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 30 Sep 2026 08:45:29 -0700 Subject: [PATCH 20/20] quest: follow-ups from the audio-codecs line review Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/audio-encode-audiotoolbox.md | 5 +++++ quest/m1/ts-aac-pce-join.md | 31 +++++++++++++++++++++++++++ 3 files changed, 37 insertions(+) create mode 100644 quest/m1/ts-aac-pce-join.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 882d79a9190..045669116eb 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -77,6 +77,7 @@ QUIC studies there on that rule. - [AAC encode refusals](/quest/m1/aac-encode-refusals.md) - on dev, `Config::encode` refuses channel counts it cannot name instead of writing stereo - [AAC parse truncated SBR](/quest/m1/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core - [GStreamer surround Opus](/quest/m1/gst-opus-surround.md) - the moq-gst sink publishes 3 to 8 channel Opus with the OpusHead its caps describe +- [TS AAC PCE joins](/quest/m1/ts-aac-pce-join.md) - PCE-described AAC over TS plays after a mid-stream join or resume, and a missing PCE silences only its track - [FFI frame duration default](/quest/m1/ffi-frame-duration-default.md) - on dev, the binding audio encoder takes the codec's own frame by default, so `aac()` needs no explicit 0 - [Opus mapping family](/quest/m1/opus-mapping-family.md) - on dev, the Opus head config keeps its mapping family only in `mapping` - [mp4-atom dOps mapping](/quest/m1/mp4-atom-dops-mapping.md) - a released mp4-atom reads and writes any `dOps` channel mapping family and table diff --git a/quest/m1/audio-encode-audiotoolbox.md b/quest/m1/audio-encode-audiotoolbox.md index dfd0ddf478c..6892e275b16 100644 --- a/quest/m1/audio-encode-audiotoolbox.md +++ b/quest/m1/audio-encode-audiotoolbox.md @@ -23,6 +23,11 @@ the encode seam as the platform candidate on macOS and iOS. - Regression: a stereo and a 5.1 encode round-trip through the AudioToolbox decoder and through symphonia (stereo only), with timestamps continuous across the priming. +- That continuity fails today for an epoch below the delay: `Producer::timestamp` + (`rs/moq-audio/src/encode/producer.rs`) clamps each priming packet to zero, + so a zero epoch stamps `[0, 0, 0, 20_000]`, and `decode::Consumer` reads the + duplicates as holes and resets AAC between them. Shift the whole timeline by + the deficit instead, including the terminal end (found in #4081 review). - Until [FFI frame duration default](/quest/m1/ffi-frame-duration-default.md) lands on `dev`, binding callers pass `frame_duration_us: 0` with `aac()`. Split out on 2026-09-30 so this quest stays on `main`. diff --git a/quest/m1/ts-aac-pce-join.md b/quest/m1/ts-aac-pce-join.md new file mode 100644 index 00000000000..e31f2e1ff99 --- /dev/null +++ b/quest/m1/ts-aac-pce-join.md @@ -0,0 +1,31 @@ +# [S] TS AAC program config survives joins and resumes + +## Goal + +AAC whose layout lives in a program config element (ADTS `channel_config` 0) +plays over MPEG-TS for a receiver that joins mid-stream or after a resume. A +missing PCE silences that one track instead of failing the whole program. + +## Plan + +Found in review of the audio-codecs line (#4081), 2026-09-30. + +- Import: `aac::in_band_config` returns `ProgramConfigMissing` when the first + frame seen has no leading PCE, and `?` ends the whole TS import, video + included (`rs/moq-mux/src/container/ts/import.rs`). Instead, keep the + reservation, drop frames on that track until one leads with a PCE, and log + once. ffmpeg's ADTS muxer writes the PCE only in the first frame, so a late + join of an ffmpeg source stays silent; say so in the docs. +- Export: `mux` takes the PCE on the first frame only + (`rs/moq-mux/src/container/ts/export.rs`), so `resume()`/`rewind()` and any + mid-stream receiver never see it. Repeat it, either on every frame or with + each PAT/PMT, and pick whichever ffmpeg and the import side accept. +- Regression: an export resumed before its first span re-emits the PCE, and an + import that starts on a PCE-less frame publishes the track once a PCE + arrives, while the other tracks keep flowing. + +Public API: none. Wire: none; TS output gains repeated PCEs. + +## Related + +- [TS Opus channel codes](/quest/m1/ts-opus-channel-codes.md) - the same per-track refusal principle for Opus descriptors