Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,11 @@ discarding the old mux buffer. The first new clock packet signals the break and
stdout pacing re-anchors. Every rendition joins the new program generation;
no track is fenced across the marker.

MPEG-TS export frames AAC as ADTS, which labels only the AAC Main, LC, SSR,
and LTP profiles. HE-AAC and HE-AACv2 go out as their AAC-LC core, and decoders
find the SBR and PS in band, as ffmpeg's ADTS output does. A track whose
profile or channel layout ADTS cannot label is refused rather than mislabeled.

A constant-rate MPEG-TS source records its multiplex rate in the catalog
(`mpegts.muxRate`, measured off the PCR clock, null stuffing included), and
`export ts` pads its output with null packets back to that rate so an IRD or
Expand Down
4 changes: 3 additions & 1 deletion quest/m1/audio-codecs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,11 @@ its own decode and encode quest so verification stays per host.
## Quests

- [AudioToolbox decode](/quest/m1/audio-codecs/decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers
- [ADTS refusals](/quest/m1/audio-codecs/adts-refusals.md) - the ADTS writer refuses channel counts and object types it cannot label instead of mislabeling them
- [HE-AAC catalog output](/quest/m1/audio-codecs/he-aac-catalog-output.md) - HE-AAC catalog entries name the output rate and layout, not the LC core
- [AudioToolbox encode](/quest/m1/audio-codecs/encode-audiotoolbox.md) - macOS and iOS encode AAC-LC
- [TS surround Opus](/quest/m1/audio-codecs/ts-opus-surround.md) - a 3 to 8 channel Opus stream from MPEG-TS imports with a family 1 OpusHead that decodes
- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - `Config::encode` refuses channel counts it cannot name instead of writing stereo
- [AAC parse truncated SBR](/quest/m1/audio-codecs/aac-parse-truncated-sbr.md) - `Config::parse` refuses SBR and PS configs cut off before their core

## Related

Expand Down
24 changes: 24 additions & 0 deletions quest/m1/audio-codecs/aac-encode-refusals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# [S] AAC config encode refuses unnameable channel counts

## Goal

`moq_mux::codec::aac::Config::encode` never writes an AudioSpecificConfig that
names the wrong layout. A channel count no channelConfiguration names is
refused, not logged and written as stereo, the way the ADTS writer refuses it.

## Plan

- `encode` returns `Bytes` today, so refusing is a published API break: this
retargets to `dev`. Decide whether it returns a `Result` or whether a
constructor validates the count up front so encoding cannot fail.
- A count a program config element could describe (7, or more than 8) is
still refused unless the caller supplies a layout; guessing speaker
positions is what this removes.
- Callers in `moq-audio` (encoder config, description synthesis) already
validate first; check they keep their error messages.

Public API: `Config::encode` (or its constructor) changes. Wire: none.

## Related

- [#4283](https://github.com/moq-dev/moq/pull/4283) - the ADTS export refusals this mirrors
21 changes: 21 additions & 0 deletions quest/m1/audio-codecs/aac-parse-truncated-sbr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# [XS] AAC config parse refuses truncated SBR and PS

## Goal

`Config::parse` refuses an AudioSpecificConfig that signals SBR or PS
(object type 5 or 29) but ends before its extension rate and core object type,
instead of accepting it as if the core were known.

## Plan

- The export path already requires these fields; `parse` stays lenient only
because `Config::encode` writes a two-byte config for those object types.
Once encode refuses or completes them, the leniency has no producer.
- Check the `moq-audio` tests that feed a two-byte HE-AAC config; they should
keep asserting a refusal, just a different one.

Public API: none beyond a new error case. Wire: none.

## Required

- [AAC encode refusals](/quest/m1/audio-codecs/aac-encode-refusals.md) - encode stops producing truncated SBR and PS configs
25 changes: 0 additions & 25 deletions quest/m1/audio-codecs/adts-refusals.md

This file was deleted.

29 changes: 29 additions & 0 deletions quest/m1/audio-codecs/he-aac-catalog-output.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# [S] HE-AAC catalog entries describe the output

## Goal

A catalog entry for HE-AAC or HE-AACv2 names what the stream plays as, for
example 48 kHz stereo, not its AAC-LC core (24 kHz, and mono for v2). Players
and gateways that size buffers or pick a decoder from the catalog see the real
output.

## Plan

- `Config::parse` reports the core rate and channel count because those lead
an explicit SBR or PS AudioSpecificConfig. The FLV and MKV importers copy
them into the catalog; fMP4 takes the sample entry instead, so the importers
disagree for the same stream.
- Derive the output from the config: the extension rate under SBR, two
channels under PS. Implicit SBR, found only in band, cannot be known from the
config; keep the documented half-rate behavior for it.
- Consumers that rebuild a config from catalog fields (description synthesis in
`moq-audio`, the MSF path) must not then mistake the output rate for the core.
- Regression: the `fdkaacenc` HE-AAC and HE-AACv2 FLV fixtures under
`rs/moq-mux/src/container/ts/test_data` import as 48 kHz stereo, matching
ffprobe.

Public API: possibly `Config` fields. Wire: catalog values change for HE-AAC.

## Related

- [#4283](https://github.com/moq-dev/moq/pull/4283) - found while labeling HE-AAC for ADTS
168 changes: 117 additions & 51 deletions rs/moq-mux/src/codec/aac/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,11 @@ impl Config {
let mut reader = BitReader::new(buf);
let (object_type, sample_rate, channel_config) = read_header(&mut reader)?;
let channel_count = match channel_config {
0 => program_config(&mut reader)?,
0 => {
let core = read_core(&mut reader, object_type)?;
read_general_audio(&mut reader, core)?;
program_config(&mut reader)?
}
_ => channel_count_from_config(channel_config)?,
};

Expand Down Expand Up @@ -135,9 +139,8 @@ impl Config {
}
}

/// Read an AudioSpecificConfig up to its channels: the audioObjectType, sample rate, and
/// channelConfiguration. For a channelConfiguration of 0 the reader stops at the program config
/// element.
/// Read an AudioSpecificConfig's leading fields: the audioObjectType, sample rate, and
/// channelConfiguration.
fn read_header<T: Buf>(reader: &mut BitReader<T>) -> Result<(u8, u32, u8)> {
if reader.buf.remaining() < 2 {
return Err(Error::ConfigTooShort);
Expand All @@ -156,35 +159,40 @@ fn read_header<T: Buf>(reader: &mut BitReader<T>) -> Result<(u8, u32, u8)> {

// channelConfiguration: 4 bits, immediately after the (possibly explicit) rate.
let channel_config = reader.read(4, Error::IncompleteConfig)? as u8;
if channel_config != 0 {
return Ok((object_type, sample_rate, channel_config));
}
Ok((object_type, sample_rate, channel_config))
}

// Explicit SBR and PS name their core object type after an extension rate; the
// GASpecificConfig carrying the program config element follows that core type.
let mut core = object_type;
if matches!(object_type, 5 | 29) {
if reader.read(4, Error::IncompleteConfig)? == 15 {
reader.read(24, Error::IncompleteConfig)?;
}
core = read_object_type(reader)?;
if core == 22 {
// extensionChannelConfiguration, only for ER BSAC.
reader.read(4, Error::IncompleteConfig)?;
}
/// Read the core audioObjectType, which explicit SBR and PS name after an extension rate. Any
/// other object type is its own core, and the leading sample rate is always the core's.
fn read_core<T: Buf>(reader: &mut BitReader<T>, object_type: u8) -> Result<u8> {
if !matches!(object_type, 5 | 29) {
return Ok(object_type);
}
if reader.read(4, Error::IncompleteConfig)? == 15 {
reader.read(24, Error::IncompleteConfig)?;
}
let core = read_object_type(reader)?;
if core == 22 {
// extensionChannelConfiguration, only for ER BSAC.
reader.read(4, Error::IncompleteConfig)?;
}
Ok(core)
}

/// Read a GASpecificConfig up to the program config element that a channelConfiguration of 0
/// puts next, refusing a `core` object type whose specific config is something else.
fn read_general_audio<T: Buf>(reader: &mut BitReader<T>, core: u8) -> Result<()> {
if !GENERAL_AUDIO.contains(&core) {
return Err(Error::ProgramConfigUnsupported(core));
}

// GASpecificConfig: frameLengthFlag, dependsOnCoreCoder (then a 14-bit
// coreCoderDelay), and extensionFlag precede the element.
// frameLengthFlag, dependsOnCoreCoder (then a 14-bit coreCoderDelay), and extensionFlag.
reader.read(1, Error::IncompleteConfig)?;
if reader.read(1, Error::IncompleteConfig)? == 1 {
reader.read(14, Error::IncompleteConfig)?;
}
reader.read(1, Error::IncompleteConfig)?;
Ok((object_type, sample_rate, 0))
Ok(())
}

/// Build the AudioSpecificConfig for a stream that signals its fields per frame, as ADTS does.
Expand Down Expand Up @@ -222,25 +230,46 @@ pub(crate) fn in_band_config(profile: u8, sample_rate: u32, channel_config: u8,
Ok(Bytes::from(out.bytes))
}

/// Split an AudioSpecificConfig into the channel signaling of a stream that carries it per frame,
/// as ADTS does; the inverse of [`in_band_config`].
///
/// Returns the channelConfiguration and, when that is 0, the program config element framed to
/// lead a raw data block: its element ID, then the element aligned to the block.
pub(crate) fn in_band_channels(asc: &[u8]) -> Result<(u8, Option<Bytes>)> {
/// The fields a stream that signals its config per frame, as ADTS does, carries in each header.
#[derive(Clone, Debug, PartialEq)]
pub(crate) struct InBand {
/// The core audioObjectType. Explicit SBR and PS name it under their own, and a decoder finds
/// them again in band, as it would in a stream that never signaled them (implicit signaling).
pub object_type: u8,
/// The core's sample rate, which SBR doubles on output.
pub sample_rate: u32,
pub channel_config: u8,
/// When `channel_config` is 0, the program config element framed to lead a raw data block:
/// its element ID, then the element aligned to the block.
pub program_config: Option<Bytes>,
}

/// Split an AudioSpecificConfig into what a stream that carries it per frame signals, as ADTS
/// does; the inverse of [`in_band_config`].
pub(crate) fn in_band(asc: &[u8]) -> Result<InBand> {
let mut asc = asc;
let mut reader = BitReader::new(&mut asc);
let (_, _, channel_config) = read_header(&mut reader)?;
if channel_config != 0 {
return Ok((channel_config, None));
}
let (object_type, sample_rate, channel_config) = read_header(&mut reader)?;
let core = read_core(&mut reader, object_type)?;

let program_config = match channel_config {
0 => {
read_general_audio(&mut reader, core)?;
let mut out = BitWriter::default();
out.write(3, ID_PCE);
reader.record = Some(out);
program_config(&mut reader)?;
Some(Bytes::from(reader.record.take().expect("recording set above").bytes))
}
_ => None,
};

let mut out = BitWriter::default();
out.write(3, ID_PCE);
reader.record = Some(out);
program_config(&mut reader)?;
let out = reader.record.take().expect("recording set above");
Ok((0, Some(Bytes::from(out.bytes))))
Ok(InBand {
object_type: core,
sample_rate,
channel_config,
program_config,
})
}

/// The raw data block element ID of a program config element (ISO 14496-3 Table 4.85).
Expand Down Expand Up @@ -554,10 +583,9 @@ mod tests {
assert_eq!(Config::parse(&mut asc.as_slice()).unwrap().channel_count, 7);
}

#[test]
fn parses_program_config_element_behind_explicit_sbr() {
// audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension
// rate, then the core type (LC) whose GASpecificConfig carries the element.
/// audioObjectType 5 (SBR), 24 kHz core, channelConfiguration 0, a 48 kHz extension rate,
/// then the core type (LC) whose GASpecificConfig carries a front and back pair.
fn sbr_pce_asc() -> Vec<u8> {
let mut out = BitWriter::default();
out.write(5, 5);
out.write(4, 6);
Expand All @@ -566,7 +594,12 @@ mod tests {
out.write(5, 2);
out.write(3, 0);
write_pce(&mut out, &[true], &[], &[true], 0);
let cfg = Config::parse(&mut out.bytes.as_slice()).unwrap();
out.bytes
}

#[test]
fn parses_program_config_element_behind_explicit_sbr() {
let cfg = Config::parse(&mut sbr_pce_asc().as_slice()).unwrap();
assert_eq!(cfg.profile, 5);
assert_eq!(cfg.channel_count, 4);
}
Expand Down Expand Up @@ -638,26 +671,59 @@ mod tests {
assert_eq!(rest, [0xFF], "the element moves out of the block");

// And back: the element leads the block again, byte for byte.
assert_eq!(in_band_channels(&asc).unwrap(), (0, Some(Bytes::from(pce))));
let in_band = in_band(&asc).unwrap();
assert_eq!(in_band.channel_config, 0);
assert_eq!(in_band.program_config, Some(Bytes::from(pce)));
}

#[test]
fn in_band_channels_of_ffmpeg_program_config_element() {
fn in_band_of_ffmpeg_program_config_element() {
// Round trip ffmpeg's own element through a raw data block and back into a config. The
// trailing SBR sync extension is not part of the element, so it drops.
let (config, pce) = in_band_channels(&FFMPEG_QUAD_ASC).unwrap();
assert_eq!(config, 0);
let pce = pce.unwrap();
let in_band = in_band(&FFMPEG_QUAD_ASC).unwrap();
assert_eq!(in_band.channel_config, 0);
let pce = in_band.program_config.unwrap();
let mut block = pce.as_ref();
let asc = in_band_config(2, 48_000, 0, &mut block).unwrap();
assert!(block.is_empty(), "the element is all that was framed");
assert_eq!(asc, FFMPEG_QUAD_ASC[..FFMPEG_QUAD_ASC.len() - 3]);
}

#[test]
fn in_band_channels_without_a_program_config_element() {
assert_eq!(in_band_channels(&[0x11, 0x90]).unwrap(), (2, None));
assert_eq!(in_band_channels(&[0x11, 0xE0]).unwrap(), (12, None));
fn in_band_without_a_program_config_element() {
let stereo = in_band(&[0x11, 0x90]).unwrap();
assert_eq!((stereo.channel_config, stereo.program_config), (2, None));
assert_eq!(in_band(&[0x11, 0xE0]).unwrap().channel_config, 12);
}

#[test]
fn in_band_of_explicit_sbr_is_its_lc_core() {
// GStreamer 1.28 `fdkaacenc` at 48 kHz stereo: HE-AAC is SBR over a 24 kHz stereo LC core,
// and HE-AACv2 is PS over a mono one.
for (asc, channel_config) in [([0x2B, 0x11, 0x88, 0x00], 2), ([0xEB, 0x09, 0x88, 0x00], 1)] {
let expected = InBand {
object_type: 2,
sample_rate: 24_000,
channel_config,
program_config: None,
};
assert_eq!(in_band(&asc).unwrap(), expected);
}

// A config naming SBR but stopping before its core has no core to name.
assert!(matches!(in_band(&[0x2A, 0x10]), Err(Error::IncompleteConfig)));
}

#[test]
fn in_band_of_explicit_sbr_keeps_its_program_config_element() {
let in_band = in_band(&sbr_pce_asc()).unwrap();
assert_eq!(in_band.object_type, 2);
assert_eq!(in_band.sample_rate, 24_000);
assert_eq!(in_band.channel_config, 0);
let pce = in_band.program_config.expect("a program config element");
let mut block = pce.as_ref();
let asc = in_band_config(2, 24_000, 0, &mut block).unwrap();
assert_eq!(Config::parse(&mut asc.as_ref()).unwrap().channel_count, 4);
}

#[test]
Expand Down
Loading
Loading