Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions c/moq/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

How to update your code for each breaking change, newest first.

## Unreleased

### Media setup refusals are typed

Audio and video subscribe setup now reports `MOQ_ERROR_UNSUPPORTED_CONFIG` when the decoder refuses a codec, layout, or description. Handle this variant to try another rendition instead of inspecting an audio/video error message or `Unsupported`. Decoder faults, such as a backend that fails to open, and later peer and decode failures keep their existing errors. Regenerate bindings and rebuild the matching native library together.

## moq-c 0.7, the hand-written ABI

The hand-written crate never published as moq-c: a libmoq 0.6 user applies these entries before moving to the generated API.
Expand Down
4 changes: 4 additions & 0 deletions cpp/moq/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Unreleased

### Media setup refusals are typed

Audio and video subscribe setup now reports `moq::Error::kUnsupportedConfig` when the decoder refuses a codec, layout, or description. Handle this variant to try another rendition instead of inspecting an audio/video error message or `Unsupported`. Decoder faults, such as a backend that fails to open, and later peer and decode failures keep their existing errors. Regenerate bindings and rebuild the matching native library together.

### JSON writes take a broadcast-clock timestamp ([#7691](https://github.com/moq-dev/moq/pull/7691))

`moq::JsonSnapshotProducer::update`, `moq::JsonStreamProducer::append`, and `moq::JsonWindowProducer::push` and `pop` take a required `moq::Timestamp` instead of an optional one. Stamp it from `moq::MediaCatalogProducer::now()`, the broadcast clock every track shares. Their constructors refuse a track created without a `timescale`. Regenerate bindings and rebuild with the matching native library.
Expand Down
58 changes: 31 additions & 27 deletions cpp/obs/src/moq-source.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ struct AudioLayout {
uint32_t sample_rate = 0;
};

// Own the ranked snapshot while setup asynchronously tries each rendition once.
struct AudioSelection {
std::vector<std::pair<std::string, moq::Audio>> candidates;
size_t next = 0;
};

// One subscribed track: what it is, the consumer once subscribed, and the pending
// call that subscribes or reads its next frame. Dropping it cancels that call.
struct Track {
Expand All @@ -140,6 +146,7 @@ struct Track {
// Video reads encoded frames to decode here; audio reads PCM moq-audio decoded.
std::shared_ptr<moq::MediaTrackConsumer> consumer;
std::shared_ptr<moq::AudioConsumer> audio_consumer;
std::shared_ptr<AudioSelection> audio_selection;
std::optional<moq::Continuation> call;
};

Expand Down Expand Up @@ -229,6 +236,8 @@ static void moq_source_subscribe_video(struct moq_source *ctx, const std::shared
static void moq_source_clear_audio_locked(struct moq_source *ctx);
static void moq_source_subscribe_audio(struct moq_source *ctx, const std::shared_ptr<Connection> &conn,
const moq::Catalog &catalog);
static void moq_source_try_audio(struct moq_source *ctx, const std::shared_ptr<Connection> &conn,
const std::shared_ptr<AudioSelection> &selection);
static Sent moq_source_send_packet(VideoDecoder &decoder, const moq::MediaFrame &frame_data);
static Decoded moq_source_receive_frame(VideoDecoder &decoder);

Expand Down Expand Up @@ -865,32 +874,16 @@ static void moq_source_on_audio_subscribed(struct moq_source *ctx, const std::sh
if (!result) {
LOG_ERROR("Failed to subscribe to Audio track: %s", result.error().to_string().c_str());
moq_source_discard_candidate(conn, track);
if (std::holds_alternative<moq::Error::kUnsupportedConfig>(result.error().get_variant()))
moq_source_try_audio(ctx, conn, track->audio_selection);
return;
}
track->audio_selection.reset();
track->audio_consumer = *result;
LOG_INFO("Subscribed to Audio track successfully");
moq_source_read_audio(ctx, conn, track);
}

// Whether moq-audio decodes the rendition: Opus, PCM, or mono or stereo AAC-LC (the one AAC
// profile and the layouts it has). moq-ffi reports a refusal only once the subscribe fails,
// as an audio error with no kind, so check here to let an unsupported rendition fall through.
// A configuration moq-audio still refuses, such as a malformed OpusHead, discards the
// candidate when it subscribes.
static bool moq_source_audio_decodes(const moq::Audio &config)
{
const auto &codec = config.codec;
if (codec == "opus" || codec == "pcm")
return true;
// hang reads the AAC profile as a decimal, so `mp4a.40.02` is AAC-LC too.
const std::string aac = "mp4a.40.";
if (codec.rfind(aac, 0) != 0)
return false;
const auto profile = codec.find_first_not_of('0', aac.size());
return profile != std::string::npos && codec.compare(profile, std::string::npos, "2") == 0 &&
(config.channel_count == 1 || config.channel_count == 2);
}

// The channels moq-audio should remix a rendition to, and where OBS puts them. A count
// names moq-audio's default layout by the WAVE convention: 3 is 2.1, 4 quad, 5 5.0, 6 5.1,
// 7 6.1, 8 7.1. OBS has no quad, 5.0, or 6.1 (its 4.0 and 4.1 put a center where quad has
Expand Down Expand Up @@ -1029,22 +1022,32 @@ static void moq_source_subscribe_audio(struct moq_source *ctx, const std::shared
return;
}

for (const auto *rendition : candidates) {
auto selection = std::make_shared<AudioSelection>();
selection->candidates.reserve(candidates.size());
for (const auto *rendition : candidates)
selection->candidates.emplace_back(rendition->first, rendition->second);
moq_source_try_audio(ctx, conn, selection);
}

static void moq_source_try_audio(struct moq_source *ctx, const std::shared_ptr<Connection> &conn,
const std::shared_ptr<AudioSelection> &selection)
{
while (selection->next < selection->candidates.size()) {
const auto *rendition = &selection->candidates[selection->next++];
const auto &id = rendition->first;
const auto &config = rendition->second;
if (moq_source_same_audio(conn->audio.get(), id, config) && conn->audio->instance == conn->instance) {
conn->pending_audio.reset();
return;
}
if (moq_source_same_audio(conn->pending_audio.get(), id, config))
if (moq_source_same_audio(conn->pending_audio.get(), id, config)) {
// Keep the in-flight setup, but fall through against the newest catalog if it refuses.
if (!conn->pending_audio->audio_consumer)
conn->pending_audio->audio_selection = selection;
return;

// A codec moq-audio can't decode, or a layout OBS can't place, falls through to the next rendition.
if (!moq_source_audio_decodes(config)) {
LOG_ERROR("Unsupported audio codec '%s' with %u channels in rendition %s", config.codec.c_str(),
config.channel_count, id.c_str());
continue;
}

// OBS's speaker layout is independent of the decoder's capabilities.
auto layout = moq_source_audio_layout(config.channel_count);
if (!layout) {
LOG_ERROR("Unsupported audio channel count %u in rendition %s: OBS has no layout for it",
Expand All @@ -1063,6 +1066,7 @@ static void moq_source_subscribe_audio(struct moq_source *ctx, const std::shared
track->instance = conn->instance;
track->audio_config = config;
track->audio_layout = *layout;
track->audio_selection = selection;
conn->pending_audio = track;

moq::AudioDecoderOutput output{};
Expand Down
40 changes: 22 additions & 18 deletions cpp/obs/test/moq-source-test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1205,23 +1205,6 @@ int main()
CHECK((names(moq_source_candidates(moq_source_ranked_audio(audios), held)) ==
std::vector<std::string>{"c-low"}));
CHECK(moq_source_candidates(moq_source_ranked_audio(audios), none).empty());

// The codecs moq-audio decodes, with AAC-LC in either spelling hang parses.
const auto decodes = [&](const char *codec, uint32_t channels) {
auto config = base_audio;
config.codec = codec;
config.channel_count = channels;
return moq_source_audio_decodes(config);
};
for (const char *codec : {"opus", "pcm", "mp4a.40.2", "mp4a.40.02"})
CHECK(decodes(codec, 2));
for (const char *codec : {"flac", "mp4a.40.5", "mp4a.40.20", "mp4a.40.", "mp4a.40.0"})
CHECK(!decodes(codec, 2));
// moq-audio decodes AAC only in mono or stereo, unlike Opus and PCM.
CHECK(decodes("mp4a.40.2", 1));
CHECK(!decodes("mp4a.40.2", 6));
CHECK(decodes("opus", 6));
CHECK(decodes("pcm", 6));
}
report("renditions rank like hang");

Expand Down Expand Up @@ -1273,6 +1256,23 @@ int main()
he_aac_audio.description.reset();
he_aac_audio.bitrate = 5500000;
catalog.audio.emplace("he-aac", he_aac_audio);
// A recognized codec with a malformed description is also a setup refusal.
auto malformed_audio = audio;
malformed_audio.description = std::vector<uint8_t>{};
malformed_audio.bitrate = 5600000;
catalog.audio.emplace("malformed", malformed_audio);
// AAC-LC by name, but its description declares SBR, which moq-audio refuses.
auto sbr_audio = audio;
sbr_audio.codec = "mp4a.40.2";
sbr_audio.description = std::vector<uint8_t>{0x12, 0x10, 0x56, 0xe5, 0x98};
sbr_audio.bitrate = 5400000;
catalog.audio.emplace("sbr", sbr_audio);
// Two bytes can still be malformed: this ASC promises a missing explicit sample rate.
auto truncated_aac_audio = audio;
truncated_aac_audio.codec = "mp4a.40.2";
truncated_aac_audio.description = std::vector<uint8_t>{0x17, 0x90};
truncated_aac_audio.bitrate = 5450000;
catalog.audio.emplace("truncated-aac", truncated_aac_audio);
// 5.1 AAC-LC: OBS has the layout, but moq-audio decodes AAC only in mono or stereo.
auto surround_aac_audio = audio;
surround_aac_audio.codec = "mp4a.40.2";
Expand All @@ -1295,11 +1295,15 @@ int main()
moq_source_subscribe_video(ctx, conn, catalog);
moq_source_subscribe_audio(ctx, conn, catalog);
CHECK(conn->pending_video && conn->pending_video->id == "c");
CHECK(conn->pending_audio && conn->pending_audio->id == "c");
// Selection continues asynchronously when setup refuses a rendition.
// The playing tracks stay until the replacement delivers.
CHECK(conn->video && conn->video->id == video_id);
CHECK(conn->audio && conn->audio->id == audio_id);
}
CHECK(publisher.WriteUntil([&] {
std::lock_guard<std::mutex> lock(ctx->mutex);
return ctx->connection->audio && ctx->connection->audio->id == "c";
}));
destroySource(source);
}
report("disabled and undecodable renditions fall through");
Expand Down
4 changes: 4 additions & 0 deletions dart/moq/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ How to update your code for each breaking change, newest first.

## Unreleased

### Media setup refusals are typed

Audio and video subscribe setup now reports `UnsupportedConfigMoqException` when the decoder refuses a codec, layout, or description. Handle this variant to try another rendition instead of inspecting an audio/video error message or `Unsupported`. Decoder faults, such as a backend that fails to open, and later peer and decode failures keep their existing errors. Regenerate bindings and rebuild the matching native library together.

### JSON writes take a broadcast-clock timestamp ([#7691](https://github.com/moq-dev/moq/pull/7691))

`json.SnapshotProducer.update`, `json.StreamProducer.append`, and `json.WindowProducer.push` / `pop` take a `required MoqTimestamp timestamp` on the broadcast clock instead of an optional one. Read it with `MoqMediaCatalogProducer(broadcast: broadcast).now()`. Their track needs a timescale, such as `MoqTrackInfo(timescale: 1000)`; one without is refused.
Expand Down
4 changes: 4 additions & 0 deletions dart/moq_ffi/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Unreleased

### Media setup refusals are typed

Audio and video subscribe setup now reports `UnsupportedConfigMoqException` when the decoder refuses a codec, layout, or description. Handle this variant to try another rendition instead of inspecting an audio/video error message or `Unsupported`. Decoder faults, such as a backend that fails to open, and later peer and decode failures keep their existing errors. Regenerate bindings and rebuild the matching native library together.

### Follow the core API migration guide

These bindings are generated from moq-ffi. Apply every change in the
Expand Down
48 changes: 46 additions & 2 deletions dart/moq_ffi/lib/src/moq.dart
Original file line number Diff line number Diff line change
Expand Up @@ -5765,6 +5765,12 @@ class FfiConverterMoqException {
lifted.value,
lifted.bytesRead - subview.offsetInBytes + 4,
);
case 34:
final lifted = UnsupportedConfigMoqException.read(subview);
return LiftRetVal<MoqException>(
lifted.value,
lifted.bytesRead - subview.offsetInBytes + 4,
);
default:
throw UniffiInternalError(
UniffiInternalError.unexpectedEnumCase,
Expand Down Expand Up @@ -6930,6 +6936,44 @@ class UnsupportedCatalogMoqException extends MoqException {
}
}

class UnsupportedConfigMoqException extends MoqException {
final String reason;
UnsupportedConfigMoqException(String this.reason);
UnsupportedConfigMoqException._(String this.reason);
static LiftRetVal<UnsupportedConfigMoqException> read(Uint8List buf) {
int new_offset = buf.offsetInBytes;
final reason_lifted = FfiConverterString.read(
Uint8List.view(buf.buffer, new_offset),
);
final reason = reason_lifted.value;
new_offset += reason_lifted.bytesRead;
return LiftRetVal(UnsupportedConfigMoqException._(reason), new_offset);
}

@override
RustBuffer lower() {
final buf = Uint8List(allocationSize());
write(buf);
return toRustBuffer(buf);
}

@override
int allocationSize() {
return FfiConverterString.allocationSize(reason) + 4;
}

@override
int write(Uint8List buf) {
buf.buffer.asByteData(buf.offsetInBytes).setInt32(0, 34);
int new_offset = buf.offsetInBytes + 4;
new_offset += FfiConverterString.write(
reason,
Uint8List.view(buf.buffer, new_offset),
);
return new_offset;
}
}

class MoqExceptionErrorHandler extends UniffiRustCallStatusErrorHandler {
@override
Exception lift(RustBuffer errorBuf) {
Expand Down Expand Up @@ -22449,7 +22493,7 @@ void _checkApiChecksums() {
throw UniffiInternalError.panicked("UniFFI API checksum mismatch");
}
if (uniffi_moq_ffi_checksum_constructor_moqaudioconsumer_subscribe() !=
10630) {
42847) {
throw UniffiInternalError.panicked("UniFFI API checksum mismatch");
}
if (uniffi_moq_ffi_checksum_constructor_moqaudioencoder_new() != 52616) {
Expand Down Expand Up @@ -22535,7 +22579,7 @@ void _checkApiChecksums() {
throw UniffiInternalError.panicked("UniFFI API checksum mismatch");
}
if (uniffi_moq_ffi_checksum_constructor_moqvideoconsumer_subscribe() !=
15507) {
20518) {
throw UniffiInternalError.panicked("UniFFI API checksum mismatch");
}
if (uniffi_moq_ffi_checksum_constructor_moqvideoencoder_new() != 22531) {
Expand Down
2 changes: 1 addition & 1 deletion doc/bin/obs.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ reconnects keep the same instance. A
stop/start, or an OBS auto-reconnect after the client gives up, republishes
the broadcast under a new instance, which viewers see as a new stream.

MoQ Source plays the best enabled audio and video rendition it can decode and place. It follows catalog changes while keeping the current audio and video playing until a replacement produces decoded output. A failed replacement leaves the current rendition playing. Disabling every rendition of a media kind keeps the one already playing; removing the media kind from the catalog clears it. It also follows a restarted or replaced publisher, including a sibling broadcast the catalog points at, playing the new one from its first keyframe.
MoQ Source plays the best enabled audio and video rendition it can decode and place. It follows catalog changes while keeping the current audio and video playing until a replacement produces decoded output. A decoder setup refusal tries the next ranked rendition, including when a supported codec name carries a description the decoder cannot use. A later peer or decode failure leaves the current rendition playing. Disabling every rendition of a media kind keeps the one already playing; removing the media kind from the catalog clears it. It also follows a restarted or replaced publisher, including a sibling broadcast the catalog points at, playing the new one from its first keyframe.

The source plays Opus, AAC-LC (mono or stereo), and PCM audio. A channel count names its speakers by the WAVE convention: 1, 2, 3 (2.1), 6 (5.1), and 8 (7.1) play on the matching OBS layout, while quad and 5.0 are upmixed to 5.1 and 6.1 to 7.1, which OBS lacks. Other counts are refused.

Expand Down
Loading
Loading