Skip to content

feat(dart): ship codecs and expose audio and video namespaces - #5839

Merged
kixelated merged 8 commits into
mainfrom
quest/m2/dart-codecs
Oct 17, 2026
Merged

kixelated merged 8 commits into
mainfrom
quest/m2/dart-codecs

Conversation

@kixelated

@kixelated kixelated commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Dart was the only binding that couldn't originate media. Its libraries were built --no-default-features, so they had no MoqAudioProducer, no MoqVideoProducer, and no encoder or decoder. The codec namespaces (#5694) have landed, so the Dart wrapper can now expose them in their final shape in one go.

Approach

  • Build Dart with moq-ffi's default features everywhere: sh/dart/generate.sh, the Native Assets hook's from-source build, and release-dart-ffi.yml. Bindings are regenerated.
  • Dart was the only caller of release-ffi.yml's default-features input, so this PR removes the input.
  • New package:moq/video.dart and package:moq/audio.dart libraries mirror Kotlin's dev.moq.video / dev.moq.audio. They are typedefs over the generated types (Producer, Encoder, Consumer, EncoderInput, EncoderOutput, DecoderOutput, Frame, ...) plus Duration views (timestamp, maxDelay, frameDuration). audio also aliases Preset and exposes the playout delay from feat(ffi): configure and observe audio playout delay in every binding #5844 as Duration views (DecoderOutput.delay, Consumer.delay). Video also gets DecodedFrame, Surface, and PixelBuffer, plus the variant constructors AutoEncoder() / HardwareEncoder() / SoftwareEncoder() / NamedEncoder(name) and Keyframe(interval).
  • Tests: decoded video frame owns its picture software-encodes H.264, subscribes through video.Consumer, and checks the I420 and RGBA pixel sizes plus RGBA alpha, matching the Kotlin test. A delay test checks that a fixed playout delay reads back unchanged and an estimated one moves as frames arrive. Another test checks the Opus audio.Producer, and the Duration test now covers the codec records. CI runs them through just dart check (check.yml's dispatcher selects dart for rs/moq-ffi/ and dart/ changes).
  • Docs: doc/lib/dart drops the "No codecs" caveat and adds encode and decode samples, which the doc-sample check compiles. doc/lib/index.md drops the Dart exception. Package READMEs and .github/ffi.md are updated too.
  • The quest file is deleted.

Cost

Numbers come from Nightly run 38897397428, which builds the same targets both ways: minimal Dart, and default-feature shared builds for Go, Kotlin and Swift. CI runners, not this host.

Shipped shared library, uncompressed:

Target Minimal Default Delta
aarch64-apple-ios (dylib) 26.6 MB 29.0 MB +2.3 MB (+9%)
aarch64-linux-android (.so) 29.4 MB 32.1 MB +2.6 MB (+9%)
x86_64-unknown-linux-gnu (.so) 31.4 MB 34.9 MB +3.5 MB (+11%)

Per-target release build job: +0 to +3 minutes. For example, Windows went from 21 to 24, Android armv7 from 7 to 9, and iOS from 12 to 14. Across Dart's 11 targets that is roughly +10 to 15 runner-minutes per nightly or tag. Locally, the extra codec compile in just dart check was about 6 minutes cold on a loaded host and is cached afterwards.

Deployment floors are unchanged (Android API 24, iOS 16).

Impact

  • Dart public API (additive): new libraries package:moq/video.dart and package:moq/audio.dart, listed above. package:moq_ffi now exports every MoqAudio* / MoqVideo* codec type, including MoqVideoConsumer and MoqVideoDecodedFrame.
  • Artifacts: the moq_ffi native assets now carry the audio and video codecs, about 9 to 11% larger.
  • CI: release-ffi.yml loses its default-features input.
  • Wire: none.

Decisions

  • Variant constructor names: the maintainer settled on (a), Go-style aliases as shipped: video.AutoEncoder() / HardwareEncoder() / SoftwareEncoder() / NamedEncoder(name), video.Keyframe(interval), and video.PixelBuffer. These mirror Go and media.dart's Named / Requested. The rejected options were Kotlin-style constants and leaving the variants unaliased.

Alternatives

  • Fold Dart into release-shared-ffi.yml now that its libraries match the shared default-feature ones. That would save about 9 duplicate native builds per run, but it changes the tag-triggered release path, and the shared set would need x86_64-apple-darwin and x86_64-apple-ios added. Left as a follow-up.
  • Add a frames() Stream helper over video.Consumer / audio.Consumer, as Kotlin does. Skipped because Dart has no Stream helper for media.TrackConsumer either. Better added for all consumers at once.

Follow-ups

  • Kotlin, Swift, and Python don't alias MoqAudioPreset (Go and now Dart do), and all of them call audio frames "interleaved" even with the planar formats.
  • moq's moq_ffi: ^0.4.3 floor is stale: no published moq_ffi has the media or codec types. Raising it is now a step in /quest/m2/release/dart-publish.md, to land with the release cut.
  • Dart joins the shared FFI coordinator (reuse-native), dropping its duplicate default-feature builds.
  • rs/moq-ffi/build.sh --no-default-features has no CI caller left. Drop it, or keep it for local minimal builds.
  • sh/size has no Dart history keys, so this size jump isn't tracked by Shipping sizes.

(written by Claude Opus 5.5)

🤖 Generated with Claude Code

@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: the quest is implemented. Local just check and all CI checks pass. The Dart Linux/Android/Windows release builds carry codecs, and the Check job ran the new Dart tests (decoded video frame owns its picture and audio producer encodes opus) on an aarch64 Linux runner.

This PR stays a draft for one open decision: the variant constructor names (video.SoftwareEncoder(), video.Keyframe(n), video.PixelBuffer). I recommend keeping them as shipped, since they mirror Go and media.dart's Named / Requested.

The size and build-time figures come from CI runners (Nightly 38897397428), not the loaded local host.

(written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 17, 2026 06:00
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 50 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6d78d79f-493b-4937-841a-ab4512e19c95
📥 Commits

Reviewing files that changed from the base of the PR and between 332cad7 and 6091baf.

📒 Files selected for processing: 1
  • dart/moq_ffi/hook/build.dart

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 04372c6d-c4aa-4955-8d24-512a1df7c431
📥 Commits

Reviewing files that changed from the base of the PR and between 8bcbc7d and 332cad7.

📒 Files selected for processing: 19
  • .github/ffi.md
  • .github/workflows/release-dart-ffi.yml
  • .github/workflows/release-ffi.yml
  • dart/README.md
  • dart/moq/lib/audio.dart
  • dart/moq/lib/video.dart
  • dart/moq/test/docs/prelude.dart
  • dart/moq/test/moq_test.dart
  • dart/moq_ffi/README.md
  • dart/moq_ffi/hook/build.dart
  • dart/moq_ffi/lib/src/moq.dart
  • doc/lib/dart/index.md
  • doc/lib/index.md
  • quest/m2/README.md
  • quest/m2/dart-codecs.md
  • quest/m2/mobile-completion.md
  • quest/m2/release/dart-publish.md
  • sh/dart/generate.sh
  • sh/gh/ffi.test.ts
💤 Files with no reviewable changes: 6
  • .github/workflows/release-dart-ffi.yml
  • quest/m2/mobile-completion.md
  • sh/gh/ffi.test.ts
  • quest/m2/README.md
  • dart/moq_ffi/hook/build.dart
  • quest/m2/dart-codecs.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

Walkthrough

Dart FFI builds now use moq-ffi default features, including native audio and video codecs. Dart adds public audio and video APIs with Duration views and integration tests for video encoding and decoding, audio playout delay, and Opus production. Documentation adds raw-video examples and updates codec capability guidance. The change also removes the Dart codec-parity planning item and updates the release plan with a minimum FFI dependency requirement.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 332ca

The codec examples and build changes have no established merge-blocking issue. The Dart dependency floor remains scheduled for the release cut.

Pre-merge checks: Passed 5
Check name Status Explanation
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely identifies the main changes: shipping Dart codecs and exposing audio and video namespaces.
Description check Passed The description directly explains the codec build changes, new Dart APIs, tests, documentation updates, release impact, and follow-up work.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files. (11 skipped: 11 unsupported.)

✨ Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T11:00:13.238720Z 332cad7 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b4874c93c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dart/moq/lib/audio.dart
import 'package:moq_ffi/moq_ffi.dart';

/// Encodes PCM and publishes it as a track; construct it from the broadcast's `CatalogProducer`.
typedef Producer = MoqAudioProducer;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Raise the moq_ffi dependency floor

When an existing application has moq_ffi 0.4.3 locked and upgrades moq, the unchanged ^0.4.3 constraint in dart/moq/pubspec.yaml permits that resolution, but that raw binding does not define MoqAudioProducer or the other newly aliased codec types, so the package fails to compile. Raise the dependency floor to the first codec-enabled moq_ffi release so every allowed resolution provides this API.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree with fixing this in this PR. The floor was already stale on main: media.dart aliases MoqMediaCatalogProducer and MoqMediaTimebase, which no published moq_ffi has, 0.4.16 included. The correct floor is a release that hasn't been cut yet, and package versions are bumped at release time. I added the floor bump to /quest/m2/release/dart-publish.md so it lands with the next moq-dart-v* tag.

(written by Claude Opus 5.5)

@kixelated
kixelated force-pushed the quest/m2/dart-codecs branch from cfe3bd6 to 1685da0 Compare October 17, 2026 06:52

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1685da0133

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dart/moq/lib/audio.dart Resolved
Comment thread dart/moq/lib/audio.dart Resolved
@kixelated
kixelated force-pushed the quest/m2/dart-codecs branch from 6537c85 to 213b42a Compare October 17, 2026 09:52
@kixelated

Copy link
Copy Markdown
Collaborator Author

Ready for review. Decision (a) is recorded in the PR body: the Go-style variant aliases stay as shipped.

Since the draft:

  • Rebased twice onto main and regenerated the bindings. The test and doc sample now use feat(ffi)!: media subscribe and fetch take a catalog rendition #5834's renamed id: argument.
  • Codex review: audio.Preset is added and the planar PCM wording is fixed. Both threads are resolved.
  • The moq_ffi floor finding is answered on its thread. The floor was already stale before this PR, so raising it is now a step in /quest/m2/release/dart-publish.md.

CI: 20 checks passed, 9 skipped, on head 213b42a. Local just check passes.

(written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging. Summary:

  • Changes: Dart builds moq-ffi with the default audio and video features. New package:moq/video.dart and package:moq/audio.dart aliases cover the producer, encoder, and consumer types, including MoqVideoConsumer. A Dart test decodes a subscribed H.264 rendition in CI. release-ffi.yml drops its unused default-features input, and the docs drop the "no codecs" caveat.
  • Decisions: the variant constructors keep the Go-style names (video.SoftwareEncoder(), video.Keyframe(n), video.PixelBuffer). The moq_ffi floor stays at ^0.4.3 here and gets raised at the Dart release cut; that step is tracked in /quest/m2/release/dart-publish.md.
  • Review: Codex reviewed head 213b42a. Its two codec findings are fixed and resolved, and the floor finding is answered on its thread.
  • Checks: CI passes on this head, local just check passes, and a test merge with current main leaves the generated Dart bindings unchanged.

(written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 17, 2026
kixelated and others added 7 commits October 17, 2026 03:51
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Dart libraries now build moq-ffi with its default features, like the
Kotlin and Swift ones, so `package:moq/video.dart` and
`package:moq/audio.dart` expose the native encoders, producers, and
decoding consumers. A Dart test decodes a subscribed H.264 rendition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebased onto #5844, which added the audio consumer's playout delay and
assigned its Dart exposure to this quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated force-pushed the quest/m2/dart-codecs branch from 213b42a to 332cad7 Compare October 17, 2026 10:56

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 332cad7a64

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dart/moq_ffi/hook/build.dart Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Re-enqueueing. The first queue attempt dropped for a merge conflict with #5844, which added the audio playout delay and assigned its Dart exposure to this quest. Since then:

  • Rebased onto feat(ffi): configure and observe audio playout delay in every binding #5844 and regenerated the bindings.
  • Exposed the delay as audio.DecoderOutput.delay and audio.Consumer.delay, with a test mirroring Kotlin's (passed 5 of 5 local reruns).
  • Codex reviewed 332cad7. Its one finding (the hook's iOS source build lacked IPHONEOS_DEPLOYMENT_TARGET=16.0) is fixed in 6091baf and resolved. That fix is the only change since the review.
  • Check and Test pass on 6091baf, and local just check passes.

(written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit 384d558 Oct 17, 2026
24 checks passed
@kixelated
kixelated deleted the quest/m2/dart-codecs branch October 17, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant