Skip to content

feat(ffi): configure and observe audio playout delay in every binding - #5844

Merged
kixelated merged 5 commits into
mainfrom
quest/m2/binding-surface
Oct 17, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m2/binding-surface

Conversation

@kixelated

@kixelated kixelated commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Completes /quest/m2/binding-surface.md (deleted here).

Summary

The native moq_audio::decode::Options::delay / Consumer::delay() (#4162) now reach moq-ffi and the hand-written wrappers:

Layer Configure Observe
moq-ffi MoqAudioDecoderOutput.delay_us: Option<u64> MoqAudioConsumer::delay_us() -> u64
Python audio.DecoderOutput.delay: timedelta | None audio.Consumer.delay property (timedelta)
Go audio.DecoderOutput.Delay *time.Duration (*audio.Consumer).Delay() time.Duration
Kotlin delayUs field, plus a AudioDecoderOutput.delay: Duration? extension delayUs(), plus a audio.Consumer.delay: Duration extension
Swift delayUs field on Audio.DecoderOutput Audio.Consumer.delayUs: UInt64
C++ inherited from the generated moq-ffi bindings inherited

None estimates the delay from arrival timing, floored at the publisher's advertised jitter: the FFI MoqAudio catalog record now carries it as jitter_us. Before this, the record dropped it, so the binding estimate started with no floor. A fixed delay longer than the track's retention is refused, same as native. max_delay_us docs no longer reserve a name for a future "min buffer": delay_us is that knob.

Why a mirrored getter: next() holds the decoder's Task lock while it waits for a packet, so a getter that took the lock would return Busy for the whole time a playback loop is running. Instead, the consumer keeps an AtomicU64 that is set at construction and refreshed after every read. That is exactly what native delay() returns whenever no read is running.

Not touched:

  • Hand-written rs/moq-c (and cpp/obs, which uses it). The quest scopes this to moq-ffi, and the generated C (/quest/m1/c) will inherit it. doc/lib/index.md says that C's decoder has no playout delay.
  • Dart, which builds --no-default-features and has no audio decoder at all. I added the delay to /quest/m2/dart-codecs.md, which owns exposing the codec consumers in Dart.
  • Per-binding doc/lib/* pages. I edited the shared feature list in doc/lib/index.md once, since doc/AGENTS.md keeps names in the API docs. It tells readers to read each frame promptly, because the estimate measures read times.

Public API and wire impact

  • Public API is additive: defaulted MoqAudioDecoderOutput.delay_us and MoqAudio.jitter_us fields and a MoqAudioConsumer::delay_us() method in moq-ffi, plus wrapper fields and getters. Rust code that builds either record as a struct literal now has to set the new field. The checked-in Dart bindings are regenerated for jitter_us.
  • Wire: none.

Tests

  • rs/moq-ffi audio_config_keeps_the_advertised_jitter: the catalog jitter survives the round trip through MoqAudio.
  • rs/moq-ffi raw_audio_decode_delay: a fixed delay reads back exactly, before and after a read and while a read is parked on the decoder. The estimated delay moves once a frame has been decoded.
  • New tests of the same shape in Python (test_audio_consumer_reports_its_playout_delay), Go (TestAudioConsumerDelay), Kotlin, and Swift.
  • Locally: just check passed (Rust, Python, Go, Kotlin, Dart, C++). This host has no Swift toolchain, so Swift was skipped locally; a dispatched Platform run (macOS) covers it.
  • just test interop --all: all 54 non-browser pairs passed. Every browser (js) leg failed because Playwright's Chromium can't load libglib-2.0.so.0 on this host; that is an environment problem, not this change.

Decisions

Settled by the maintainer (recommended options taken):

  1. The observed delay is a lock-free getter, refreshed after each read. It returns the same value as native delay(), works while a playback loop holds the decoder, and adds no new record. Rejected: putting the delay on each decoded frame (MoqAudioFrame is shared with the producer side, so this needs a separate decoded-frame record), and a getter that takes the lock and returns Busy during reads.
  2. Swift keeps microseconds (delayUs: UInt64), matching the rest of the Swift wrapper (timestampUs, frameDurationUs). Moving Swift to Duration would be a wrapper-wide follow-up, not a change to one field.

Suggested follow-ups

  • The hand-written rs/moq-c audio decoder still says max_delay leaves room for a future min_buffer_us. Either expose delay there or leave it to the generated C.
  • The FFI MoqAudio record also drops the catalog's warmup, which the native decoder uses after a join. Carry it through, probably with the rendition reshape in feat(ffi)!: media subscribe and fetch take a catalog rendition #5834.

🤖 Generated with Claude Code

(written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: the quest is implemented and every check that can run on this host passed. It is a draft only because two API-shape decisions in the description need a maintainer: how the observed delay is exposed, and the Swift duration type. Swift wasn't run locally (no toolchain here), and the interop failures are all the browser leg, where Playwright's Chromium is missing libglib on this host.

(written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 17, 2026 04:05
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 16 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 65c1f9a2-9168-47a1-8bef-235e9928b097
📥 Commits

Reviewing files that changed from the base of the PR and between d2cfdba and 2d903d0.

📒 Files selected for processing: 18
  • dart/moq_ffi/lib/src/moq.dart
  • doc/lib/index.md
  • go/wrapper/audio/audio.go
  • go/wrapper/moq_test.go
  • kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Durations.kt
  • kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/audio/Audio.kt
  • kt/moq/src/jvmAndAndroidTest/kotlin/dev/moq/SmokeTest.kt
  • py/moq-rs/moq/audio.py
  • py/moq-rs/tests/test_local.py
  • quest/m2/README.md
  • quest/m2/binding-surface.md
  • quest/m2/dart-codecs.md
  • rs/moq-ffi/CHANGELOG.md
  • rs/moq-ffi/src/audio.rs
  • rs/moq-ffi/src/media.rs
  • rs/moq-ffi/src/test.rs
  • swift/Sources/Moq/Audio.swift
  • swift/Tests/MoqTests/SmokeTests.swift
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T08:39:48.498438Z 2d903d0 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b8967d93c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-ffi/src/audio.rs
config.output.sample_rate = output.sample_rate;
config.output.layout = output.channels.map(moq_audio::Layout::from_channels).transpose()?;
config.max_delay = output.max_delay_us.map(Duration::from_micros).unwrap_or_default();
config.delay = output.delay_us.map(Duration::from_micros);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve advertised jitter in the estimated delay

When delay_us is unset and the rendition's advertised jitter exceeds the live estimate, this selects automatic estimation, but audio_config reconstructs hang::catalog::AudioConfig from MoqAudio without carrying its jitter field. Since the native consumer uses catalog.jitter as the estimator floor, delay_us() can consequently tell every wrapper to buffer less than the publisher declared necessary, causing avoidable startup or reordering underruns. Preserve the catalog jitter through MoqAudio and the reconstructed config before exposing the automatic delay.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in a01048b: MoqAudio gains jitter_us, filled from the catalog and copied back by audio_config, so the estimated delay is floored at the advertised jitter as it is natively. The round trip is covered by audio_config_keeps_the_advertised_jitter.

(written by Claude Opus 5.5)

Comment thread rs/moq-ffi/src/audio.rs
Comment on lines +138 to +143
/// A fixed playout delay in microseconds, or `None` to estimate it from
/// arrival timing. This is the jitter buffer to hold ahead of the playhead,
/// reported by [`MoqAudioConsumer::delay_us`]. A fixed delay beyond the
/// track's retention is refused.
#[uniffi(default = None)]
pub delay_us: Option<u64>,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Mirror audio delay into the current C binding

For applications using the shipped hand-written C library, this surface never reaches them: rs/moq-c/src/audio.rs still ends moq_audio_decoder_output at max_delay_us, moq_decode_audio only sets config.max_delay, and there is no delay getter. The C binding therefore cannot configure or observe this feature even though the shared documentation now says every binding can; mirror the field and observation API into rs/moq-c rather than deferring it to a future generated replacement.

AGENTS.md reference: rs/moq-ffi/AGENTS.md:L9-L11

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly agree. The quest scoped this to moq-ffi on purpose and excluded the hand-written moq-c, because the generated C bindings (/quest/m1/c) inherit moq-ffi's surface. So I'm leaving moq-c out of this PR. You're right that the shared docs overclaimed, though: a01048b says in doc/lib/index.md that C's audio decoder neither takes nor reports a playout delay. Mirroring it into moq-c is listed as a follow-up in the PR description.

(written by Claude Opus 5.5)

@kixelated
kixelated force-pushed the quest/m2/binding-surface branch 2 times, most recently from a01048b to 9bf9216 Compare October 17, 2026 07:13

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9bf9216860

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread doc/lib/index.md
- **Reconnect** automatically with backoff when the transport drops.
- **Discover** broadcasts by prefix, wait for a specific one, or request one by path. Advertise an exact path (a created broadcast is invisible until announced), or claim a prefix and serve requests beneath it on demand.
- **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate.
- **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate. The audio decoder reports the playout delay to buffer ahead of the playhead, fixed or estimated from arrivals, so read each frame as soon as the last is handled rather than pacing reads to the speaker.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Mirror playout-delay guidance into binding docs

For Python, Go, Swift, Kotlin, and generated C++ users, the only new guide text is this shared sentence; none of the language-specific pages explain the fixed-versus-estimated behavior or the requirement to read frames promptly. Users who follow the normal binding guide can therefore miss the new capability and use the estimator incorrectly, so add the relevant guidance to each affected language page.

AGENTS.md reference: AGENTS.md:L102-L104

Useful? React with 👍 / 👎.

kixelated and others added 5 commits October 17, 2026 01:22
MoqAudioDecoderOutput gains delay_us (fixed playout delay, or None to
estimate from arrivals) and MoqAudioConsumer gains delay_us(), mirrored
after each read so it stays readable while next() holds the decoder.
Python, Go, Kotlin, and Swift expose both in their own idiom.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MoqAudio dropped the catalog's jitter, so the binding decoder estimated
its playout delay with no floor and delay_us() could report less than
the publisher declared necessary. Scope the shared docs to exclude C,
whose hand-written decoder has no playout delay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated force-pushed the quest/m2/binding-surface branch from 9bf9216 to 2d903d0 Compare October 17, 2026 08:37
@kixelated

Copy link
Copy Markdown
Collaborator Author

Ready for review, with both decisions settled by the maintainer: (a) the lock-free delay getter refreshed after every read, and microseconds in Swift.

Since marking ready:

Status: all 18 PR checks pass. A dispatched Platform run on 2d903d0 passed macOS and Windows, including testAudioConsumerReportsItsPlayoutDelay. Codex re-reviewed the latest head with no new findings.

(written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary: moq-ffi and the Python, Go, Kotlin, and Swift wrappers can configure the audio playout delay (delay_us) and observe it (delay_us()). The estimate is floored at the catalog's advertised jitter (MoqAudio.jitter_us).

Decisions, settled by the maintainer: a lock-free getter refreshed after every read, and microseconds in Swift. Hand-written moq-c is out of scope per the quest, and the docs say so. Dart is covered by /quest/m2/dart-codecs.md.

Review: Codex reviewed the current head with no outstanding findings, and both earlier comments are answered. CI: all PR checks pass, and a dispatched Platform run (macOS and Windows) passed on this head.

(written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit 8bcbc7d Oct 17, 2026
28 checks passed
@kixelated
kixelated deleted the quest/m2/binding-surface branch October 17, 2026 10:48
kixelated added a commit that referenced this pull request Oct 17, 2026
Rebased onto #5844, which added the audio consumer's playout delay and
assigned its Dart exposure to this quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant