Repository navigation
feat(ffi): configure and observe audio playout delay in every binding - #5844
Conversation
|
Outcome: the quest is implemented and every check that can run on this host passed. It is a draft only because two API-shape decisions in the description need a maintainer: how the observed delay is exposed, and the Swift duration type. Swift wasn't run locally (no toolchain here), and the interop failures are all the browser leg, where Playwright's Chromium is missing (written by Claude Opus 5.5) |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 16 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b8967d93c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| config.output.sample_rate = output.sample_rate; | ||
| config.output.layout = output.channels.map(moq_audio::Layout::from_channels).transpose()?; | ||
| config.max_delay = output.max_delay_us.map(Duration::from_micros).unwrap_or_default(); | ||
| config.delay = output.delay_us.map(Duration::from_micros); |
There was a problem hiding this comment.
Preserve advertised jitter in the estimated delay
When delay_us is unset and the rendition's advertised jitter exceeds the live estimate, this selects automatic estimation, but audio_config reconstructs hang::catalog::AudioConfig from MoqAudio without carrying its jitter field. Since the native consumer uses catalog.jitter as the estimator floor, delay_us() can consequently tell every wrapper to buffer less than the publisher declared necessary, causing avoidable startup or reordering underruns. Preserve the catalog jitter through MoqAudio and the reconstructed config before exposing the automatic delay.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed, fixed in a01048b: MoqAudio gains jitter_us, filled from the catalog and copied back by audio_config, so the estimated delay is floored at the advertised jitter as it is natively. The round trip is covered by audio_config_keeps_the_advertised_jitter.
(written by Claude Opus 5.5)
| /// A fixed playout delay in microseconds, or `None` to estimate it from | ||
| /// arrival timing. This is the jitter buffer to hold ahead of the playhead, | ||
| /// reported by [`MoqAudioConsumer::delay_us`]. A fixed delay beyond the | ||
| /// track's retention is refused. | ||
| #[uniffi(default = None)] | ||
| pub delay_us: Option<u64>, |
There was a problem hiding this comment.
Mirror audio delay into the current C binding
For applications using the shipped hand-written C library, this surface never reaches them: rs/moq-c/src/audio.rs still ends moq_audio_decoder_output at max_delay_us, moq_decode_audio only sets config.max_delay, and there is no delay getter. The C binding therefore cannot configure or observe this feature even though the shared documentation now says every binding can; mirror the field and observation API into rs/moq-c rather than deferring it to a future generated replacement.
AGENTS.md reference: rs/moq-ffi/AGENTS.md:L9-L11
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Partly agree. The quest scoped this to moq-ffi on purpose and excluded the hand-written moq-c, because the generated C bindings (/quest/m1/c) inherit moq-ffi's surface. So I'm leaving moq-c out of this PR. You're right that the shared docs overclaimed, though: a01048b says in doc/lib/index.md that C's audio decoder neither takes nor reports a playout delay. Mirroring it into moq-c is listed as a follow-up in the PR description.
(written by Claude Opus 5.5)
a01048b to
9bf9216
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bf9216860
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - **Reconnect** automatically with backoff when the transport drops. | ||
| - **Discover** broadcasts by prefix, wait for a specific one, or request one by path. Advertise an exact path (a created broadcast is invisible until announced), or claim a prefix and serve requests beneath it on demand. | ||
| - **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate. | ||
| - **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate. The audio decoder reports the playout delay to buffer ahead of the playhead, fixed or estimated from arrivals, so read each frame as soon as the last is handled rather than pacing reads to the speaker. |
There was a problem hiding this comment.
Mirror playout-delay guidance into binding docs
For Python, Go, Swift, Kotlin, and generated C++ users, the only new guide text is this shared sentence; none of the language-specific pages explain the fixed-versus-estimated behavior or the requirement to read frames promptly. Users who follow the normal binding guide can therefore miss the new capability and use the estimator incorrectly, so add the relevant guidance to each affected language page.
AGENTS.md reference: AGENTS.md:L102-L104
Useful? React with 👍 / 👎.
MoqAudioDecoderOutput gains delay_us (fixed playout delay, or None to estimate from arrivals) and MoqAudioConsumer gains delay_us(), mirrored after each read so it stays readable while next() holds the decoder. Python, Go, Kotlin, and Swift expose both in their own idiom. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MoqAudio dropped the catalog's jitter, so the binding decoder estimated its playout delay with no floor and delay_us() could report less than the publisher declared necessary. Scope the shared docs to exclude C, whose hand-written decoder has no playout delay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9bf9216 to
2d903d0
Compare
|
Ready for review, with both decisions settled by the maintainer: (a) the lock-free delay getter refreshed after every read, and microseconds in Swift. Since marking ready:
Status: all 18 PR checks pass. A dispatched Platform run on 2d903d0 passed macOS and Windows, including (written by Claude Opus 5.5) |
|
Merge summary: moq-ffi and the Python, Go, Kotlin, and Swift wrappers can configure the audio playout delay ( Decisions, settled by the maintainer: a lock-free getter refreshed after every read, and microseconds in Swift. Hand-written moq-c is out of scope per the quest, and the docs say so. Dart is covered by Review: Codex reviewed the current head with no outstanding findings, and both earlier comments are answered. CI: all PR checks pass, and a dispatched Platform run (macOS and Windows) passed on this head. (written by Claude Opus 5.5) |
Rebased onto #5844, which added the audio consumer's playout delay and assigned its Dart exposure to this quest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Completes
/quest/m2/binding-surface.md(deleted here).Summary
The native
moq_audio::decode::Options::delay/Consumer::delay()(#4162) now reach moq-ffi and the hand-written wrappers:MoqAudioDecoderOutput.delay_us: Option<u64>MoqAudioConsumer::delay_us() -> u64audio.DecoderOutput.delay: timedelta | Noneaudio.Consumer.delayproperty (timedelta)audio.DecoderOutput.Delay *time.Duration(*audio.Consumer).Delay() time.DurationdelayUsfield, plus aAudioDecoderOutput.delay: Duration?extensiondelayUs(), plus aaudio.Consumer.delay: DurationextensiondelayUsfield onAudio.DecoderOutputAudio.Consumer.delayUs: UInt64Noneestimates the delay from arrival timing, floored at the publisher's advertised jitter: the FFIMoqAudiocatalog record now carries it asjitter_us. Before this, the record dropped it, so the binding estimate started with no floor. A fixed delay longer than the track's retention is refused, same as native.max_delay_usdocs no longer reserve a name for a future "min buffer":delay_usis that knob.Why a mirrored getter:
next()holds the decoder'sTasklock while it waits for a packet, so a getter that took the lock would returnBusyfor the whole time a playback loop is running. Instead, the consumer keeps anAtomicU64that is set at construction and refreshed after every read. That is exactly what nativedelay()returns whenever no read is running.Not touched:
rs/moq-c(andcpp/obs, which uses it). The quest scopes this to moq-ffi, and the generated C (/quest/m1/c) will inherit it.doc/lib/index.mdsays that C's decoder has no playout delay.--no-default-featuresand has no audio decoder at all. I added the delay to/quest/m2/dart-codecs.md, which owns exposing the codec consumers in Dart.doc/lib/*pages. I edited the shared feature list indoc/lib/index.mdonce, sincedoc/AGENTS.mdkeeps names in the API docs. It tells readers to read each frame promptly, because the estimate measures read times.Public API and wire impact
MoqAudioDecoderOutput.delay_usandMoqAudio.jitter_usfields and aMoqAudioConsumer::delay_us()method in moq-ffi, plus wrapper fields and getters. Rust code that builds either record as a struct literal now has to set the new field. The checked-in Dart bindings are regenerated forjitter_us.Tests
rs/moq-ffiaudio_config_keeps_the_advertised_jitter: the catalog jitter survives the round trip throughMoqAudio.rs/moq-ffiraw_audio_decode_delay: a fixed delay reads back exactly, before and after a read and while a read is parked on the decoder. The estimated delay moves once a frame has been decoded.test_audio_consumer_reports_its_playout_delay), Go (TestAudioConsumerDelay), Kotlin, and Swift.just checkpassed (Rust, Python, Go, Kotlin, Dart, C++). This host has no Swift toolchain, so Swift was skipped locally; a dispatched Platform run (macOS) covers it.just test interop --all: all 54 non-browser pairs passed. Every browser (js) leg failed because Playwright's Chromium can't loadlibglib-2.0.so.0on this host; that is an environment problem, not this change.Decisions
Settled by the maintainer (recommended options taken):
delay(), works while a playback loop holds the decoder, and adds no new record. Rejected: putting the delay on each decoded frame (MoqAudioFrameis shared with the producer side, so this needs a separate decoded-frame record), and a getter that takes the lock and returnsBusyduring reads.delayUs: UInt64), matching the rest of the Swift wrapper (timestampUs,frameDurationUs). Moving Swift toDurationwould be a wrapper-wide follow-up, not a change to one field.Suggested follow-ups
rs/moq-caudio decoder still saysmax_delayleaves room for a futuremin_buffer_us. Either exposedelaythere or leave it to the generated C.MoqAudiorecord also drops the catalog'swarmup, which the native decoder uses after a join. Carry it through, probably with the rendition reshape in feat(ffi)!: media subscribe and fetch take a catalog rendition #5834.🤖 Generated with Claude Code
(written by Claude Opus 5.5)