Repository navigation
feat(spf): opt-in MoQ audio renderer over the shared worklet audio core - #67
Conversation
Adds createWorkletAudioRendererActor behind the AudioRendererActor seam, driving @mux/moq-media's createAudioPlayer through a structural interface injected as createAudioPlayer (engine config). Keeps the join-at-edge anchor (including re-join placement), a clock that holds where the worklet ran dry, and underrun counting from input stats; meters frames so the worklet holds its own jitter target and v10's latency stays in the subscriber buffer. Opt-in: the core has no playback rate yet (muxinc/relay-sdks#329), so playoutRate nudges do not reach audio on this path. Refs #66
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31f33919fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /** See `audio-renderer`'s threshold of the same name: a jump this large, either way, is a new timeline. */ | ||
| const DISCONTINUITY_THRESHOLD_US = TIMELINE_DISCONTINUITY_US; | ||
|
|
||
| const performanceNowUs = (): number => performance.now() * 1000; |
There was a problem hiding this comment.
Freeze the worklet clock while the audio context is suspended
When MoqMediaMixin.pause() suspends the AudioContext, this default performance.now() clock continues advancing. Consequently, getClockTimeUs() advances toward pushedEndUs, the tick can falsely classify the suspended worklet as dry and enqueue a refill, and the first report after resume can re-anchor the exposed media clock backward. This affects every pause/resume on the opt-in worklet path and can add latency or stall/slip the audio-slaved video; use a time source that freezes with the audio context, or explicitly freeze the renderer clock across suspension.
AGENTS.md reference: packages/spf/src/AGENTS.md:L21-L21
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a38555e. The renderer's clock and fill-stall timer now run on AudioContext.currentTime, which stops when the adapter suspends the context. performance.now() remains only as the core's localNow, used to convert each report's render time into context time. Covered by 'freezes the clock and stops metering while the context is suspended'. Also checked live in headless Chromium: during a 2 s pause the clock held at 3.985 s, nothing was pushed, there were no underruns, and playback resumed from the same point.
📦 Bundle Size Report🎨 @videojs/html
Presets (7)
Media (19)
Players (5)
Skins (29)
UI Components (62)
⚛️ @videojs/react — no changesPresets (7)
Media (22)
Extensions (2)
Players (5)
Skins (18)
UI Components (39)
🧩 @videojs/core — no changesEntries (76)
🏷️ @videojs/element — no changesEntries (2)
📦 @videojs/store — no changesEntries (3)
🔧 @videojs/utils — no changesEntries (13)
📦 @videojs/cdn — no changes📦 @videojs/cloudflare-video — no changes📦 @videojs/dash-video — no changes📦 @videojs/google-cast — no changes📦 @videojs/hlsjs-video — no changes📦 @videojs/media — no changesEntries (3)
📦 @videojs/mux — no changes📦 @videojs/mux-audio — no changesEntries (2)
📦 @videojs/mux-data — no changes📦 @videojs/mux-video — no changesEntries (2)
📦 @videojs/native-hls-video — no changes📦 @videojs/shaka-video — no changes📦 @videojs/spf
Entries (8)
📦 @videojs/spotify-audio — no changes📦 @videojs/tiktok-video — no changes📦 @videojs/twitch-video — no changes📦 @videojs/other — no changesEntries (3)
📦 @videojs/vimeo-video — no changes📦 @videojs/wistia-video — no changesEntries (2)
📦 @videojs/youtube-video — no changesℹ️ How to interpretEach entry is independently bundled, minified, and brotli-compressed. Initial size includes its static import graph; lazy dynamic chunks are reported separately. Entries are not additive because their dependency graphs overlap. Preset rows represent realistic combined bundles. Changes of 300 B or less across initial, lazy, and total size are collapsed, not discarded. Run |
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack
View guided diff | Turn on auto-fix | Re-trigger cubic
…ause Run the renderer's clock and fill-stall timer on AudioContext.currentTime, which stops when the adapter suspends the context, instead of performance.now(); scope each input's clock so reports from a removed input are ignored; report output.connect failures; guard the BaseAudioContext global in the adapter. Refs #66
…nd pause Hold the underrun position as a floor until the refill's future-dated anchor reaches it, and time the core's reports on context time (localNow) so its render-clock calibration stops across a suspend instead of jumping the clock by the pause on resume. Refs #66
…gth ignores loss gaps Take the recovery floor from what the clock read when the hold released, not the render-side dry point, and read Opus packet durations from the TOC byte (else the shortest step between frames) so a gap of lost packets does not extend pushed audio past where it ends. Refs #66
… reports A placed clock now stops at the end of the audio pushed and holds there until the core plays, with the first real report floored like an underrun recovery. The underrun hold ends only once the core has reported itself stopped (or on a report clearly past the dry point), so a late report of the old tail no longer releases it. Refs #66
…efill Count underruns on the renderer's own dry edges, as audio-renderer does, instead of the core's counter, which skips stalls longer than its jitter ceiling that adaptLatencyTarget still needs. Treat a frozen placed clock as filling so the fill-stall rescue can start the worklet across a packet gap. Refs #66
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
View guided diff | Turn on auto-fix | Re-trigger cubic
Detect a pending dry edge in setTrack before the input and timeline are reset, so an underrun between the last tick and the switch still counts. Refs #66
Refs #66. This is the unblocked part of the plan in #66 (comment). It does not close #66.
What
createWorkletAudioRendererActorputs the MoQ audio renderer on top of the shared AudioWorklet audio core,@mux/moq-media'screateAudioPlayerfrom muxinc/relay-sdks#323. It sits behind the existingAudioRendererActorseam, so the video renderer,trackPlayoutTime,syncLatencyandadaptLatencyTargetare unchanged.The engine takes the core as
createAudioPlayerin its config. When it is absent, theAudioBufferSourceNoderenderer stays the default.AudioPlayer,AudioPlayerInputandPlayoutClockit uses.@mux/moq-mediais not a dependency (see below).input.pushso the worklet holds about its ownjitterTargetUs. v10's target latency stays in the subscriber buffer. The core's adaptive target is left alone, and the target latency v10 resolves caps its ceiling throughclock.targetLatencyUs.getJoinAnchorUs()are dropped before the first push. A timeline jump over 1 s callsinput.cut()and re-arms the anchor. When a re-join's whole buffer is already past the anchor, the clock is placed at the anchor and the head is held back until it is due.getClockTimeUs()keeps its contract:undefinedbefore audio, held where the worklet ran dry (not wall-paced), and resuming seamlessly when audio does. Reports of audio that was not pushed on the current timeline are ignored.audio-renderercounts them, across track changes. The core'sstats.underrunsskips stalls longer than its jitter ceiling, whichadaptLatencyTargetstill needs as failure evidence.baseLatencyandoutputLatency, and exposesbaseContext, the real context anAudioWorkletNodeneeds.Why opt-in
playoutRatenudges. The core can only trim or run dry, so this path ignores nudges. I filed muxinc/relay-sdks#329 forAudioPlayerInput.playbackRate.muxinc/relay-sdksis private and this repo is public, so vendoring the core would publish unreleased code. The npm release is tracked in muxinc/relay-sdks#222 and Docs: Site Setup videojs/v10#224.Verification
pnpm -F @videojs/spf test: 2410 passed, 11 skipped. That includes 9 newcreateWorkletAudioRendererActortests against a fake core and asetupAudioRendererselection test.tsgo -p packages/spf/tsconfig.json: clean.vp lint/vp fmton the changed files: no new warnings.140b8a0into the gitignoredapps/sandbox/src/; none of it is committed. Results over 12 s:nudgingat 0.95stableafter 7.5 sThe latency row is gap 1 measured live. A fresh join plays at the edge, and only a rate nudge can build depth back to the 500 ms target. Until videojs#329 lands, the worklet path stays
nudging, which also freezesadaptLatencyTarget.Follow-up (#66)
When
@mux/moq-mediais published withplaybackRate: add the dependency, applygetPlaybackRate()toinput.playbackRate, make the core the default, and delete theAudioBufferSourceNodepath.🤖 Generated with Claude Code
Summary by cubic
Adds an opt-in MoQ audio renderer over the shared AudioWorklet audio core,
@mux/moq-media'screateAudioPlayer, behind the existingAudioRendererActorseam. When nocreateAudioPlayeris configured in the engine config, theAudioBufferSourceNoderenderer stays the default.Frames move from the subscriber's jitter buffer into the worklet so the core holds about its own jitter target, while v10's target latency stays in the subscriber buffer and caps the core's ceiling via the clock. The video renderer,
trackPlayoutTime,syncLatency, andadaptLatencyTargetare unchanged. The clock keeps its contract —undefinedbefore audio, held where the worklet ran dry — and never steps back in recovery: the hold releases only when the core reports itself stopped or a report passes the dry point, and a floor keeps the clock at the held position until the refill's first report reaches it. A placed re-join clock stops at the end of the audio pushed and holds there until the core plays, and a frozen placement refills like an underrun so a gap past its fill horizon cannot strand it. Reports are timed onAudioContext.currentTime, so the clock freezes through pause and resumes without jumping, and reports from removed inputs are ignored. Underruns count every time the worklet runs dry, asaudio-renderercounts them, including a dry edge that lands on a track change before the next tick. Frame durations are read from each Opus packet's TOC byte, falling back to the shortest step between frames, so gaps of lost packets don't extend pushed audio past where it ends.Why opt-in
playoutRatenudges are ignored on this path.@mux/moq-mediais unreleased and this repo is public, so the core is injected as a structural interface rather than a dependency.Written for commit c2beabf. Summary will update on new commits.