Repository navigation
fix: recover playout timing and frame presentation - #4
Merged
Merged
Conversation
Co-authored-by: GPT-6 <noreply@openai.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: GPT-5 <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Merge the explicit-time and driver-deadline refactors, datagram buffering, and io_uring ECN support while retaining the fork's changes. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Integrate upstream a93fdb8 and preserve fork discovery handling. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Shorten the histogram's decay so a temporary queue does not keep Auto buffered for a minute after arrivals recover. Keep the gradual target reduction and the same behavior for sustained jitter in Rust and JS. Add a twenty-second recovery regression, regenerate the shared corpus, and document the distinction between history age and quantile recovery. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Reset and reconfigure AudioEncoder when framed input leaves its continuous timeline. Chrome and Firefox otherwise stamp resumed audio beside the last pre-gap packet, putting playback behind the current video. The existing media lifecycle regression failed at 10.0 fps after resume before this change and passed at 29.8 fps after it. All local lifecycle cases and JavaScript gates pass. Clarify the mocked track test's scope. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Keep live group expiry active while FIN is unacknowledged in both JS and Rust publishers. Apply Lite priority changes throughout that wait and let the async transport adapter update priorities before acknowledgement. Regressions fail on blocked FIN and stale priority before the fixes. Verified with 2,148 JavaScript tests, JS checks/builds, and 1,593 Rust network/transport tests. Sustained-loss browser QA remains in progress. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Own worklet effects with the AudioContext so rate changes cancel pending module loads before closing the old context. Preserve errors from module loads on the current context. Reproduced the late-join AbortError in a real browser with a 44.1 kHz default context and a 48 kHz stream. The regression fails before the change; the matching browser run and full playback lifecycle now pass. JS checks/builds and all 2,150 tests pass. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
…oth on PRs (moq-dev#3850) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Reevaluate the local container budget when the shared playhead moves. Preserve completed wire groups while parsing their final buffered frames, and retain the audio clock when video skips a group. Add regressions for budget changes without new arrivals and for a buffered keyframe becoming due. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Count canvas presentation timestamps instead of decoder input. Show zero while the picture does not advance, including a paused player. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-authored-by: GPT-5 <noreply@openai.com>
Share capture with a 360p rendition capped at 30 fps and 400 kbps, inheriting lower user caps. Disable it when the main rendition is no larger. The existing watcher selection can now lower quality when the main stream exceeds capacity. Co-Authored-By: GPT-6 Astra <noreply@openai.com>
Co-authored-by: Muse Spark <noreply@meta.com>
Co-authored-by: GPT-5 <noreply@openai.com>
Co-authored-by: Codex <codex@openai.com>
The CPU bench asserted two things of each stretch path: that its thousandth-percentile quantum stays under the 0.67 ms budget, a quarter of a render quantum, and that it stays under 30 times the plain path's. The ratio came in with the features it measures (d43703e for the time stretch, 0cd6e59 for concealment), not after a regression, and the comment over it claims only that the cost is not the kind that shows up as a dropout, which is what the budget asserts directly. Since the plain quantum stopped allocating, its p999 is 2 to 6 us here, so 30 times it is a wall-clock reading rather than a cost bound: full `bun test` runs failed the concealing ratio most of the time, at 37 to 67, with the budget never close. It already failed 3 of 8 full runs at c0bbc90 on this machine. The ratio is gone; the budget stays, and so do the printed p50, p95, p999, and max. Ten full `bun test` runs in js/watch under the current load: 477 pass and 0 fail each, concealing p999 0.086 to 0.095 ms and stretching p999 0.008 to 0.011 ms against the 0.67 ms budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plain ring's mic-remote row still carried two ceilings from 5410052 that the replay stopped reaching: it measures no accelerate at all on that recording, at c1f899e and every commit since. A replay ceiling is exactly what was measured, and a loose one would let a regression back up to it pass, so both come down to the measurement: plain accelerates_per_min 7.7 -> 0 stretched_samples_per_min 38.6 -> 0 Every replay ceiling now equals its measurement, 172 of 172, and just test audio-quality --runtime replay --enforce passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The handoff's open item on `skipping covered group` printing at console.warn on the happy path is done: b92e6e1 moved it to console.debug. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
…oq-dev#3934) Co-authored-by: opencode agent <agent@opencode.ai> Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: moq-bot[bot] <186640430+moq-bot[bot]@users.noreply.github.com>
…#4014) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…moq-dev#4015) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…moq-dev#3973) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ev#4007) Co-authored-by: Jakub Migdał <j.migdal42@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…q-dev#3977) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…age test (moq-dev#3969) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ment (moq-dev#3966) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Merges the 49 upstream commits in 7ee2b02..ffa5b81 onto PR #4 at 0bd542a. Breaking upstream changes this brings: none carries a `!` marker. Behavior changes worth knowing: - moq-dev#3982 Container.Legacy.Producer.cut() now publishes a discontinuity marker group, and the audio encoder declares a demand gap with an endpoint marker. - moq-dev#3973 the container consumer plays the head group once the gap from where presentation left off exceeds the age budget. - moq-dev#3934 a refused getUserMedia is terminal until settings, devices or permission change; a camera source announces only once every enabled track is live. - moq-dev#3935 catalog readers refuse updates with more than 64 renditions. - moq-dev#3963 test/smoke is test/interop: `just test interop [--all]` and `just test media` replace `smoke` and `smoke-media`. Upstream landed no audio playout target work in this range: the audio-jitter-target quest only moved from quest/main to quest/m0, and jitter.ts, rs/moq-audio's playout and doc/concept/audio-jitter.md are untouched upstream, so the fork's estimator, rings, stall monitor and corpus stay byte for byte as they were. Conflicts: - bun.lock: upstream's version, see the last line. - doc/lib/js/net.md: upstream's Discovery line; keeps the fork's Error.Expired in Subscriptions. - doc/lib/js/publish.md: announce row carries upstream's camera wait and the fork's latch; the refusal paragraph gains the busy-device retry. - drafts/draft-lcurley-moq-hang.md: upstream's "until a later group begins", then the fork's mute endpoint rules, the last one shortened so the draft does not say the untrimmed next group twice. - js/hang/src/container/consumer.test.ts: upstream's rewrite of the shared waited-out-gap test, whose expectation (resume at B) matches the fork's. - js/justfile: upstream's interop path plus the fork's playout corpus. - js/publish/src/audio/encoder.ts: see audio below. - js/publish/src/element.ts: the fork's announce latch now closes only once upstream's readiness holds (every enabled track for a camera), so a refused microphone still withholds the broadcast and a device swap still keeps every subscriber. - js/publish/src/source/camera.ts, microphone.ts: the fork's release wait and Attempt shape; a refusal goes through Retry.refused (below), and upstream's per-attempt error clear is dropped because the fork's Retry keeps a repeating busy error set until capture runs, which that clear would toggle once a retry. - js/publish/src/source/retry.test.ts: the fake keeps both the fork's hold and upstream's denied switch. - js/watch/src/audio/decoder.ts: the fork's #declareEnd and #onNext, with upstream's `group` in #onNext's argument. - js/watch/src/audio/terminal.test.ts: both endpoint tests. - package.json: the audio-quality workspace plus upstream's interop ones. - rs/moq-cli/src/play/media.rs: upstream's shared engine and tail drain, on the fork's jitter buffer and device cushion. - test/interop/clients/js-native/tsconfig.json: the fork's file, at the renamed path. Audio, per file: - js/publish/src/audio/encoder.ts: upstream's #end field and demand-gap marker effect stay as written. The fork's mute endpoint now reads and clears the same #end, so a pause and the demand gap after it are declared once, and #end keeps the fork's frame-duration fallback. The demand-gate comments said a demand gap is left for the subscriber to find; they now say it is declared. - js/watch/src/audio/terminal.ts: upstream's endGroup rule merged clean; the fork's epoch and hole handling is untouched. - js/hang/src/container/consumer.ts: upstream's promotion (moq-dev#3973) and the fork's walk in #checkMaxAge (c1f899e) fix the same held picture with different measures, and both are kept. Upstream's measures from where presentation left off, so it fires first on a lone head; the fork's measures from the head's own start and still covers a hole with nothing presented yet, or one only later groups prove. Comments on both say which covers what. - js/hang/src/container/legacy.ts: upstream's cut() would throw writing its marker into a closed track; it now returns first, as the fork's #close already does, so cutting after the track closed stays a no-op. Adapted outside the conflict markers: - js/hang/src/container/consumer.test.ts: the fork's lone-group test starts from an empty first group, the case only the fork's walk covers; its old setup now resumes on upstream's rule at the first live frame. - js/publish/src/audio/encoder.test.ts: the fake track has `closed`, a demand gap now expects its endpoint on both the fake and the real broadcast path, and a new test pins one marker for a pause and the demand gap after it. - js/publish/src/source/retry.ts: Retry.refused retries a busy device (NotReadableError, AbortError) as the fork did and makes any other refusal terminal as upstream does; the fork's still-being-released camera test fails if every refusal is terminal. - js/watch/src/audio/terminal.test.ts, js/watch/src/sync.replay.test.ts: the fork's Terminal.update calls carry `group`. - doc/bin/cli.md: the retired rendition plays out all it buffered before the replacement starts; only the device cushion overlaps its fill. - js/publish/README.md: busy devices are retried. - test/audio-quality/clients/js/driver.ts, safari.ts: import the harness from test/interop. - test/interop/clients/js/harness.ts, .github/workflows/nightly.yml, js/hang/src/container/stall.test.ts, js/watch/src/audio/decoder.test.ts: smoke naming follows the rename. bun.lock was taken from upstream and must be regenerated with bun install. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merge took upstream's bun.lock, which has no entry for our @moq/audio-quality workspace (test/audio-quality/clients/js), so `bun install --frozen-lockfile` would reject the tree. `bun install` restores that workspace entry and moves the two renamed interop packages to their sorted place; no resolution changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream's media lane now denies the publisher's camera and microphone and fails on any console error the page reports. The fork's element logged every capture failure with console.error, so the refusal that test provokes failed the lane before its first assertion. A refusal is the user's answer rather than a fault, and the source's out.error and the element's errors output already carry it, so it warns. A device that is busy, stopped, or produced no track still logs an error. `just test media` fails without this on "console: moq-publish: camera unavailable: Permission denied" and passes with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…safety net" This reverts commit f736d9a. The run-dry safety net (91d2150, c0bbc90) is coming out: the user decided against it as a patch over the symptom rather than a fix. These two mic-firefox rows were re-recorded only because the safety net moved them, so they go back to what the replay measured before it, at c1f899e and 98f475c alike. The mic-remote plain row keeps 063a14d's re-recording, which did not depend on the safety net. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This reverts commit c0bbc90. The user decided against the run-dry safety net as a patch over the symptom rather than a fix. This takes out its second half: STALL_WINDOW, the stalled-arrival record starved() checked against, the two corpus cases and the constant it added, and its doc and tests. The revert of 91d2150 after this one takes out the rest. It reverted clean. The files it touches now match 91d2150, except decoder.test.ts, which keeps 98f475c's cloning port and the upstream sync's media-lane wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This reverts commit 91d2150. The user decided against the run-dry safety net as a patch over the symptom rather than a fix. With c0bbc90 already reverted, this takes out the rest: Container.Jitter.starved and HOLD, the held level in the target, the Starvation watcher and its wiring in the decoder and the replay harness, the Rust Jitter::starved and HOLD, the four run-dry corpus cases and hold_ms, the doc section, and every test that exercised them. The corpus is back to its seventeen cases, byte for byte. One conflict: 98f475c made the decoder test's fake processor port clone what it posts, and that port only served the run-dry tests, so it goes with them. 98f475c's copy of the postMessage ring's snapshot in the replay harness stays: the harness keeps a snapshot at warmup and in its progress series, and without the copy both would alias the ring's one refilled object. Against c1f899e, the commit below the safety net, every file this touches is identical except for three lines that came from elsewhere: that replay harness copy, and the upstream sync's `group` field in the decoder's #onNext and its media-lane wording in the decoder test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fperex
marked this pull request as ready for review
September 24, 2026 09:01
…it spans the budget" This reverts commit c1f899e. Upstream's moq-dev#3973 (db73903) fixes the same frozen picture on a lite-06 rejoin: next() plays the head once it reaches past where presentation left off by more than the budget. Keeping both meant two mechanisms for one case and a divergence from upstream in #checkMaxAge, so ours goes and upstream's stays as it landed. Conflicts, both against the upstream sync (12f9514): - consumer.ts: the walk onto a lone head before the loop goes, along with the sentences the sync added to it. The loop's condition and its two comments are what they were before c1f899e. - consumer.test.ts: upstream's two moq-dev#3973 tests stay exactly as upstream wrote them, and the fork's lone-group test, which the sync had rebuilt around an empty first group, goes. It only exercised our rule. Also outside the conflict markers: the sync reworded upstream's comment in next() to point at our walk. It is upstream's sentence again, so consumer.ts differs from c1f899e's parent only by moq-dev#3973 and by b92e6e1's covered-group debug log. What upstream's rule does not cover, and ours did: a cursor below a lone head with nothing presented yet (the first group closed empty and the next never came). That now waits for a second group, as upstream does. just test media passes three times out of three on this tree, the rejoin window in step each time: 38.2, 38.5 and 38.5 fps presented, skew median 0 ms and p95 200 ms. The lane no longer discriminates this case, though. With moq-dev#3973's promotion disabled too it still passed three of three, and consumer logging over three more runs never saw the rejoin leave the cursor on a missing group. The shape showed up once, on the resume after a pause: a lone video head still downloading, 464 ms past where presentation left off against a 20 ms budget, which moq-dev#3973 promoted at its first frame. Upstream's two moq-dev#3973 unit tests are what hold the rule. js/hang's container tests pass, 169 of 169. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update: upstream sync, run-dry safety net, audio-thread allocation (head
0bd542a84)f5506ad9d). On pages without cross-origin isolation the postMessage ring reallocated on every Auto target step, and the engine read that as a new timeline, turning an ordinary underrun into about 96 ms of silence. A resize now starts a new timeline only when it drops samples. Five tests failed before; the audio suite passes after.7ee2b0241(60 commits, including feat(quic)!: build on moq-noq, the MoQ fork of noq moq-dev/moq#3866 moq-noq QUIC, refactor(video)!: separate decoder output from subscription policy moq-dev/moq#3875 video decoder output and feat(net)!: promote moq-lite-06 ALPN on main moq-dev/moq#3941 moq-lite-06 ALPN). feat(watch): add headless Player pipeline moq-dev/moq#3931 moved the watch element's wiring intoPlayer; the fork's element behaviours (conceal, a detached player releasing its audio context, Sync without the RTT probe, per-track spread wiring) are ported into it. The playout engine, estimator, stall monitor and rings are byte-identical across the merge.Container.Jitter.starved(gap, now)raises the target to cover a run-dry when a stalled arrival was seen within 200 ms, holds it forHOLD(provisionally 30 s) after the last one, then falls as before. Run-dries without a page block (lost media, path outages, late delivery) change nothing. Simulated 120 ms freezes every 1.5 s for 30 s: 20 underruns before, 1 at a 30 s hold and 6 at a 4 s hold, on both rings. The shared corpus gains six cases, read by both languages.audio-qualityworkflow runs the replay lane on pull requests that touch the player and the Chromium matrix on demand. Replay budgets were re-recorded where the changes above moved them, with every value in the commit bodies.Validation at this head:
just check --all,just test(JavaScript and Rust), the playout corpus in both languages, the replay lane with enforcement,just test smoke-media(3 of 3 on moq-lite-06, 1 of 1 on moq-lite-05) andjust test smoke --all(32 of 32 cells). Browser rows on one Mac over loopback, on the previous head: without freezes the player holds a 20 ms Auto target with no underruns; injected 30 and 60 ms main-thread freezes are absorbed; 120 ms freezes cut on every freeze, which the safety net addresses (its browser rows are pending). A listening check on the previous head heard no chops.Not measured yet: the safety net and the synced build in real browsers, Firefox and Safari on this head, phones, and a wide-area path. Two open findings: a publish start can lose the first few hundred ms of speech at a listener whose player is still starting, and Opus DTX (on by default for voice in the room page) raises the Auto target and clips word starts.
API:
Container.Jitter.starvedis new and additive. Nothing on the wire changes beyond what the upstream sync brings.Problem
Follow-up to #3, targeting
debug-findings-solution. Delivery gaps and quality changes can leave video behind audio. The player can report 60 FPS while displaying fewer frames, and Auto can retain old congestion. Completed streams can lose expiry handling before acknowledgement. A Firefox publisher also stops sending video while its own window is hidden, so a watcher sees the picture freeze while the sound keeps playing.Approach
91b43964). Firefox exposes noMediaStreamTrackProcessorin the window or in a worker, so it captures through the video element polyfill, and it throttlesrequestVideoFrameCallbackand main-thread timers to once a second in a hidden document while the element keeps advancing at 1.0x. A dedicated worker interval is the one clock that survives, at 27 FPS hidden, so the capture worker now keeps time and the polyfill captures on that tick whenever the frame callback has been quiet for two frame periods.a93fdb8e8, migrate affected callers, and addbash .agents/commands/sync-upstreamwith clean-tree checks, backup refs and CI tests.Validation
Measured on one macOS host, with a real camera and a real 48 kHz microphone, publishing and watching through a loopback relay, one browser engine at a time.
A real-camera matrix on
9a8e8424bran 40 browser rows plus the deterministic media lifecycle fixture. 39 rows exited 0, and the single exit 1 reports only the sampler gap a genuinely throttled hidden Safari tab is expected to produce. 71 graded transitions carry zero findings between them. Gates are 80 ms of A/V skew, 250 ms of display gap, and a 200 ms Auto target outside the deliberate loss profiles.That matrix produced the Firefox publisher finding, and
91b43964fixes it. The same row, publisher window hidden for 60 seconds, before and after:After the fix, three stalls of about 0.55 s remain in the first 17 seconds hidden, and camera-back-on carries one 3.1 s skew transient.
Regressions fail before the corresponding corrections. The adjacent-frame regression independently fails against frozen upstream too.
CI on
91b43964: Check and Test and Full Smoke are green, after re-running two failed jobs whose failures did not reproduce, amoq-tokioreconnect 200 ms timeout and a media cold-start audio-tone agreement of 88% against a 90% floor following an autoplay warning. Full Smoke is run 35720939426. The prior head9a8e8424bhad Check and Test green and Full Smoke green on its parent.Local
just checkpasses on this source, the full workspace including the Nix flake step. Localjust testpasses on head91b43964: 4,744 Rust tests passed, 0 failed, 0 timed out, plus 2,164 JavaScript tests and 64 Python tests passed, in 6 minutes on a fresh build directory. An earlier run on a build directory that had grown to 678,000 entries showed 4 failures and 74 timeouts, all traced to macOS scanning the executable's containing directory during VideoToolbox session setup, none to code.Not measured: Safari publishing from a real camera, which WebDriver forbids, and physical end-to-end latency. Every row ran on loopback through a local shaper on this one host, so nothing here describes other hardware or a wide-area path. The bench grades counters and timestamps, not picture quality, sound quality or lip sync feel.
Impact
js/publish; the Chrome and Safari paths throughMediaStreamTrackProcessorare untouched.Alternatives
Clock-estimate smoothing did not prevent repeated presentation loss and was removed. The bounded renderer queue addresses the reproduced overwrite. Polling the hidden Firefox document from the main thread was rejected because every main-thread clock there is throttled to once a second. No arbitrary jitter ceiling conceals delivery lag; upload adaptation needs separate scope.
Follow-ups
High-quality upload congestion remains unresolved, and no row in this round exercised it. The retained result stands as recorded: uplink-only jitter during a 720p/30 to 1080p/60 change raised Auto to 600 ms and took 5.7 seconds to recover, where downlink-only jitter stayed at 80 ms and recovered in 555 ms.
The congested access point profile, about 4.8 Mb/s with loss, reorder and bursts, holds cadence and skew but runs an Auto target of 360 ms with a 182 ms maximum frame gap. That profile is deliberately outside the 200 ms gate and is the remaining bad-network latency limit.
Safari uses WebSocket under upstream's WebKit gate, so nothing in the Safari rows is evidence about QUIC behaviour under loss, and Safari real-camera publishing is still owed as a manual check. A stall gate spanning the whole row, rather than only the graded windows, would have caught the Firefox publisher finding earlier.
This head is synced with upstream main through
7ee2b0241at the author's request; the earlier separate merge branch is superseded. Further upstream syncs happen only when requested.All local and CI gates pass; the draft flag is left for the author to lift after their own playback check.
Raw media, screenshots and device logs remain private. Nothing has been posted upstream.
(Written by gpt-6-astra; updated by Claude Fable 5.1; updated by Claude Opus 5.5)