Skip to content

fix(audio): a gap after a declared end does not clip the next utterance - #9214

Merged
kixelated merged 3 commits into
mainfrom
quest/m0/nightly/push-to-talk-native
Nov 20, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m0/nightly/push-to-talk-native

Conversation

@kixelated

@kixelated kixelated commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m0/nightly/push-to-talk-native.md.

Problem

Nightly's Audio quality push-to-talk step failed all four native-listener checks on 2026-11-18 and 11-19, with every native metric null. The native player did play. It resumed every utterance exactly 80 ms (four 20 ms groups) after its head, so the analyzer, which matches heads by exact timestamp, never placed the native clock.

Bisected locally between the 11-17 head (cd9a8e9) and the 11-18 head (4f3ba33), on native head offset:

commit native head offset
cd9a8e9 (11-17 Nightly head) 0 on all 22 presses
9a4ed22 (moq-dev/moq#9031, voice gate) 0
789ece4 (parent of #9027) 0
0c14c87 (moq-dev/moq#9027) 80 ms on all 22
2afc94b (parent of #8984) 80 ms
d8ec56d (moq-dev/moq#8984) 80 ms
main (a4b6410) 80 ms

The cause: #9027 keys @moq/hang groups by their start in microseconds instead of 0, 1, 2. Across a pause, the publisher skips the span between its end marker and the next utterance, and declares that skip. moq-lite-06, which every session in the check negotiates (lite-07 is still -wip), names a skip as a drop. So the reader cannot prove the span empty. It reads End, then Gap, then the fresh encoder's first packet. Both players then primed the codec warmup at that gap (80 ms of Opus, from moq-dev/moq#8980) and trimmed it off the start of every utterance. A debug trace of the native run showed moq_mux handing over each utterance's head group (skipping missing groups ... continues=false), so the 80 ms was lost in the decoder, not the transport.

The browser listener had the same clip. Its render stage (ring filled to audible) moved about 80 ms later, which is the browser half of the onset regression the 11-18 night showed (median onset 62 to 152 ms).

Changes

  • rs/moq-audio/src/decode/consumer.rs: starting a fresh encoder (fresh_encoder()) drops any warmup still pending, so its first packet decodes unprimed whether a gap, a join, or both came before it. That is what a declared end with no gap already did. An undeclared loss still primes as before.
  • js/watch/src/audio/terminal.ts: the same rule for the browser's Terminal: #restart(), which a fresh encoder's first frame runs, clears #prime.
  • Tests: a_loss_after_a_declared_end_trims_the_whole_warmup becomes a_gap_after_a_declared_end_trims_no_warmup, and a_join_during_a_pause_trims_no_warmup covers a listener that joins during a pause, with and without a gap (new Break::DeclaredGap). Two Terminal tests mirror them for the browser. All four fail on the old code (Rust: stamp 1_010_000 instead of 1_000_000; JS: 0 frames instead of 960).
  • Deletes the quest.

Validation

  • just test push-to-talk on this branch: passes (exit 0). The native player resumes on the head for all 22 presses, native.offset is -1289 ms (null before), comfort noise is present on every separated mute, and there are no native underruns.
  • Second, independent measure, the browser's onset path from the same run against main: median onset 121 ms (main 197 ms, good commits 128 to 153 ms locally), and render -67 ms (main -0.5 ms, good commits -72 to -93 ms).
  • Every local run, good commits included, failed only the browser check "utterance head was not decoded" before this fix. The host was loaded (load average about 40) during those runs; this branch's run passed it.
  • cargo nextest run -p moq-audio --lib decode::consumer: 47 passed. bun test src/audio/terminal.test.ts in js/watch: 17 passed.
  • The join-during-a-pause fix (review round 1) came after the push-to-talk run. That run joins before the first press, so it does not exercise the join path; the unit tests above do.
  • just check: passed, on both heads.
  • just ci: passed (1884 Rust tests among its checks).
  • Not run: Nightly itself (never dispatched). The next scheduled night is the check.

Impact

Public API: none. Wire: none. Behavior: after a declared end, both players play the fresh encoder's first packet whole even when a gap precedes it. That gap is what every pause produces before lite-07.

Decisions

Taken while the maintainer is away; each can be undone in review:

  • Fixed the decoders' reading of the ambiguous gap, not the transport. Before lite-07 a declared skip and a real loss of the span after an end marker arrive identically (a drop with the same code). Telling them apart would mean matching on the drop's error or carrying a new bit through every relay hop. Rejected: making the container consumer stop reporting the gap. That would also hide real loss on lite-07, where the skip is carried and a gap there means media was lost.
  • The trade-off: a genuine loss of a fresh encoder's first groups is no longer primed, so its next packet decodes against the stopped encoder's state for a few frames instead of trimming 80 ms. That case needs a drop right at a resume. The clip it replaces hit every utterance after every pause on the deployed wire. This partly reverses moq-dev/moq#8980's rule that a loss after a declared end trims the whole warmup.
  • Fixed the browser in the same PR, since it is the same cause and rule in the player's mirror. It is outside the quest's native-only Goal.
  • Left the analyzer's exact head match alone: it caught a real clip.

Caused by

  • moq-dev/moq#9027 - keyed @moq/hang groups by start, so on moq-lite-06 every pause reads as End then Gap, and both players primed and trimmed 80 ms off each utterance (found by check, the Nightly push-to-talk native rows)

Follow-ups

None. The relay fetch error ... err=old bursts seen at #9027 (about 100 per long pause from the browser watcher) are already gone on main.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, high effort

(Written by Claude Opus 5.5 with high)

Review

Rewritten by quest review after each round.

  • Head: d8c196d

  • Base: origin/main at b3de94d

  • Risk: medium, The audio warmup changes could affect playback at joins and after pauses for JavaScript and native listeners, but preserve public APIs and wire formats and are fully undone by a revert.

  • Classified: medium by Codex, gpt-6.1-sol, low effort, session 01a1f6f2-3722-7fe1-8b4b-bf6faee49baf

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1f6eb-84b5-78f0-9b54-389eba53be8f, round 2

  • Score: 10/10, The incremental changes fix pending warmup in both players and add regression coverage for joins during pauses, with no further issues found by code inspection.

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • The last round's findings, at 30773ee:

    • Fixed: medium, rs/moq-audio/src/decode/consumer.rs:439: Clear an already-pending warmup after a declared end; Rust Consumer::fresh_encoder() now clears warmup_pending and JavaScript Terminal.#restart() clears #prime, with regression tests covering joins during pauses both with and without a gap.

kixelated and others added 2 commits November 19, 2026 17:55
Since groups are keyed by start (#9027), the run a publisher skips across a
pause arrives on moq-lite-06 as a drop, so every resume after a declared end
reads as End, Gap, then the fresh encoder's first packet. Both players primed
the codec warmup there, trimming the first 80 ms of every utterance. A gap
after a declared end now reaches the fresh encoder's first packet unprimed,
as a declared end alone does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Push-to-talk plays on the native listener again fix(audio): a gap after a declared end does not clip the next utterance Nov 20, 2026
…ket whole

A fresh encoder's first packet needs no warmup, so starting one drops the
warmup a join or a gap left pending, in both players.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 20, 2026 03:14
@coderabbitai

coderabbitai Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8ff6e21e-4668-4574-8694-b1b9fd1c3f1a
Commits

Reviewing files that changed from the base of the PR and between b3de94d and d8c196d.

Files selected for processing: 4

  • js/watch/src/audio/terminal.test.ts
  • js/watch/src/audio/terminal.ts
  • quest/m0/nightly/push-to-talk-native.md
  • rs/moq-audio/src/decode/consumer.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. The decoders drop the pending warmup when a fresh encoder starts, rather than trying to tell a pre-lite-07 declared skip apart from real loss at the transport. This fixes the cause of the 80 ms clip in both players without touching the wire.

Reviewed changes

All four files in the diff, with the Consumer::read gap and frame arms and Terminal.update and #restart traced against their callers.

  • Rust fresh_encoder() clears warmup_pending: the gap arm now sets the flag before fresh_encoder(). A gap after a declared end therefore decodes the fresh encoder's first packet unprimed. The frame arm already calls fresh_encoder() for a later group after the end, so a join during a pause gets the same fix: the initial warmup_pending = true is cleared there. If a joiner receives the stopped encoder's terminal packets in the end's own group, they still take the warmup first. That ordering is correct.
  • JS Terminal.#restart() clears #prime: update(gap) sets #prime, then ends() sees a group later than #endGroup and #restart() clears it before the #prime check. This mirrors the Rust rule.
  • Tests: the renamed a_gap_after_a_declared_end_trims_no_warmup, the new a_join_during_a_pause_trims_no_warmup with Break::DeclaredGap, and two mirrored Terminal tests cover the gap and no-gap join paths in both languages.
  • Quest file deleted.

ℹ️ The no-prime rule after a declared end is a pre-lite-07 accommodation with no follow-up tracking it

On lite-07 a skip is carried on the wire, so a Gap after an End there really does mean the fresh encoder's first groups were lost. After this PR that case decodes against reset codec state for a few frames instead of trimming the warmup. The trade-off is reasonable while every deployed session negotiates lite-06. The comments say it is "before lite-07", but the PR lists no follow-ups, so nothing will bring this back up once lite-07 publishes.

Technical details
# Revisit warmup-after-declared-end once lite-07 is published

## Affected sites
- rs/moq-audio/src/decode/consumer.rs:433-442 — gap arm clears warmup via `fresh_encoder()` whenever `self.end.is_some()`
- rs/moq-audio/src/decode/consumer.rs:805-809 — `fresh_encoder()` clears `warmup_pending` (keep this: the join-during-pause part is correct on every wire)
- js/watch/src/audio/terminal.ts:186-197 — `#restart()` clears `#prime`

## Required outcome
- Once lite-07 is the negotiated version, a gap after a declared end is real loss again and should prime/trim as #8980 intended. The join-during-a-pause case should still trim no warmup.

## Suggested approach (optional)
- Track this as a quest tied to lite-07 publication rather than leave it to the code comments.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge pick (maintainer away, granted the coordinator merges it confidently recommends): ✅ /quest-merge at d8c196d.

(Written by Claude Opus 5.5 with high)

@kixelated
kixelated added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 4d9190a Nov 20, 2026
9 checks passed
@kixelated
kixelated deleted the quest/m0/nightly/push-to-talk-native branch November 20, 2026 04:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant