Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions quest/m0/interop-green/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ so it only needs to stay green.
with `gh run list --repo moq-dev/moq --workflow interop.yml --event schedule`,
then the run's STEPS summary. The JS route port that [moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290)
waited on landed in [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443),
so the pass now runs the datagram step; the `replacement` case stays
quarantined by default. Decided 2026-11-09 (maintainer, AFK, accepting the
so the pass now runs the datagram step; the `replacement` case is
quarantined by default until its child quest restores it. Decided 2026-11-09 (maintainer, AFK, accepting the
recommendation): close this quest on a pass, which the maintainer set with
those cases skipped; the pass that closes it now runs them.
those cases skipped; the pass that closes it runs all but `replacement`.
Release remeasure now updates stale planned-break pins after publication
and requests landing through the main merge queue. Check its run and PR
when a scheduled run reports stale pins.
Expand Down Expand Up @@ -221,9 +221,11 @@ Scheduled runs
fallback plays` on `moq-lite-06` and `moq-transport-22` or the fallback audio
tone, with no END_OF_TRACK violation in the log kept. Both failing logs show
`skipping missing groups` on the audio track before the stall, so the audio
stalls past the 1039 ms bound after the route falls back. Still open:
root-cause that hole before the next scheduled run, or quarantine `replacement`
on Chromium again, as on Firefox, until it is.
stalls past the 1039 ms bound after the route falls back. Reproduced
at 1 of 8 and 2 of 12 loaded local runs (the audio ring still stalled 600 ms after the first
audio group of the returned incumbent); not root-caused. Decided 2026-11-15 (AFK, recommendation, for review):
`replacement` is quarantined by default on every engine again, and the child
[replacement audio](/quest/m0/interop-green/replacement-audio.md) root-causes and restores it.

Public API: none. Wire: none.

35 changes: 35 additions & 0 deletions quest/m0/interop-green/replacement-audio.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# [M] Restore the replacement media case: audio after a route switch

## Goal

`just test media --media-cases replacement` passes every run on Chromium under CPU load, and the
default media run runs it again. Remove its quarantine in `test/interop/clients/js/media.ts` and
`test/interop/README.md`.

## Plan

Evidence from the quest's PR, with the case at `ce616b52c` and `cbb742cec` on a loaded host
(`quest gate just test media --media-cases replacement` in a loop, with and without 24 busy
loops): it failed 1 of 8, then 2 of 12 runs, and none of 31 runs after a temporary `console.warn`
per decoded audio frame, so it is a timing race. Each failure is `audioStalled:true` in
`live incumbent fallback plays` (lite-06, lite-07) or `live replacement plays`, with the picture
painting.

One failing lite-07 fallback, timed from the log:

- 238 ms after the newer publisher stopped the incumbent's announce returned; its audio
subscription was ok at 298 ms. The incumbent's `fetch error: ... group=62 error=group is past
the budget` and `skipping missing groups: track=audio 0 -> 69` follow, so the first audio group
the reader got was 69, about a second of retained audio behind the incumbent's live picture
(the stalled ring's `audioTimestamp` 9507 ms against a picture at 10548 ms).
- 600 ms after that first group, at the 1039 ms bound, the ring was still stalled.

Open: where those 600 ms go. The decoder resets the ring on a changed instance
(`js/watch/src/audio/decoder.ts` `#runDecoder`), so the join snaps onto the clock; the question is
whether the ring waits for audio at the clock behind a backlog it must drop (then a join at the
live edge is the fix), or the starved page just decodes late (then the bound needs a load term
derived from the host, not a constant). Trace the ring's join and un-stall in the worklet on a
failing run; the audio-playout epic's [largest replay](/quest/m2/audio-playout/replay-largest.md)
may overlap.

Public API: none. Wire: none.
9 changes: 5 additions & 4 deletions test/interop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -550,10 +550,11 @@ at QUIC's 1 ms timer granularity, the 500 ms GOP, one frame to decode, the 50 ms
playback delay, and one page sample plus one driver poll. Publisher preparation
is not part of it.

Firefox's replacement case is quarantined from the default run after intermittent
audio-tone failures. Investigate the missing tone samples alongside
[the audio playout work](https://github.com/moq-dev/moq/pull/8058), then restore it.
Run it explicitly with `just test media --media-engines firefox --media-cases replacement`;
The replacement case is quarantined from the default run on every engine: under CPU
load it intermittently leaves the audio stalled past the switch bound after a route
switch or fallback (the viewer logs `skipping missing groups` on the audio track first),
and Firefox also showed intermittent missing tone samples. Run it explicitly with
`just test media --media-cases replacement`, or `--media-cases replacement --media-engines firefox` for Firefox;
its content, sync, switch, and fallback assertions remain active.

The CLI integration suite's `instances` cases exercise the real QUIC relay with
Expand Down
6 changes: 3 additions & 3 deletions test/interop/clients/js/media.ts
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ const selected = new Set<string>(
values.cases === undefined
? native
? NATIVE_CASES
: CASES.filter((name) => requestedPlayer !== "firefox" || name !== "replacement")
: CASES.filter((name) => name !== "replacement")
: values.cases === "none"
? []
: values.cases.split(","),
Expand Down Expand Up @@ -161,9 +161,9 @@ if (
const publisherEngine = requestedPublisher as Engine;
const playerEngine = requestedPlayer as Engine;
const wants = (name: Case) => selected.has(name);
if (!native && values.cases === undefined && playerEngine === "firefox") {
if (!native && values.cases === undefined) {
console.error(
"QUARANTINE\treplacement\tintermittent audio tone; investigate with https://github.com/moq-dev/moq/pull/8058",
"QUARANTINE\treplacement\tintermittent audio stall after a route switch under CPU load; see quest/m0/interop-green/README.md",
);
}

Expand Down