diff --git a/quest/m0/interop-green/README.md b/quest/m0/interop-green/README.md index 27bca842fd0..3f95fab0f09 100644 --- a/quest/m0/interop-green/README.md +++ b/quest/m0/interop-green/README.md @@ -17,10 +17,10 @@ so it only needs to stay green. with `gh run list --repo moq-dev/moq --workflow interop.yml --event schedule`, then the run's STEPS summary. The JS route port that [moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290) waited on landed in [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443), - so the pass now runs the datagram step; the `replacement` case stays - quarantined by default. Decided 2026-11-09 (maintainer, AFK, accepting the + so the pass now runs the datagram step; the `replacement` case is + quarantined by default until its child quest restores it. Decided 2026-11-09 (maintainer, AFK, accepting the recommendation): close this quest on a pass, which the maintainer set with - those cases skipped; the pass that closes it now runs them. + those cases skipped; the pass that closes it runs all but `replacement`. Release remeasure now updates stale planned-break pins after publication and requests landing through the main merge queue. Check its run and PR when a scheduled run reports stale pins. @@ -221,9 +221,11 @@ Scheduled runs fallback plays` on `moq-lite-06` and `moq-transport-22` or the fallback audio tone, with no END_OF_TRACK violation in the log kept. Both failing logs show `skipping missing groups` on the audio track before the stall, so the audio - stalls past the 1039 ms bound after the route falls back. Still open: - root-cause that hole before the next scheduled run, or quarantine `replacement` - on Chromium again, as on Firefox, until it is. + stalls past the 1039 ms bound after the route falls back. Reproduced + at 1 of 8 and 2 of 12 loaded local runs (the audio ring still stalled 600 ms after the first + audio group of the returned incumbent); not root-caused. Decided 2026-11-15 (AFK, recommendation, for review): + `replacement` is quarantined by default on every engine again, and the child + [replacement audio](/quest/m0/interop-green/replacement-audio.md) root-causes and restores it. Public API: none. Wire: none. diff --git a/quest/m0/interop-green/replacement-audio.md b/quest/m0/interop-green/replacement-audio.md new file mode 100644 index 00000000000..a1a62c01fa9 --- /dev/null +++ b/quest/m0/interop-green/replacement-audio.md @@ -0,0 +1,35 @@ +# [M] Restore the replacement media case: audio after a route switch + +## Goal + +`just test media --media-cases replacement` passes every run on Chromium under CPU load, and the +default media run runs it again. Remove its quarantine in `test/interop/clients/js/media.ts` and +`test/interop/README.md`. + +## Plan + +Evidence from the quest's PR, with the case at `ce616b52c` and `cbb742cec` on a loaded host +(`quest gate just test media --media-cases replacement` in a loop, with and without 24 busy +loops): it failed 1 of 8, then 2 of 12 runs, and none of 31 runs after a temporary `console.warn` +per decoded audio frame, so it is a timing race. Each failure is `audioStalled:true` in +`live incumbent fallback plays` (lite-06, lite-07) or `live replacement plays`, with the picture +painting. + +One failing lite-07 fallback, timed from the log: + +- 238 ms after the newer publisher stopped the incumbent's announce returned; its audio + subscription was ok at 298 ms. The incumbent's `fetch error: ... group=62 error=group is past + the budget` and `skipping missing groups: track=audio 0 -> 69` follow, so the first audio group + the reader got was 69, about a second of retained audio behind the incumbent's live picture + (the stalled ring's `audioTimestamp` 9507 ms against a picture at 10548 ms). +- 600 ms after that first group, at the 1039 ms bound, the ring was still stalled. + +Open: where those 600 ms go. The decoder resets the ring on a changed instance +(`js/watch/src/audio/decoder.ts` `#runDecoder`), so the join snaps onto the clock; the question is +whether the ring waits for audio at the clock behind a backlog it must drop (then a join at the +live edge is the fix), or the starved page just decodes late (then the bound needs a load term +derived from the host, not a constant). Trace the ring's join and un-stall in the worklet on a +failing run; the audio-playout epic's [largest replay](/quest/m2/audio-playout/replay-largest.md) +may overlap. + +Public API: none. Wire: none. diff --git a/test/interop/README.md b/test/interop/README.md index 19f6c9df2fe..5927bc14502 100644 --- a/test/interop/README.md +++ b/test/interop/README.md @@ -550,10 +550,11 @@ at QUIC's 1 ms timer granularity, the 500 ms GOP, one frame to decode, the 50 ms playback delay, and one page sample plus one driver poll. Publisher preparation is not part of it. -Firefox's replacement case is quarantined from the default run after intermittent -audio-tone failures. Investigate the missing tone samples alongside -[the audio playout work](https://github.com/moq-dev/moq/pull/8058), then restore it. -Run it explicitly with `just test media --media-engines firefox --media-cases replacement`; +The replacement case is quarantined from the default run on every engine: under CPU +load it intermittently leaves the audio stalled past the switch bound after a route +switch or fallback (the viewer logs `skipping missing groups` on the audio track first), +and Firefox also showed intermittent missing tone samples. Run it explicitly with +`just test media --media-cases replacement`, or `--media-cases replacement --media-engines firefox` for Firefox; its content, sync, switch, and fallback assertions remain active. The CLI integration suite's `instances` cases exercise the real QUIC relay with diff --git a/test/interop/clients/js/media.ts b/test/interop/clients/js/media.ts index 6109c1d73ac..74eaa6210af 100644 --- a/test/interop/clients/js/media.ts +++ b/test/interop/clients/js/media.ts @@ -133,7 +133,7 @@ const selected = new Set( values.cases === undefined ? native ? NATIVE_CASES - : CASES.filter((name) => requestedPlayer !== "firefox" || name !== "replacement") + : CASES.filter((name) => name !== "replacement") : values.cases === "none" ? [] : values.cases.split(","), @@ -161,9 +161,9 @@ if ( const publisherEngine = requestedPublisher as Engine; const playerEngine = requestedPlayer as Engine; const wants = (name: Case) => selected.has(name); -if (!native && values.cases === undefined && playerEngine === "firefox") { +if (!native && values.cases === undefined) { console.error( - "QUARANTINE\treplacement\tintermittent audio tone; investigate with https://github.com/moq-dev/moq/pull/8058", + "QUARANTINE\treplacement\tintermittent audio stall after a route switch under CPU load; see quest/m0/interop-green/README.md", ); }