From f565092b8b5e902fd6ad12aa608012fb794ef3b3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 21:52:25 -0700 Subject: [PATCH 1/2] quest(m1): extend the TS video-content flake with its evidence #5497 planned test-flakes-2/ts-video-content.md while this PR was open, so the evidence folds into it instead of a second quest: the 4 of 41 CI failures, the one-GOP-earlier join they share, a local reproduction of two more silent-loss modes, the hypotheses, and mocked-time fixes that keep the live-edge pick and make the chosen group atomic. Resized to L. Co-Authored-By: Claude Opus 5.5 --- quest/m1/test-flakes-2/README.md | 2 +- quest/m1/test-flakes-2/ts-video-content.md | 137 +++++++++++++++++---- 2 files changed, 114 insertions(+), 25 deletions(-) diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 913556a7e2d..b32a1237d3f 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -32,7 +32,7 @@ Public API: none. Wire: none. ## Required -- [TS video content](/quest/m1/test-flakes-2/ts-video-content.md) - CI's TS compliance `video-content` check stops failing intermittently, fixed at its cause +- [TS decoded content](/quest/m1/test-flakes-2/ts-video-content.md) - CI's TS compliance decoded-content checks fail only on real loss, and the exporter names every frame it drops - [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session - [Handshake ACK](/quest/m1/test-flakes-2/handshake-ack.md) - the impaired cluster drill's connect survives a lost handshake ACK - [Auth refresh mocks time](/quest/m1/test-flakes-2/auth-refresh-mock-time.md) - moq-relay's auth refresh and fixed-lease tests measure a grant's life on one mocked clock diff --git a/quest/m1/test-flakes-2/ts-video-content.md b/quest/m1/test-flakes-2/ts-video-content.md index 2294cbe135c..88477a3d2d3 100644 --- a/quest/m1/test-flakes-2/ts-video-content.md +++ b/quest/m1/test-flakes-2/ts-video-content.md @@ -1,30 +1,119 @@ -# [S] TS compliance keeps every decoded picture +# [L] TS compliance fails only on real loss ## Goal -CI's `TS compliance` step (`just test ts`, `.github/workflows/ts.yml`) stops -failing intermittently with check `video-content` ("decoded pictures are not -an intact suffix of the source", `test/ts/decode.py`), fixed at its cause. -The check stays as strict: a captured suffix that loses or duplicates a -picture still fails. +CI's `TS compliance` step (`just test ts`, `.github/workflows/ts.yml`) +gives the same verdict on every run of the same code. Its decoded-content +checks (`test/ts/decode.py`) fail only when the exporter really loses or +corrupts content, never because of how the host scheduled the run. Each +frame the exporter drops is counted and logged, so a failure says what was +dropped and why. The checks stay as strict: a captured suffix that loses or +duplicates a picture or an audio frame still fails. ## Plan -Seen 2026-10-13 on #5471, #5481 and two unrelated branches within the same -hour; each passed on rerun. Both PRs have since merged and touch the path: -#5471 changed the TS export (`rs/moq-mux/src/container/ts/export.rs`) and -shared source and selection code, and #5481 changed `rs/moq-mux/src/clock.rs` -and the catalog clock. The unrelated branches failing too points to an older -cause, but check a failing capture against those diffs before ruling them -out. - -This check guards real delivery, so treat it as a possible product bug first, -not a harness flake. Reproduce it by looping `just test ts` beside a CPU hog, -keep the source and capture from a failing run, and diff the picture digests -to see whether pictures are missing, duplicated, corrupted, or the suffix -start is misjudged. Then fix the side at fault: the import, relay, or export -losing or altering a picture, or the harness picking the wrong admitted -start. Never loosen the comparison, widen a timeout, or add a retry. Land a -regression that fails without the fix where one is easy. - -Public API: none expected. Wire: none. +### Evidence + +CI: seen 2026-10-13 on #5471, #5481 and two unrelated branches within the +same hour, each passing on rerun. Then, from `gh run list --workflow ts.yml` +on 2026-10-14: since #5328 merged (2026-10-13 23:49 UTC), 4 of 41 completed +PR runs failed `video-content` ("decoded pictures are not an intact suffix of +the source"), each on a branch that does not touch TS: #5446 at f6629aac8 +([38506130438](https://github.com/moq-dev/moq/actions/runs/38506130438)), +#5148 at 5adfd84a6 +([38502761750](https://github.com/moq-dev/moq/actions/runs/38502761750)), +`quest/m1/catalog-wall-clock-precision` at 312ac6260 +([38503267980](https://github.com/moq-dev/moq/actions/runs/38503267980)), and +`quest/m1/cli-duration-defaults` at e7e3eb1c4 +([38502673714](https://github.com/moq-dev/moq/actions/runs/38502673714)). +Only the default `just test ts` round failed; every later round passed in the +same runs. Main has no run yet: the workflow runs on matching PRs and a +nightly that had not fired. Failures on branches that predate #5471 and +#5481 point to an older cause. + +Every failing capture was 24531556 bytes against 23281544 in passing runs. +The difference, 1250012 bytes, is 1 s at the padded 10 Mb/s rate, so the +failing export started one 25-frame GOP earlier (group 1, not group 2). The +`--hrd` and `--headroom` rounds show the same one-GOP swing between runs but +do not run `decode.py`. Each failing run uploaded a `ts-failure` artifact +(`decode/*/frames.sha256`, `sub.log`), kept 7 days. + +Local, on 2026-10-14, `test/ts/run.sh` on a host at load ~28 on 24 cores +failed 3 of 4 runs, in two other ways: + +- `video-content`, once. The subscriber's `https://` race was won by + WebSocket, then upgraded to WebTransport; the old session "did not drain in + time" (GOAWAY timeout), and the exporter logged "current group evicted; + skipping to next buffered group" (`rs/moq-mux/src/container/consumer.rs`) + and one "frame missed its deadline". About 30 pictures across groups 10 and + 11 were missing, yet FFmpeg decoded the capture without an error. +- `audio-timing` ("capture has a gap or overlap in decoded audio"), twice, + over WebTransport with no upgrade and intact video. Two or three AAC frames + (43 to 64 ms) vanished mid-stream with no warning anywhere, and the + exporter's release clock reported `dropped=0`. + +So in each case the checks saw real loss. The flake is what makes the +exporter lose content, and the loss goes unreported. + +### Hypotheses + +1. Startup truncation (CI). The importer holds the catalog for its 2 s PCR + measurement window (`rs/moq-mux/src/container/ts/mux_rate.rs`), so the + exporter subscribes around group 2's IDR, and whether group 1 is still + the newest is a wall-clock race. A group-1 join can then keep only a + prefix of group 1 before skipping to group 2. Two candidates: + - The live subscription starts with `max_delay = 0` and only relaxes to + the budget after the first frame (`rs/moq-mux/src/container/source.rs`), + so moq-net may expire group 1 partway through. + - The jitter buffer's acquisition judges staleness per frame against its + own read time (`judge`/`queue` in `rs/moq-mux/src/jitter.rs`). The held + IDR can pass while the frames read after the tables were built are + stale, and the track then waits for the next keyframe. Those drops are + `Wait::Acquired`, which is not counted in `dropped` or logged. +2. Transport upgrade (local mode 1). The WebSocket-to-QUIC upgrade's handover + evicts the current group. [#5484](https://github.com/moq-dev/moq/pull/5484) + (open) removes the upgrade. Check whether a GOAWAY drain still loses the + current group's tail, since drains remain. +3. Silent audio loss under load (local mode 2). Each AAC frame is its own + group (`rs/moq-mux/src/container/ts/import.rs`), so a group expired by the + latency budget, or skipped by the consumer, drops one frame with no count. + +Ruled out from reading the code: picture duplication, timestamp rebasing, and +pre-IDR content in group 0 (the importer opens a group only at a keyframe). + +### Steps + +- Reproduce first. Download a `ts-failure` artifact from one of the CI runs + above before it expires (about 2026-10-21) and map the captured pictures + onto the source to confirm hypothesis 1. Locally, loop `test/ts/run.sh` + beside a CPU hog, as above. +- Fail loud: every frame the exporter drops (acquisition, late, or a skipped + or expired group) is counted and logged, and the harness prints that count + beside a failing check. +- Fix each confirmed cause at its source. Recommended for hypothesis 1: an + export starts at a whole group, so acquisition admits or drops a group + entirely rather than emitting a prefix. Keep the zero-budget live-edge + pick (`ExportSource::live`): starting with the full budget would let a + join reach back to an older retained group, adding startup latency and + replay, a user-visible change. Make the group it picks atomic instead. + Each fix lands with a regression that fails without it, on a paused Tokio + clock or the controlled clock, never wall time. The jitter and export + tests in `rs/moq-mux` already run on mocked time. +- Keep the harness on its real-time path; this quest fixes the exporter, not + the grader. If #5484 has landed, rerun on top of it before chasing + hypothesis 2. Never loosen the comparison, widen a timeout or `--delay`, or + add a retry. +- Open question for the implementer to put to the maintainer, once the causes + are known: whether a frame dropped because the host starved the exporter + past its latency budget fails `decode.py`, or fails a separate named check + fed by the exporter's drop count. Recommended: a separate check, so + `video-content` and `audio-content` mean corruption only. + +Public API: none expected; the drop count may surface in the exporter's +summary. Wire: none. + +## Related + +- [#5484](https://github.com/moq-dev/moq/pull/5484) - removes the WebSocket-to-QUIC upgrade behind local mode 1 +- [Loss and recovery tests](/quest/m1/ts-reliability/recovery.md) - deliberate MoQ content loss, which this flake hits by accident +- [Controlled time](/quest/m1/time/README.md) - the clock the regressions should run on From 21b738a9943932f3d65c696d466c80fa373b2f6c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 21:58:38 -0700 Subject: [PATCH 2/2] quest(m1): keep #5481's shared-clock caveat in the TS flake evidence Co-Authored-By: Claude Opus 5.5 --- quest/m1/test-flakes-2/ts-video-content.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/quest/m1/test-flakes-2/ts-video-content.md b/quest/m1/test-flakes-2/ts-video-content.md index 88477a3d2d3..0f827e11379 100644 --- a/quest/m1/test-flakes-2/ts-video-content.md +++ b/quest/m1/test-flakes-2/ts-video-content.md @@ -18,18 +18,21 @@ CI: seen 2026-10-13 on #5471, #5481 and two unrelated branches within the same hour, each passing on rerun. Then, from `gh run list --workflow ts.yml` on 2026-10-14: since #5328 merged (2026-10-13 23:49 UTC), 4 of 41 completed PR runs failed `video-content` ("decoded pictures are not an intact suffix of -the source"), each on a branch that does not touch TS: #5446 at f6629aac8 +the source"): #5446 at f6629aac8 ([38506130438](https://github.com/moq-dev/moq/actions/runs/38506130438)), #5148 at 5adfd84a6 ([38502761750](https://github.com/moq-dev/moq/actions/runs/38502761750)), -`quest/m1/catalog-wall-clock-precision` at 312ac6260 -([38503267980](https://github.com/moq-dev/moq/actions/runs/38503267980)), and `quest/m1/cli-duration-defaults` at e7e3eb1c4 -([38502673714](https://github.com/moq-dev/moq/actions/runs/38502673714)). +([38502673714](https://github.com/moq-dev/moq/actions/runs/38502673714)), and +#5481's branch `quest/m1/catalog-wall-clock-precision` at 312ac6260 +([38503267980](https://github.com/moq-dev/moq/actions/runs/38503267980)). +The first three do not touch TS; #5481 changed the shared clock this +pipeline reads, so keep it out of any argument that narrows the cause. Only the default `just test ts` round failed; every later round passed in the same runs. Main has no run yet: the workflow runs on matching PRs and a -nightly that had not fired. Failures on branches that predate #5471 and -#5481 point to an older cause. +nightly that had not fired. Failures on branches without #5471 or #5481 +point to an older cause, but check a failing capture against those diffs +before ruling them out. Every failing capture was 24531556 bytes against 23281544 in passing runs. The difference, 1250012 bytes, is 1 s at the padded 10 Mb/s rate, so the