Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion quest/m1/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Public API: none. Wire: none.

## Required

- [TS video content](/quest/m1/test-flakes-2/ts-video-content.md) - CI's TS compliance `video-content` check stops failing intermittently, fixed at its cause
- [TS decoded content](/quest/m1/test-flakes-2/ts-video-content.md) - CI's TS compliance decoded-content checks fail only on real loss, and the exporter names every frame it drops
- [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session
- [Handshake ACK](/quest/m1/test-flakes-2/handshake-ack.md) - the impaired cluster drill's connect survives a lost handshake ACK
- [Auth refresh mocks time](/quest/m1/test-flakes-2/auth-refresh-mock-time.md) - moq-relay's auth refresh and fixed-lease tests measure a grant's life on one mocked clock
Expand Down
140 changes: 116 additions & 24 deletions quest/m1/test-flakes-2/ts-video-content.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,122 @@
# [S] TS compliance keeps every decoded picture
# [L] TS compliance fails only on real loss

## Goal

CI's `TS compliance` step (`just test ts`, `.github/workflows/ts.yml`) stops
failing intermittently with check `video-content` ("decoded pictures are not
an intact suffix of the source", `test/ts/decode.py`), fixed at its cause.
The check stays as strict: a captured suffix that loses or duplicates a
picture still fails.
CI's `TS compliance` step (`just test ts`, `.github/workflows/ts.yml`)
gives the same verdict on every run of the same code. Its decoded-content
checks (`test/ts/decode.py`) fail only when the exporter really loses or
corrupts content, never because of how the host scheduled the run. Each
frame the exporter drops is counted and logged, so a failure says what was
dropped and why. The checks stay as strict: a captured suffix that loses or
duplicates a picture or an audio frame still fails.

## Plan

Seen 2026-10-13 on #5471, #5481 and two unrelated branches within the same
hour; each passed on rerun. Both PRs have since merged and touch the path:
#5471 changed the TS export (`rs/moq-mux/src/container/ts/export.rs`) and
shared source and selection code, and #5481 changed `rs/moq-mux/src/clock.rs`
and the catalog clock. The unrelated branches failing too points to an older
cause, but check a failing capture against those diffs before ruling them
out.

This check guards real delivery, so treat it as a possible product bug first,
not a harness flake. Reproduce it by looping `just test ts` beside a CPU hog,
keep the source and capture from a failing run, and diff the picture digests
to see whether pictures are missing, duplicated, corrupted, or the suffix
start is misjudged. Then fix the side at fault: the import, relay, or export
losing or altering a picture, or the harness picking the wrong admitted
start. Never loosen the comparison, widen a timeout, or add a retry. Land a
regression that fails without the fix where one is easy.

Public API: none expected. Wire: none.
### Evidence

CI: seen 2026-10-13 on #5471, #5481 and two unrelated branches within the
same hour, each passing on rerun. Then, from `gh run list --workflow ts.yml`
on 2026-10-14: since #5328 merged (2026-10-13 23:49 UTC), 4 of 41 completed
PR runs failed `video-content` ("decoded pictures are not an intact suffix of
the source"): #5446 at f6629aac8
([38506130438](https://github.com/moq-dev/moq/actions/runs/38506130438)),
#5148 at 5adfd84a6
([38502761750](https://github.com/moq-dev/moq/actions/runs/38502761750)),
`quest/m1/cli-duration-defaults` at e7e3eb1c4
([38502673714](https://github.com/moq-dev/moq/actions/runs/38502673714)), and
#5481's branch `quest/m1/catalog-wall-clock-precision` at 312ac6260
([38503267980](https://github.com/moq-dev/moq/actions/runs/38503267980)).
The first three do not touch TS; #5481 changed the shared clock this
pipeline reads, so keep it out of any argument that narrows the cause.
Only the default `just test ts` round failed; every later round passed in the
same runs. Main has no run yet: the workflow runs on matching PRs and a
nightly that had not fired. Failures on branches without #5471 or #5481
point to an older cause, but check a failing capture against those diffs
before ruling them out.

Every failing capture was 24531556 bytes against 23281544 in passing runs.
The difference, 1250012 bytes, is 1 s at the padded 10 Mb/s rate, so the
failing export started one 25-frame GOP earlier (group 1, not group 2). The
`--hrd` and `--headroom` rounds show the same one-GOP swing between runs but
do not run `decode.py`. Each failing run uploaded a `ts-failure` artifact
(`decode/*/frames.sha256`, `sub.log`), kept 7 days.

Local, on 2026-10-14, `test/ts/run.sh` on a host at load ~28 on 24 cores
failed 3 of 4 runs, in two other ways:

- `video-content`, once. The subscriber's `https://` race was won by
WebSocket, then upgraded to WebTransport; the old session "did not drain in
time" (GOAWAY timeout), and the exporter logged "current group evicted;
skipping to next buffered group" (`rs/moq-mux/src/container/consumer.rs`)
and one "frame missed its deadline". About 30 pictures across groups 10 and
11 were missing, yet FFmpeg decoded the capture without an error.
- `audio-timing` ("capture has a gap or overlap in decoded audio"), twice,
over WebTransport with no upgrade and intact video. Two or three AAC frames
(43 to 64 ms) vanished mid-stream with no warning anywhere, and the
exporter's release clock reported `dropped=0`.

So in each case the checks saw real loss. The flake is what makes the
exporter lose content, and the loss goes unreported.

### Hypotheses

1. Startup truncation (CI). The importer holds the catalog for its 2 s PCR
measurement window (`rs/moq-mux/src/container/ts/mux_rate.rs`), so the
exporter subscribes around group 2's IDR, and whether group 1 is still
the newest is a wall-clock race. A group-1 join can then keep only a
prefix of group 1 before skipping to group 2. Two candidates:
- The live subscription starts with `max_delay = 0` and only relaxes to
the budget after the first frame (`rs/moq-mux/src/container/source.rs`),
so moq-net may expire group 1 partway through.
- The jitter buffer's acquisition judges staleness per frame against its
own read time (`judge`/`queue` in `rs/moq-mux/src/jitter.rs`). The held
IDR can pass while the frames read after the tables were built are
stale, and the track then waits for the next keyframe. Those drops are
`Wait::Acquired`, which is not counted in `dropped` or logged.
2. Transport upgrade (local mode 1). The WebSocket-to-QUIC upgrade's handover
evicts the current group. [#5484](https://github.com/moq-dev/moq/pull/5484)
(open) removes the upgrade. Check whether a GOAWAY drain still loses the
current group's tail, since drains remain.
3. Silent audio loss under load (local mode 2). Each AAC frame is its own
group (`rs/moq-mux/src/container/ts/import.rs`), so a group expired by the
latency budget, or skipped by the consumer, drops one frame with no count.

Ruled out from reading the code: picture duplication, timestamp rebasing, and
pre-IDR content in group 0 (the importer opens a group only at a keyframe).

### Steps

- Reproduce first. Download a `ts-failure` artifact from one of the CI runs
above before it expires (about 2026-10-21) and map the captured pictures
onto the source to confirm hypothesis 1. Locally, loop `test/ts/run.sh`
beside a CPU hog, as above.
- Fail loud: every frame the exporter drops (acquisition, late, or a skipped
or expired group) is counted and logged, and the harness prints that count
beside a failing check.
- Fix each confirmed cause at its source. Recommended for hypothesis 1: an
export starts at a whole group, so acquisition admits or drops a group
entirely rather than emitting a prefix. Keep the zero-budget live-edge
pick (`ExportSource::live`): starting with the full budget would let a
join reach back to an older retained group, adding startup latency and
replay, a user-visible change. Make the group it picks atomic instead.
Each fix lands with a regression that fails without it, on a paused Tokio
clock or the controlled clock, never wall time. The jitter and export
tests in `rs/moq-mux` already run on mocked time.
- Keep the harness on its real-time path; this quest fixes the exporter, not
the grader. If #5484 has landed, rerun on top of it before chasing
hypothesis 2. Never loosen the comparison, widen a timeout or `--delay`, or
add a retry.
- Open question for the implementer to put to the maintainer, once the causes
are known: whether a frame dropped because the host starved the exporter
past its latency budget fails `decode.py`, or fails a separate named check
fed by the exporter's drop count. Recommended: a separate check, so
`video-content` and `audio-content` mean corruption only.

Public API: none expected; the drop count may surface in the exporter's
summary. Wire: none.

## Related

- [#5484](https://github.com/moq-dev/moq/pull/5484) - removes the WebSocket-to-QUIC upgrade behind local mode 1
- [Loss and recovery tests](/quest/m1/ts-reliability/recovery.md) - deliberate MoQ content loss, which this flake hits by accident
- [Controlled time](/quest/m1/time/README.md) - the clock the regressions should run on
Loading