Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions quest/m0/interop-green/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,17 @@ synchronized tone within 1039ms ... "audioStalled":true,"rms":0`). It was the
first night the case ran on Firefox, since on 2026-11-17 and 2026-11-18 the
job failed earlier on "after resume", which
[moq-dev/moq#9141](https://github.com/moq-dev/moq/pull/9141) fixed. This
epic owns the replacement case's history; reproduce it before planning a
child.
epic owns the replacement case's history.

Decided 2026-11-20 (maintainer, /quest-plan), from the triage of Nightly run
[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952) and
scheduled Interop: two children,
[the replacement join](/quest/m0/interop-green/replacement-join-delay.md)
(that stall, which also reproduces on Chromium `moq-transport-22`) and
[the moq-transport JS joining FETCH](/quest/m0/interop-green/ietf-js-joining-fetch.md).
The intermittent `go -> js` silent audio went to
[test flakes](/quest/m0/test-flakes-3/go-js-silent-audio.md), and the datagram
cells are [moq-dev/moq#9197](https://github.com/moq-dev/moq/pull/9197)'s.

Decided 2026-11-12 (maintainer), since green scheduled Interop is this epic's goal and stale pins caused four of the last six red runs:

Expand Down
34 changes: 34 additions & 0 deletions quest/m0/interop-green/ietf-js-joining-fetch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# [S] moq-transport JS subscribers get the catalog's joining FETCH

## Goal

The scheduled Interop's `js -> js`, `js -> js-native-node` and
`js -> js-native-bun` cells pass on moq-transport again: the relay answers
the joining FETCH for a JS-published track instead of refusing it.

## Plan

Found by the 2026-11-20 triage of scheduled Interop runs; failing every night
since 2026-11-17, still on 11-20
([44776434797](https://github.com/moq-dev/moq/actions/runs/44776434797)).
The catalog SUBSCRIBE succeeds, but the joining FETCH is refused with
`joining prefix unavailable` (`rs/moq-net/src/ietf/publisher.rs`, the
`Error::NotFound` reject), and the catalog track never gets another group,
so the subscriber times out (`timed out waiting for data`).

Decided 2026-11-20 (maintainer, /quest-plan): ✅ m0 under interop-green
(rejected: m3), since this cell keeps scheduled Interop red every night and
hides new breaks.

- Reproduce with
`just test interop --publishers js --subscribers js-native-node --versions moq-transport-18`.
- Prime suspect: [moq-dev/moq#8918](https://github.com/moq-dev/moq/pull/8918)
(group IDs carried verbatim on moq-transport, merged 2026-11-17). Bisect
13e593cdb1..bc66d3eaec if reverting it doesn't clear the cell.
- Related, not the same: [the kept draft-18 join](/quest/m3/ietf-join-pin-head.md)
covers a Rust publisher dropping its producer after `finish()`.

Caused by: [moq-dev/moq#8918](https://github.com/moq-dev/moq/pull/8918), if
the bisect confirms it.

Public API: none expected. Wire: none.
51 changes: 51 additions & 0 deletions quest/m0/interop-green/replacement-join-delay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# [M] A replacement or fallback joins live at the viewer's min delay

## Goal

After a broadcast replacement or route fallback, the JS player's audio
resumes within the `replacement` case's 1039 ms bound on every engine and
wire, because the incoming copy starts at the viewer's min delay instead of
replaying its retained backlog. The `replacement` case passes on Chromium
`moq-transport-22` and Firefox `moq-lite-07-wip`, and the quarantined rows
come back.

## Plan

Found by the 2026-11-20 triage of Nightly runs
[44562804083](https://github.com/moq-dev/moq/actions/runs/44562804083) and
[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952)
(Firefox `moq-lite-07-wip`: `live replacement plays` and `live incumbent
fallback plays`, `audioStalled:true`, rms 0) and scheduled Interop on 11-14,
11-18 and 11-20 (Chromium `moq-transport-22`).
`just test media --media-cases replacement --media-engines chromium` failed
5/5 on main at a759e4cf64 and 4/6 at a4fa80a747, so this is an old flake,
not a regression in that window. Firefox passed 9/9 locally, so CI's 4-vCPU
ARM runner may matter.

Mechanism: the container reader (`js/hang/src/container/consumer.ts`, the
slow-group skip near line 660) drops the oldest group once the buffered span
passes the latency budget. Each audio frame is its own 20 ms group, so a
backlog replayed from behind the live edge arrives a group at a time and the
reader drops one group per arrival ("skipping slow groups ... count=1"
dozens of times), marking a loss each time. The audio ring keeps resyncing
while video plays.

Decided 2026-11-20 (maintainer, /quest-plan): ✅ shrink the replay by joining
at the viewer's min delay (rejected: drop the whole stale backlog in one
reader pass; do both). On moq-transport, which states no delay, the adapters
issue a fill FETCH covering the min delay instead of the joining FETCH's
backlog.

- `js/watch/src/audio/decoder.ts` already sets a subscription min delay
("where the publisher starts a live join"), yet lite-07 still replays.
First find why the replacement or fallback copy does not honor it, such as
the relay's new route or a resumed reader that keeps the old window.
- [moq-dev/moq#9023](https://github.com/moq-dev/moq/pull/9023) deferred
this publisher-side alternative; read its discussion first.
- Rust and JS move together where the join is shared (`moq-net` and
`js/net`), and the moq-transport fill lives in each language's IETF
adapter.
- Restore the `replacement` rows the quarantine still skips once they pass.

Public API: none expected. Wire: none on lite; the moq-transport adapter
sends a different FETCH.
21 changes: 21 additions & 0 deletions quest/m0/nightly/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,27 @@ against 5, a one-night move near the threshold, and the Kotlin wrapper's
Gradle plugin resolution failure, a one-off after two passing nights
(re-check either if it repeats).

Decided 2026-11-20 (maintainer, /quest-plan), from the triage of runs
[44562804083](https://github.com/moq-dev/moq/actions/runs/44562804083) and
[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952): two
more children (resume and relay route mutants, drill leak sensitivity), and
the Firefox `replacement` stall went to
[interop green](/quest/m0/interop-green/replacement-join-delay.md). Fixed on
`main`: the ingest fuzz bound
([moq-dev/moq#9216](https://github.com/moq-dev/moq/pull/9216)), Miri's
boottime clock ([moq-dev/moq#9212](https://github.com/moq-dev/moq/pull/9212)),
the earlier resume mutants
([moq-dev/moq#9217](https://github.com/moq-dev/moq/pull/9217)), and
push-to-talk ([moq-dev/moq#9214](https://github.com/moq-dev/moq/pull/9214)).
Owned by open PRs: the JS benches
([moq-dev/moq#9099](https://github.com/moq-dev/moq/pull/9099)), the Rust bench
timeout ([moq-dev/moq#9199](https://github.com/moq-dev/moq/pull/9199)), and the
container export slowdown from
[moq-dev/moq#9196](https://github.com/moq-dev/moq/pull/9196)
([moq-dev/moq#9235](https://github.com/moq-dev/moq/pull/9235)). Dropped as
one-offs: the Windows C and Cpp release builds' runner bash startup failure,
and the Benchmark site's artifact fetch (re-check either if it repeats).

The mesh replay's violations with no known entry on 2026-11-19 and 11-20
came from debug-built relays starved for CPU on the 4-core runner, and from
a strand counted again at every later event; both fixed in
Expand Down
25 changes: 25 additions & 0 deletions quest/m0/nightly/drill-leak-sensitivity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# [S] The drill catches a subscriber that leaks broadcasts again

## Goal

`Tests (test drill-sensitivity)` passes: with the `subscriber-leaks-broadcasts`
fault applied, `cancel_under_backpressure_releases_the_reader::loopback`
fails, as `test/drill/README.md` says it must.

## Plan

Found by the 2026-11-20 triage of Nightly run
[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952); it
passed on 11-17 through 11-19.

```
FAIL: cancel_under_backpressure_releases_the_reader::loopback still passed with 'subscriber-leaks-broadcasts' applied; it is not watching this behavior
```

Find the 11-19..11-20 commit that changed this: either the fault no longer
applies to the code path (the patch target moved), or the test no longer
observes the release (a change made the leak harmless or unobservable).
Fix whichever moved, the fault or the test, so the test watches the
behavior again; don't drop the pairing.

Public API: none. Wire: none.
29 changes: 29 additions & 0 deletions quest/m0/nightly/resume-route-mutants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# [S] No resume or relay route mutant survives

## Goal

Nightly's mutants shards report no survivors in
`rs/moq-net/src/model/resume.rs` or `rs/moq-net/src/route/relay.rs`.

## Plan

Found by the 2026-11-20 triage of Nightly run
[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952), after
[moq-dev/moq#9217](https://github.com/moq-dev/moq/pull/9217) killed the
previous round. The survivors as of a759e4cf64, all in code changed since:

- `resume.rs`: `unraised` (700, 703: the `start < floor` and `end < floor`
guards and their arithmetic), `Copy::ask_whole` (854), `Subscriber::poll_peek_group`
(1172), `Subscriber::landed` (1328), `Resume::add` (1513, `at <= next`),
`Mirrored::insert`/`remove` (1653, 1659), `Reader::resolve` (1936),
`Reader::sync` (2043).
- `relay.rs`: `Router::publish` (2089 `Change::Live`, 2092),
`Router::withdraw` (2149, 2163), `Router::adopt` (2391).

Line numbers drift; rerun `cargo mutants -f <file>` on current main first and
work from what still survives. For each one, add a test of the edge case the
guard decides, or delete the guard if no behavior depends on it (and say so
in the PR). The `verify-resume` and `verify-failover` models may show which
orderings a guard exists for.

Public API: none. Wire: none.
30 changes: 30 additions & 0 deletions quest/m0/test-flakes-3/go-js-silent-audio.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# [S] go -> js audio plays its tone every night

## Goal

The interop matrix's `go -> js` cell stops failing intermittently with
silent Opus audio (`the opus rendition plays opus with 2 channels, per
channel rms 0`).

## Plan

Found by the 2026-11-20 triage of scheduled Interop runs: failed on 11-15,
11-17 and 11-19 (1 of 4 matrices on 11-16), passed on 11-18 and 11-20. It
passes locally, even with every subscriber pinned to 4 cores, and a one-time
1 s stall injected into the Go client did not cause silence.

Decided 2026-11-20 (maintainer, /quest-plan): ✅ plan it now, working from CI
traces (rejected: note it and wait for repeats).

- Pull the go-js trace from the next failing night's `interop-failure`
artifact and compare the audio and video timestamps the player received,
alongside the Go client's wall-clock write times and the player's arrival
times. The tone's timestamps count samples written, so they stay contiguous
even when ticks are dropped; only the wall-clock times can show a gap. Log
the write times if the trace lacks them.
- The Go test client paces its tone with a `time.Ticker`, which drops ticks
under load; Python and C++ pace against the start time. Switch it to
start-anchored pacing, but don't claim that as the fix unless the write
times show gaps the ticker should not have left.

Public API: none. Wire: none.