diff --git a/quest/m0/interop-green/README.md b/quest/m0/interop-green/README.md index 65c6791c03..d372321c90 100644 --- a/quest/m0/interop-green/README.md +++ b/quest/m0/interop-green/README.md @@ -34,8 +34,17 @@ synchronized tone within 1039ms ... "audioStalled":true,"rms":0`). It was the first night the case ran on Firefox, since on 2026-11-17 and 2026-11-18 the job failed earlier on "after resume", which [moq-dev/moq#9141](https://github.com/moq-dev/moq/pull/9141) fixed. This -epic owns the replacement case's history; reproduce it before planning a -child. +epic owns the replacement case's history. + +Decided 2026-11-20 (maintainer, /quest-plan), from the triage of Nightly run +[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952) and +scheduled Interop: two children, +[the replacement join](/quest/m0/interop-green/replacement-join-delay.md) +(that stall, which also reproduces on Chromium `moq-transport-22`) and +[the moq-transport JS joining FETCH](/quest/m0/interop-green/ietf-js-joining-fetch.md). +The intermittent `go -> js` silent audio went to +[test flakes](/quest/m0/test-flakes-3/go-js-silent-audio.md), and the datagram +cells are [moq-dev/moq#9197](https://github.com/moq-dev/moq/pull/9197)'s. Decided 2026-11-12 (maintainer), since green scheduled Interop is this epic's goal and stale pins caused four of the last six red runs: diff --git a/quest/m0/interop-green/ietf-js-joining-fetch.md b/quest/m0/interop-green/ietf-js-joining-fetch.md new file mode 100644 index 0000000000..c14d2c18ec --- /dev/null +++ b/quest/m0/interop-green/ietf-js-joining-fetch.md @@ -0,0 +1,34 @@ +# [S] moq-transport JS subscribers get the catalog's joining FETCH + +## Goal + +The scheduled Interop's `js -> js`, `js -> js-native-node` and +`js -> js-native-bun` cells pass on moq-transport again: the relay answers +the joining FETCH for a JS-published track instead of refusing it. + +## Plan + +Found by the 2026-11-20 triage of scheduled Interop runs; failing every night +since 2026-11-17, still on 11-20 +([44776434797](https://github.com/moq-dev/moq/actions/runs/44776434797)). +The catalog SUBSCRIBE succeeds, but the joining FETCH is refused with +`joining prefix unavailable` (`rs/moq-net/src/ietf/publisher.rs`, the +`Error::NotFound` reject), and the catalog track never gets another group, +so the subscriber times out (`timed out waiting for data`). + +Decided 2026-11-20 (maintainer, /quest-plan): ✅ m0 under interop-green +(rejected: m3), since this cell keeps scheduled Interop red every night and +hides new breaks. + +- Reproduce with + `just test interop --publishers js --subscribers js-native-node --versions moq-transport-18`. +- Prime suspect: [moq-dev/moq#8918](https://github.com/moq-dev/moq/pull/8918) + (group IDs carried verbatim on moq-transport, merged 2026-11-17). Bisect + 13e593cdb1..bc66d3eaec if reverting it doesn't clear the cell. +- Related, not the same: [the kept draft-18 join](/quest/m3/ietf-join-pin-head.md) + covers a Rust publisher dropping its producer after `finish()`. + +Caused by: [moq-dev/moq#8918](https://github.com/moq-dev/moq/pull/8918), if +the bisect confirms it. + +Public API: none expected. Wire: none. diff --git a/quest/m0/interop-green/replacement-join-delay.md b/quest/m0/interop-green/replacement-join-delay.md new file mode 100644 index 0000000000..f836b6f9aa --- /dev/null +++ b/quest/m0/interop-green/replacement-join-delay.md @@ -0,0 +1,51 @@ +# [M] A replacement or fallback joins live at the viewer's min delay + +## Goal + +After a broadcast replacement or route fallback, the JS player's audio +resumes within the `replacement` case's 1039 ms bound on every engine and +wire, because the incoming copy starts at the viewer's min delay instead of +replaying its retained backlog. The `replacement` case passes on Chromium +`moq-transport-22` and Firefox `moq-lite-07-wip`, and the quarantined rows +come back. + +## Plan + +Found by the 2026-11-20 triage of Nightly runs +[44562804083](https://github.com/moq-dev/moq/actions/runs/44562804083) and +[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952) +(Firefox `moq-lite-07-wip`: `live replacement plays` and `live incumbent +fallback plays`, `audioStalled:true`, rms 0) and scheduled Interop on 11-14, +11-18 and 11-20 (Chromium `moq-transport-22`). +`just test media --media-cases replacement --media-engines chromium` failed +5/5 on main at a759e4cf64 and 4/6 at a4fa80a747, so this is an old flake, +not a regression in that window. Firefox passed 9/9 locally, so CI's 4-vCPU +ARM runner may matter. + +Mechanism: the container reader (`js/hang/src/container/consumer.ts`, the +slow-group skip near line 660) drops the oldest group once the buffered span +passes the latency budget. Each audio frame is its own 20 ms group, so a +backlog replayed from behind the live edge arrives a group at a time and the +reader drops one group per arrival ("skipping slow groups ... count=1" +dozens of times), marking a loss each time. The audio ring keeps resyncing +while video plays. + +Decided 2026-11-20 (maintainer, /quest-plan): ✅ shrink the replay by joining +at the viewer's min delay (rejected: drop the whole stale backlog in one +reader pass; do both). On moq-transport, which states no delay, the adapters +issue a fill FETCH covering the min delay instead of the joining FETCH's +backlog. + +- `js/watch/src/audio/decoder.ts` already sets a subscription min delay + ("where the publisher starts a live join"), yet lite-07 still replays. + First find why the replacement or fallback copy does not honor it, such as + the relay's new route or a resumed reader that keeps the old window. +- [moq-dev/moq#9023](https://github.com/moq-dev/moq/pull/9023) deferred + this publisher-side alternative; read its discussion first. +- Rust and JS move together where the join is shared (`moq-net` and + `js/net`), and the moq-transport fill lives in each language's IETF + adapter. +- Restore the `replacement` rows the quarantine still skips once they pass. + +Public API: none expected. Wire: none on lite; the moq-transport adapter +sends a different FETCH. diff --git a/quest/m0/nightly/README.md b/quest/m0/nightly/README.md index eb9b96798f..f97ba61bb5 100644 --- a/quest/m0/nightly/README.md +++ b/quest/m0/nightly/README.md @@ -128,6 +128,27 @@ against 5, a one-night move near the threshold, and the Kotlin wrapper's Gradle plugin resolution failure, a one-off after two passing nights (re-check either if it repeats). +Decided 2026-11-20 (maintainer, /quest-plan), from the triage of runs +[44562804083](https://github.com/moq-dev/moq/actions/runs/44562804083) and +[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952): two +more children (resume and relay route mutants, drill leak sensitivity), and +the Firefox `replacement` stall went to +[interop green](/quest/m0/interop-green/replacement-join-delay.md). Fixed on +`main`: the ingest fuzz bound +([moq-dev/moq#9216](https://github.com/moq-dev/moq/pull/9216)), Miri's +boottime clock ([moq-dev/moq#9212](https://github.com/moq-dev/moq/pull/9212)), +the earlier resume mutants +([moq-dev/moq#9217](https://github.com/moq-dev/moq/pull/9217)), and +push-to-talk ([moq-dev/moq#9214](https://github.com/moq-dev/moq/pull/9214)). +Owned by open PRs: the JS benches +([moq-dev/moq#9099](https://github.com/moq-dev/moq/pull/9099)), the Rust bench +timeout ([moq-dev/moq#9199](https://github.com/moq-dev/moq/pull/9199)), and the +container export slowdown from +[moq-dev/moq#9196](https://github.com/moq-dev/moq/pull/9196) +([moq-dev/moq#9235](https://github.com/moq-dev/moq/pull/9235)). Dropped as +one-offs: the Windows C and Cpp release builds' runner bash startup failure, +and the Benchmark site's artifact fetch (re-check either if it repeats). + The mesh replay's violations with no known entry on 2026-11-19 and 11-20 came from debug-built relays starved for CPU on the 4-core runner, and from a strand counted again at every later event; both fixed in diff --git a/quest/m0/nightly/drill-leak-sensitivity.md b/quest/m0/nightly/drill-leak-sensitivity.md new file mode 100644 index 0000000000..e11180556d --- /dev/null +++ b/quest/m0/nightly/drill-leak-sensitivity.md @@ -0,0 +1,25 @@ +# [S] The drill catches a subscriber that leaks broadcasts again + +## Goal + +`Tests (test drill-sensitivity)` passes: with the `subscriber-leaks-broadcasts` +fault applied, `cancel_under_backpressure_releases_the_reader::loopback` +fails, as `test/drill/README.md` says it must. + +## Plan + +Found by the 2026-11-20 triage of Nightly run +[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952); it +passed on 11-17 through 11-19. + +``` +FAIL: cancel_under_backpressure_releases_the_reader::loopback still passed with 'subscriber-leaks-broadcasts' applied; it is not watching this behavior +``` + +Find the 11-19..11-20 commit that changed this: either the fault no longer +applies to the code path (the patch target moved), or the test no longer +observes the release (a change made the leak harmless or unobservable). +Fix whichever moved, the fault or the test, so the test watches the +behavior again; don't drop the pairing. + +Public API: none. Wire: none. diff --git a/quest/m0/nightly/resume-route-mutants.md b/quest/m0/nightly/resume-route-mutants.md new file mode 100644 index 0000000000..b66ef7c51b --- /dev/null +++ b/quest/m0/nightly/resume-route-mutants.md @@ -0,0 +1,29 @@ +# [S] No resume or relay route mutant survives + +## Goal + +Nightly's mutants shards report no survivors in +`rs/moq-net/src/model/resume.rs` or `rs/moq-net/src/route/relay.rs`. + +## Plan + +Found by the 2026-11-20 triage of Nightly run +[44767045952](https://github.com/moq-dev/moq/actions/runs/44767045952), after +[moq-dev/moq#9217](https://github.com/moq-dev/moq/pull/9217) killed the +previous round. The survivors as of a759e4cf64, all in code changed since: + +- `resume.rs`: `unraised` (700, 703: the `start < floor` and `end < floor` + guards and their arithmetic), `Copy::ask_whole` (854), `Subscriber::poll_peek_group` + (1172), `Subscriber::landed` (1328), `Resume::add` (1513, `at <= next`), + `Mirrored::insert`/`remove` (1653, 1659), `Reader::resolve` (1936), + `Reader::sync` (2043). +- `relay.rs`: `Router::publish` (2089 `Change::Live`, 2092), + `Router::withdraw` (2149, 2163), `Router::adopt` (2391). + +Line numbers drift; rerun `cargo mutants -f ` on current main first and +work from what still survives. For each one, add a test of the edge case the +guard decides, or delete the guard if no behavior depends on it (and say so +in the PR). The `verify-resume` and `verify-failover` models may show which +orderings a guard exists for. + +Public API: none. Wire: none. diff --git a/quest/m0/test-flakes-3/go-js-silent-audio.md b/quest/m0/test-flakes-3/go-js-silent-audio.md new file mode 100644 index 0000000000..9205a6749f --- /dev/null +++ b/quest/m0/test-flakes-3/go-js-silent-audio.md @@ -0,0 +1,30 @@ +# [S] go -> js audio plays its tone every night + +## Goal + +The interop matrix's `go -> js` cell stops failing intermittently with +silent Opus audio (`the opus rendition plays opus with 2 channels, per +channel rms 0`). + +## Plan + +Found by the 2026-11-20 triage of scheduled Interop runs: failed on 11-15, +11-17 and 11-19 (1 of 4 matrices on 11-16), passed on 11-18 and 11-20. It +passes locally, even with every subscriber pinned to 4 cores, and a one-time +1 s stall injected into the Go client did not cause silence. + +Decided 2026-11-20 (maintainer, /quest-plan): ✅ plan it now, working from CI +traces (rejected: note it and wait for repeats). + +- Pull the go-js trace from the next failing night's `interop-failure` + artifact and compare the audio and video timestamps the player received, + alongside the Go client's wall-clock write times and the player's arrival + times. The tone's timestamps count samples written, so they stay contiguous + even when ticks are dropped; only the wall-clock times can show a gap. Log + the write times if the trace lacks them. +- The Go test client paces its tone with a `time.Ticker`, which drops ticks + under load; Python and C++ pace against the start time. Switch it to + start-anchored pacing, but don't claim that as the fix unless the write + times show gaps the ticker should not have left. + +Public API: none. Wire: none.