diff --git a/quest/m0/interop-green/README.md b/quest/m0/interop-green/README.md index 239aea65216..e7ffe2091f0 100644 --- a/quest/m0/interop-green/README.md +++ b/quest/m0/interop-green/README.md @@ -1,4 +1,4 @@ -# [XS] Interop and Video pass on main +# Interop and Video pass on main ## Goal @@ -10,12 +10,32 @@ so it only needs to stay green. ## Waiting -- moq-dev/moq: a scheduled Interop run on `main` passes. Check with - `gh run list --repo moq-dev/moq --workflow interop.yml --event schedule`. - Triage any red run, then delete this quest once one passes. +- moq-dev/moq: the next scheduled Interop run on `main` passes, once the + [resumed-audio child](/quest/m0/interop-green/resume-sync.md) lands + ([moq-dev/moq#8572](https://github.com/moq-dev/moq/pull/8572), the max-age + fix, merged 2026-11-12). Check + with `gh run list --repo moq-dev/moq --workflow interop.yml --event schedule`, + then the run's STEPS summary. The JS route port that [moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290) + waited on landed in [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443), + so the pass now runs the datagram step; the `replacement` case stays + quarantined by default. Decided 2026-11-09 (maintainer, AFK, accepting the + recommendation): close this quest on a pass, which the maintainer set with + those cases skipped; the pass that closes it now runs them. Decided + 2026-11-12 (maintainer): this epic also waits for its + [planned-break remeasure](/quest/m0/interop-green/planned-break-remeasure.md) + child, so the pass closes the epic only once that child lands too. Until then, + a red run whose only failure is a stale planned break is fixed by hand: run + `just test wire-compat --remeasure` and land the rewritten + `test/interop/compat/planned-breaks.json`, as RELEASE.md's release + procedure describes. ## Plan +Decided 2026-11-12 (maintainer), since green scheduled Interop is this epic's goal and stale pins caused four of the last six red runs: + +- ✅ the release-remeasure pipeline is a child quest of quest/m0/interop-green/ (rejected: quest/m3/release/; drop the follow-up). See [planned-break remeasure](/quest/m0/interop-green/planned-break-remeasure.md). +- ✅ the release cut keeps its manual post-publish `just test wire-compat --remeasure` step until that child lands, and does not require it (rejected: block the cut on the new quest; accept a red Interop run per release). + Decided 2026-10-27 (maintainer), from an external release-to-main audit and a triage of Interop runs [40517499786](https://github.com/moq-dev/moq/actions/runs/40517499786), @@ -102,4 +122,80 @@ moq-watch's `buffer`, which `lookahead`. Fixed in this quest's PR: the case tests `lookahead`, and a new check confirms `buffer` reports its rename and recovers once removed. +Scheduled run +[42491651687](https://github.com/moq-dev/moq/actions/runs/42491651687) +(2026-11-08, at e2271bad9c) failed two steps, and neither is a flake. Both come +from the JS lite-07 route wire, which Rust moved to but JS has not ported, so +they were owned by the JS route port, which landed as +[moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443), not this quest: + +- Datagram tracks: `datagram_tracks_javascript` times out in its first case + (relay false, JS publishes false). It fails every time on main's content + (`just test datagram-tracks`), not under load. +- Media output and lifecycle: chromium's `replacement` case crashes both + moq-lite-07-wip pages with `page crashed`. It fails every time on main's + content (`just test media --media-cases replacement`), though it passed on an + older base. + +[moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290) skipped the +datagram step and quarantined `replacement` by default, which this quest's +triage confirms. On main's content, `just test media` passes; the datagram test +still fails until the port lands. No new quest or retry is needed here. + +Scheduled run +[42673842120](https://github.com/moq-dev/moq/actions/runs/42673842120) +(2026-11-09) and +[42851744235](https://github.com/moq-dev/moq/actions/runs/42851744235) +(2026-11-10) failed Harness regressions and Released wire compatibility. The +staging fixture's fake `cargo` lacked `tree`, which `test/interop/compat.sh` +calls, so its test exited 2; [moq-dev/moq#8455](https://github.com/moq-dev/moq/pull/8455) +stubs it. The planned break's pin, moq-cli 0.14.14, went stale when 0.14.15 +shipped. + +Scheduled run +[43052960736](https://github.com/moq-dev/moq/actions/runs/43052960736) +(2026-11-11, at a4bc922c6) failed the same two steps. The invalid-attribute +browser case restored `lookahead` to the number `0` after its `try` block, +which the setter refuses, so the evaluate threw. +[moq-dev/moq#8512](https://github.com/moq-dev/moq/pull/8512) passes the page's +own Duration and remeasures the pin to moq-cli 0.14.16. + +`main` at d3b6c2573 has both fixes, and every pin matches the latest release of +its package (moq-cli 0.14.16, moq-relay 0.17.16, @moq/net 0.4.7). The stale pin +is the recurring cause: each published release of a pinned package turns the +next scheduled run red until someone remeasures. + +Scheduled run +[43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067) +(2026-11-12, at e9c6df852) failed three steps, each with its own cause: + +- Optional publisher retention: `max_age_relay_javascript` fails on lite-07 + track 1, which reads a max age of 0 where the test expects none. + [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443) added + `moq-lite-07-wip` back to `max_age_relay_javascript` in `max_age.rs`, but the + JS subscriber kept its moq-transport-only expectation of no max age; lite + carries it, as the Rust side of the test expects. + [moq-dev/moq#8572](https://github.com/moq-dev/moq/pull/8572) fixes the + expectation, and is open. +- Interop (full matrix): `go -> python` fails with `audio gap after 648 samples` + at 2 s of 30 s. It passed once locally, and 8 more local runs of + `--publishers go --subscribers python,rust,go` passed every cell. A lead, not + checked: [moq-dev/moq#8531](https://github.com/moq-dev/moq/pull/8531) now waits on + a missing group in moq-audio only for the playout delay, which is small at a + cold start, so a reordered group early on reads as a gap. No open PR names this cell; + [moq-dev/moq#8587](https://github.com/moq-dev/moq/pull/8587) covers `python -> go`. + Decided 2026-11-12 (maintainer): the cell moves to + [test flakes](/quest/m0/test-flakes-2/interop-go-python.md), which checks + #8531 first. +- Media output and lifecycle: after the pause case resumes, the audio trails the + picture by about 1.7 s. [moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507) + keeps audio that trails the clock up to its 2 s stretch bound and plays it + faster, and a resume keeps the ring it had. Fixed by the child + [resumed audio snaps to the clock](/quest/m0/interop-green/resume-sync.md), + decided 2026-11-12 (maintainer) to snap to the live picture rather than + stretch the lag back. + +The 11-11 run, before #8507, passed all three steps. + Public API: none. Wire: none. + diff --git a/quest/m0/interop-green/planned-break-remeasure.md b/quest/m0/interop-green/planned-break-remeasure.md new file mode 100644 index 00000000000..7f38fa349c9 --- /dev/null +++ b/quest/m0/interop-green/planned-break-remeasure.md @@ -0,0 +1,38 @@ +# [M] Re-measure planned breaks in the release pipeline + +## Goal + +Each release that publishes a package a planned break pins (moq-cli, +moq-relay, `@moq/net`) re-measures those pins against the published versions +and lands the rewritten `test/interop/compat/planned-breaks.json` on `main` +with the release. No scheduled Interop run goes red on a stale pin, and no one +runs the remeasure by hand. + +## Plan + +Decided 2026-11-11 (maintainer), after moq-cli 0.14.15 and 0.14.16 each made +the `moq-lite-07-wip` pin stale and the scheduled Interop runs on Nov 9 to 11 +went red on it: + +- ✅ the release pipeline runs `just test wire-compat --remeasure` and lands the rewritten `test/interop/compat/planned-breaks.json` with the release (rejected: keep the manual post-publish step; warn on stale pins in scheduled runs). + +Decided 2026-11-12 (maintainer), since stale pins caused four of the last six red scheduled Interop runs. This quest is a child of the [Interop epic](/quest/m0/interop-green/README.md): + +- ✅ the release-remeasure pipeline is a child quest of quest/m0/interop-green/ (rejected: quest/m3/release/; drop the follow-up). +- ✅ the release cut keeps its manual post-publish `just test wire-compat --remeasure` step until this quest lands, and does not require it (rejected: block the cut on this quest; accept a red Interop run per release). + +The job runs after the publish jobs for the three packages the planned-breaks +file pins, in the workflows that publish from `release` (`.github/workflows/release-*.yml`). +It re-measures the pins against the versions just published and lands the +rewritten file on `main`, which RELEASE.md's release procedure does by +hand today. Check `main`'s ruleset first: where it requires a pull request, the +job opens one. Its write credentials go in an environment limited to protected +branches. + +Verify it by running the job on its own against the current pins (moq-cli +0.14.16, moq-relay 0.17.16, `@moq/net` 0.4.7). It should reproduce the file +with no diff. + +The same PR updates RELEASE.md's release bullet, which says to run the +remeasure by hand, to name the job. The nightly wire-compat check keeps +running as it does now. diff --git a/quest/m0/interop-green/resume-sync.md b/quest/m0/interop-green/resume-sync.md new file mode 100644 index 00000000000..a3715a23740 --- /dev/null +++ b/quest/m0/interop-green/resume-sync.md @@ -0,0 +1,26 @@ +# [S] Resumed audio snaps to the clock + +## Goal + +A viewer that pauses and resumes hears its audio in sync with the picture from the first frame, as it did before [moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507). So `just test media`'s pause and rejoin steps pass on every run, and a resumed viewer is not seconds behind for the stretch to catch up. Audio that falls behind because of network delay still catches up by playing faster; that is #8507's stretch and stays. + +Not in scope: the JS max-age expectation ([#8572](https://github.com/moq-dev/moq/pull/8572)) and the `go -> python` media cell. + +## Plan + +Found by the triage of scheduled Interop run [43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067) (2026-11-12, head e9c6df852). Its `Media output and lifecycle` step failed on the pause case: `audio tone: after resume: 70/81 samples` in CI, and locally `audio/video sync: after resume: 0/79 samples within 200ms, median skew 600ms`. The 11-11 run, before #8507, passed that step. + +Cause: #8507 keeps audio that trails the shared clock by up to the stretch's 2 s bound and plays it faster to catch up (`js/watch/src/audio/ring-buffer.ts` `read`, bound from `js/watch/src/audio/latency.ts` `bound`). Before it, the ring skipped whatever trailed the clock. A return from pause resubscribes to the same broadcast instance, so `js/watch/src/audio/decoder.ts` keeps the predecessor's ring without a join. The resumed picture lands on the live edge, and the audio that trails it by about 1.7 s is stretched back over about 5 s. + +Evidence, `just test media --media-cases pause` in `test/interop/clients/js/media.ts`: + +- `main` at 18fc33ecc: 3 of 4 runs fail the resume sync (median skew 200 ms, or 0 of 79 in sync). +- Parent of #8507 (ac73d9e0b): passes, median skew 0 ms (one run). On a host at load 30, four full `just test media` runs there passed 1, and the other 3 failed only `audio tone` (67 to 70 of 81 samples, the CI signature), never sync; the 11-11 head a4bc922c6 passed 4 of 6, failing the same way. #8507 itself (4262a0691) failed 4 of 4 on sync, median skew 400 to 600 ms after resume. So the CI step can stay red under load after this fix, from a tone flake older than #8507. +- Experiment, not committed: a same-instance resubscribe calls the decoder's `reset()`, so the ring joins the clock again. 4 of 4 runs pass, median skew 0 ms. One full `just test media` run also passed the pause and rejoin steps, then failed the later `live incumbent fallback` case, which was not investigated. + +Decided 2026-11-12 (maintainer): ✅ snap to the live picture. Re-join the ring when a viewer returns from absence on the same instance, so resumed audio lands on the clock the way a tune-in does, and keep the 2 s bound for stalls. Rejected: stretch catch-up with #8507's time-stretch, which plays the lag back over about 5 s and costs the lip sync that #8031 fixed; relaxing the resume assertion to match it, since `js/watch/AGENTS.md` says to judge a player change by its lag first. Check it with `just test media --media-cases pause`. + +Open for the implementer: + +- Scope the re-join to a return from absence. The handover path (`#handover` in `decoder.ts`) also inherits the ring for a rendition swap and a fallback, which should keep their seamless handover. Check the fallback case. +- Native parity: `moq play` pauses through `rs/moq-audio`. Check whether its ring inherits the same way; `just test media` does not cover it. diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 5ae4f1d6c10..ba124f00daf 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -280,4 +280,12 @@ its third failure since 2026-10-16 and its first after [moq-dev/moq#8526](https://github.com/moq-dev/moq/pull/8526). It is quarantined with `#[ignore]` until its child fixes it. +Added 2026-11-12 (maintainer, from the Interop triage of scheduled run +[43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067)): the +`go -> python` media cell, which failed `audio gap after 648 samples` once and +passed 8 local runs after it. Decided: ✅ add it here as the child +[interop's go -> python media cell](/quest/m0/test-flakes-2/interop-go-python.md), +not quarantined (rejected: a new quest after a second red run; quarantining the +cell). Check [moq-dev/moq#8531](https://github.com/moq-dev/moq/pull/8531) first. + Public API: none. Wire: none. diff --git a/quest/m0/test-flakes-2/interop-go-python.md b/quest/m0/test-flakes-2/interop-go-python.md new file mode 100644 index 00000000000..b6a0b53f1c3 --- /dev/null +++ b/quest/m0/test-flakes-2/interop-go-python.md @@ -0,0 +1,29 @@ +# [S] Interop's go -> python media cell holds up + +## Goal + +The `go -> python` media cell of `just test interop --all` passes reliably, +fixed at its cause, never by a quarantine, a longer timeout, or a retry. + +## Plan + +Found 2026-11-12 in the Interop triage of scheduled run +[43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067) (head +e9c6df852, the first run after #8455 and #8512). Its full-matrix step failed +`go -> python` with `audio gap after 648 samples` at 2 s of 30 s. It passed +once locally, and 8 more local runs of `--publishers go --subscribers +python,rust,go` passed every cell. No open PR names this cell. + +Decided 2026-11-12 (maintainer): ✅ track it as this child of test flakes +(rejected: a new quest after a second red run; quarantining the cell). + +Check first: [moq-dev/moq#8531](https://github.com/moq-dev/moq/pull/8531) +(`fix(hang,moq-mux): wait on a missing group for the delay, not the max`) merged +on 2026-11-12 and is in the failing run's head. A lead, not checked: it waits on +a missing group only for the playout delay, which is small at a cold start, so a +group that arrives out of order early on reads as an audio gap. + +Reproduce with `just test interop --publishers go --subscribers python` and the +interop logs kept. Confirm or rule out #8531 first, then fix the cause. + +Public API: none expected. Wire: none expected. diff --git a/quest/m3/release/cut.md b/quest/m3/release/cut.md index 042eb7e03fe..747b4e9c22e 100644 --- a/quest/m3/release/cut.md +++ b/quest/m3/release/cut.md @@ -31,7 +31,10 @@ compatibility, draft, package, and all-language interop checks, and verify publication outcomes. Separate artifact install proofs and package-specific tags follow this cut. This planning PR cuts and publishes nothing. After publication, re-measure the planned wire-compat breaks against the -published versions, as RELEASE.md's release procedure describes. +published versions, as RELEASE.md's release procedure describes. Until the +[planned-break remeasure](/quest/m0/interop-green/planned-break-remeasure.md) +moves that step into the release pipeline (decided 2026-11-11), run it by hand; +the cut does not wait on that quest (decided 2026-11-12). This cut carries stats epochs, so it drops [#4810](https://github.com/moq-dev/moq/pull/4810)'s release-only wall-clock