Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 100 additions & 4 deletions quest/m0/interop-green/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [XS] Interop and Video pass on main
# Interop and Video pass on main

## Goal

Expand All @@ -10,12 +10,32 @@ so it only needs to stay green.

## Waiting

- moq-dev/moq: a scheduled Interop run on `main` passes. Check with
`gh run list --repo moq-dev/moq --workflow interop.yml --event schedule`.
Triage any red run, then delete this quest once one passes.
- moq-dev/moq: the next scheduled Interop run on `main` passes, once the
[resumed-audio child](/quest/m0/interop-green/resume-sync.md) lands
([moq-dev/moq#8572](https://github.com/moq-dev/moq/pull/8572), the max-age
fix, merged 2026-11-12). Check
with `gh run list --repo moq-dev/moq --workflow interop.yml --event schedule`,
then the run's STEPS summary. The JS route port that [moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290)
waited on landed in [moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443),
so the pass now runs the datagram step; the `replacement` case stays
quarantined by default. Decided 2026-11-09 (maintainer, AFK, accepting the
recommendation): close this quest on a pass, which the maintainer set with
those cases skipped; the pass that closes it now runs them. Decided
2026-11-12 (maintainer): this epic also waits for its
[planned-break remeasure](/quest/m0/interop-green/planned-break-remeasure.md)
child, so the pass closes the epic only once that child lands too. Until then,
a red run whose only failure is a stale planned break is fixed by hand: run
`just test wire-compat --remeasure` and land the rewritten
`test/interop/compat/planned-breaks.json`, as RELEASE.md's release
procedure describes.

## Plan

Decided 2026-11-12 (maintainer), since green scheduled Interop is this epic's goal and stale pins caused four of the last six red runs:

- ✅ the release-remeasure pipeline is a child quest of quest/m0/interop-green/ (rejected: quest/m3/release/; drop the follow-up). See [planned-break remeasure](/quest/m0/interop-green/planned-break-remeasure.md).
- ✅ the release cut keeps its manual post-publish `just test wire-compat --remeasure` step until that child lands, and does not require it (rejected: block the cut on the new quest; accept a red Interop run per release).

Decided 2026-10-27 (maintainer), from an external release-to-main audit
and a triage of Interop runs
[40517499786](https://github.com/moq-dev/moq/actions/runs/40517499786),
Expand Down Expand Up @@ -102,4 +122,80 @@ moq-watch's `buffer`, which
`lookahead`. Fixed in this quest's PR: the case tests `lookahead`, and a new
check confirms `buffer` reports its rename and recovers once removed.

Scheduled run
[42491651687](https://github.com/moq-dev/moq/actions/runs/42491651687)
(2026-11-08, at e2271bad9c) failed two steps, and neither is a flake. Both come
from the JS lite-07 route wire, which Rust moved to but JS has not ported, so
they were owned by the JS route port, which landed as
[moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443), not this quest:

- Datagram tracks: `datagram_tracks_javascript` times out in its first case
(relay false, JS publishes false). It fails every time on main's content
(`just test datagram-tracks`), not under load.
- Media output and lifecycle: chromium's `replacement` case crashes both
moq-lite-07-wip pages with `page crashed`. It fails every time on main's
content (`just test media --media-cases replacement`), though it passed on an
older base.

[moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290) skipped the
datagram step and quarantined `replacement` by default, which this quest's
triage confirms. On main's content, `just test media` passes; the datagram test
still fails until the port lands. No new quest or retry is needed here.

Scheduled run
[42673842120](https://github.com/moq-dev/moq/actions/runs/42673842120)
(2026-11-09) and
[42851744235](https://github.com/moq-dev/moq/actions/runs/42851744235)
(2026-11-10) failed Harness regressions and Released wire compatibility. The
staging fixture's fake `cargo` lacked `tree`, which `test/interop/compat.sh`
calls, so its test exited 2; [moq-dev/moq#8455](https://github.com/moq-dev/moq/pull/8455)
stubs it. The planned break's pin, moq-cli 0.14.14, went stale when 0.14.15
shipped.

Scheduled run
[43052960736](https://github.com/moq-dev/moq/actions/runs/43052960736)
(2026-11-11, at a4bc922c6) failed the same two steps. The invalid-attribute
browser case restored `lookahead` to the number `0` after its `try` block,
which the setter refuses, so the evaluate threw.
[moq-dev/moq#8512](https://github.com/moq-dev/moq/pull/8512) passes the page's
own Duration and remeasures the pin to moq-cli 0.14.16.

`main` at d3b6c2573 has both fixes, and every pin matches the latest release of
its package (moq-cli 0.14.16, moq-relay 0.17.16, @moq/net 0.4.7). The stale pin
is the recurring cause: each published release of a pinned package turns the
next scheduled run red until someone remeasures.

Scheduled run
[43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067)
(2026-11-12, at e9c6df852) failed three steps, each with its own cause:

- Optional publisher retention: `max_age_relay_javascript` fails on lite-07
track 1, which reads a max age of 0 where the test expects none.
[moq-dev/moq#8443](https://github.com/moq-dev/moq/pull/8443) added
`moq-lite-07-wip` back to `max_age_relay_javascript` in `max_age.rs`, but the
JS subscriber kept its moq-transport-only expectation of no max age; lite
carries it, as the Rust side of the test expects.
[moq-dev/moq#8572](https://github.com/moq-dev/moq/pull/8572) fixes the
expectation, and is open.
- Interop (full matrix): `go -> python` fails with `audio gap after 648 samples`
at 2 s of 30 s. It passed once locally, and 8 more local runs of
`--publishers go --subscribers python,rust,go` passed every cell. A lead, not
checked: [moq-dev/moq#8531](https://github.com/moq-dev/moq/pull/8531) now waits on
a missing group in moq-audio only for the playout delay, which is small at a
cold start, so a reordered group early on reads as a gap. No open PR names this cell;
[moq-dev/moq#8587](https://github.com/moq-dev/moq/pull/8587) covers `python -> go`.
Decided 2026-11-12 (maintainer): the cell moves to
[test flakes](/quest/m0/test-flakes-2/interop-go-python.md), which checks
#8531 first.
- Media output and lifecycle: after the pause case resumes, the audio trails the
picture by about 1.7 s. [moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507)
keeps audio that trails the clock up to its 2 s stretch bound and plays it
faster, and a resume keeps the ring it had. Fixed by the child
[resumed audio snaps to the clock](/quest/m0/interop-green/resume-sync.md),
decided 2026-11-12 (maintainer) to snap to the live picture rather than
stretch the lag back.

The 11-11 run, before #8507, passed all three steps.

Public API: none. Wire: none.

38 changes: 38 additions & 0 deletions quest/m0/interop-green/planned-break-remeasure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# [M] Re-measure planned breaks in the release pipeline

## Goal

Each release that publishes a package a planned break pins (moq-cli,
moq-relay, `@moq/net`) re-measures those pins against the published versions
and lands the rewritten `test/interop/compat/planned-breaks.json` on `main`
with the release. No scheduled Interop run goes red on a stale pin, and no one
runs the remeasure by hand.

## Plan

Decided 2026-11-11 (maintainer), after moq-cli 0.14.15 and 0.14.16 each made
the `moq-lite-07-wip` pin stale and the scheduled Interop runs on Nov 9 to 11
went red on it:

- ✅ the release pipeline runs `just test wire-compat --remeasure` and lands the rewritten `test/interop/compat/planned-breaks.json` with the release (rejected: keep the manual post-publish step; warn on stale pins in scheduled runs).

Decided 2026-11-12 (maintainer), since stale pins caused four of the last six red scheduled Interop runs. This quest is a child of the [Interop epic](/quest/m0/interop-green/README.md):

- ✅ the release-remeasure pipeline is a child quest of quest/m0/interop-green/ (rejected: quest/m3/release/; drop the follow-up).
- ✅ the release cut keeps its manual post-publish `just test wire-compat --remeasure` step until this quest lands, and does not require it (rejected: block the cut on this quest; accept a red Interop run per release).

The job runs after the publish jobs for the three packages the planned-breaks
file pins, in the workflows that publish from `release` (`.github/workflows/release-*.yml`).
It re-measures the pins against the versions just published and lands the
rewritten file on `main`, which RELEASE.md's release procedure does by
hand today. Check `main`'s ruleset first: where it requires a pull request, the
job opens one. Its write credentials go in an environment limited to protected
branches.

Verify it by running the job on its own against the current pins (moq-cli
0.14.16, moq-relay 0.17.16, `@moq/net` 0.4.7). It should reproduce the file
with no diff.

The same PR updates RELEASE.md's release bullet, which says to run the
remeasure by hand, to name the job. The nightly wire-compat check keeps
running as it does now.
26 changes: 26 additions & 0 deletions quest/m0/interop-green/resume-sync.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# [S] Resumed audio snaps to the clock

## Goal

A viewer that pauses and resumes hears its audio in sync with the picture from the first frame, as it did before [moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507). So `just test media`'s pause and rejoin steps pass on every run, and a resumed viewer is not seconds behind for the stretch to catch up. Audio that falls behind because of network delay still catches up by playing faster; that is #8507's stretch and stays.

Not in scope: the JS max-age expectation ([#8572](https://github.com/moq-dev/moq/pull/8572)) and the `go -> python` media cell.

## Plan

Found by the triage of scheduled Interop run [43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067) (2026-11-12, head e9c6df852). Its `Media output and lifecycle` step failed on the pause case: `audio tone: after resume: 70/81 samples` in CI, and locally `audio/video sync: after resume: 0/79 samples within 200ms, median skew 600ms`. The 11-11 run, before #8507, passed that step.

Cause: #8507 keeps audio that trails the shared clock by up to the stretch's 2 s bound and plays it faster to catch up (`js/watch/src/audio/ring-buffer.ts` `read`, bound from `js/watch/src/audio/latency.ts` `bound`). Before it, the ring skipped whatever trailed the clock. A return from pause resubscribes to the same broadcast instance, so `js/watch/src/audio/decoder.ts` keeps the predecessor's ring without a join. The resumed picture lands on the live edge, and the audio that trails it by about 1.7 s is stretched back over about 5 s.

Evidence, `just test media --media-cases pause` in `test/interop/clients/js/media.ts`:

- `main` at 18fc33ecc: 3 of 4 runs fail the resume sync (median skew 200 ms, or 0 of 79 in sync).
- Parent of #8507 (ac73d9e0b): passes, median skew 0 ms (one run). On a host at load 30, four full `just test media` runs there passed 1, and the other 3 failed only `audio tone` (67 to 70 of 81 samples, the CI signature), never sync; the 11-11 head a4bc922c6 passed 4 of 6, failing the same way. #8507 itself (4262a0691) failed 4 of 4 on sync, median skew 400 to 600 ms after resume. So the CI step can stay red under load after this fix, from a tone flake older than #8507.
- Experiment, not committed: a same-instance resubscribe calls the decoder's `reset()`, so the ring joins the clock again. 4 of 4 runs pass, median skew 0 ms. One full `just test media` run also passed the pause and rejoin steps, then failed the later `live incumbent fallback` case, which was not investigated.

Decided 2026-11-12 (maintainer): ✅ snap to the live picture. Re-join the ring when a viewer returns from absence on the same instance, so resumed audio lands on the clock the way a tune-in does, and keep the 2 s bound for stalls. Rejected: stretch catch-up with #8507's time-stretch, which plays the lag back over about 5 s and costs the lip sync that #8031 fixed; relaxing the resume assertion to match it, since `js/watch/AGENTS.md` says to judge a player change by its lag first. Check it with `just test media --media-cases pause`.

Open for the implementer:

- Scope the re-join to a return from absence. The handover path (`#handover` in `decoder.ts`) also inherits the ring for a rendition swap and a fallback, which should keep their seamless handover. Check the fallback case.
- Native parity: `moq play` pauses through `rs/moq-audio`. Check whether its ring inherits the same way; `just test media` does not cover it.
8 changes: 8 additions & 0 deletions quest/m0/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,4 +280,12 @@ its third failure since 2026-10-16 and its first after
[moq-dev/moq#8526](https://github.com/moq-dev/moq/pull/8526). It is
quarantined with `#[ignore]` until its child fixes it.

Added 2026-11-12 (maintainer, from the Interop triage of scheduled run
[43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067)): the
`go -> python` media cell, which failed `audio gap after 648 samples` once and
passed 8 local runs after it. Decided: ✅ add it here as the child
[interop's go -> python media cell](/quest/m0/test-flakes-2/interop-go-python.md),
not quarantined (rejected: a new quest after a second red run; quarantining the
cell). Check [moq-dev/moq#8531](https://github.com/moq-dev/moq/pull/8531) first.

Public API: none. Wire: none.
29 changes: 29 additions & 0 deletions quest/m0/test-flakes-2/interop-go-python.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# [S] Interop's go -> python media cell holds up

## Goal

The `go -> python` media cell of `just test interop --all` passes reliably,
fixed at its cause, never by a quarantine, a longer timeout, or a retry.

## Plan

Found 2026-11-12 in the Interop triage of scheduled run
[43260674067](https://github.com/moq-dev/moq/actions/runs/43260674067) (head
e9c6df852, the first run after #8455 and #8512). Its full-matrix step failed
`go -> python` with `audio gap after 648 samples` at 2 s of 30 s. It passed
once locally, and 8 more local runs of `--publishers go --subscribers
python,rust,go` passed every cell. No open PR names this cell.

Decided 2026-11-12 (maintainer): ✅ track it as this child of test flakes
(rejected: a new quest after a second red run; quarantining the cell).

Check first: [moq-dev/moq#8531](https://github.com/moq-dev/moq/pull/8531)
(`fix(hang,moq-mux): wait on a missing group for the delay, not the max`) merged
on 2026-11-12 and is in the failing run's head. A lead, not checked: it waits on
a missing group only for the playout delay, which is small at a cold start, so a
group that arrives out of order early on reads as an audio gap.

Reproduce with `just test interop --publishers go --subscribers python` and the
interop logs kept. Confirm or rule out #8531 first, then fix the cause.

Public API: none expected. Wire: none expected.
5 changes: 4 additions & 1 deletion quest/m3/release/cut.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,10 @@ compatibility, draft, package, and all-language interop checks, and verify
publication outcomes. Separate artifact install proofs and package-specific
tags follow this cut. This planning PR cuts and publishes nothing.
After publication, re-measure the planned wire-compat breaks against the
published versions, as RELEASE.md's release procedure describes.
published versions, as RELEASE.md's release procedure describes. Until the
[planned-break remeasure](/quest/m0/interop-green/planned-break-remeasure.md)
moves that step into the release pipeline (decided 2026-11-11), run it by hand;
the cut does not wait on that quest (decided 2026-11-12).

This cut carries stats epochs, so it drops
[#4810](https://github.com/moq-dev/moq/pull/4810)'s release-only wall-clock
Expand Down