Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions quest/m0/ci/c-fixtures-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# [XS] The merge gate compiles the C fixtures against the generated header

## Goal

A PR that changes a libmoq C function's signature fails the merge gate if any `rs/libmoq/c-tests/*.c` still calls the old shape, instead of the break first showing up in Nightly.

## Plan

Found by the 11-17 Nightly triage: [moq-dev/moq#8999](https://github.com/moq-dev/moq/pull/8999) merged 3 minutes after [moq-dev/moq#8996](https://github.com/moq-dev/moq/pull/8996) added `moq_consume_catalog`'s `delay_us` argument, and `abort-unwanted.c` kept the 3-argument call until [moq-dev/moq#8941](https://github.com/moq-dev/moq/pull/8941). Nothing in the gate compiles those files.

Recommended: `cc -fsyntax-only` each fixture against the generated header in the gate, about a second. `test/smoke/clients/c/subscribe.c` builds against the published package on purpose, so it stays out.

The Swift break the same night (`SmokeTests.swift`, [moq-dev/moq#8906](https://github.com/moq-dev/moq/pull/8906) merging after [moq-dev/moq#8948](https://github.com/moq-dev/moq/pull/8948)) stays nightly-only: Swift doesn't build on Linux runners.

Public API: none. Wire: none.
25 changes: 25 additions & 0 deletions quest/m0/interop-green/media-resume-delay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# [L] A viewer's resume doesn't freeze playback on a late video burst

## Goal

The interop `media` step passes under CPU load: after a pause and resume, or a codec rename, audio and video play without a freeze inside the measurement window, in the scheduled Interop run and Nightly's `Media (firefox player)` job. Both have failed every day since 2026-11-12.

## Plan

Both fail `FAIL audio tone: after resume: N/81 samples carried the fixture tone` in "media output + lifecycle". The `remote error: 1`/`54`, `stream aborted` and `fetch error … not found: group 10` lines are the expected cancels of pause and rename, also present in passing runs.

The first bad commit was [moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507) (`4262a0691e`; after resume, audio caught up by playing up to 1.6x). That looks fixed by [moq-dev/moq#8675](https://github.com/moq-dev/moq/pull/8675): the pause case alone under load now passes 4/4.

What remains on `main` fails about 25% under load: after resume, the first frames of a new video group arrive together 400–900 ms late, the jitter estimator (`js/hang/src/container/jitter.ts`) counts that as path delay, and the auto delay jumps from 162 ms to 700–880 ms, so audio and video freeze about 0.5 s. The last good commit showed the same bursts, smaller (180–280 ms), so this may be old flakiness made visible.

Steps:

1. Find whether the burst is real (the Chromium publisher's keyframe encode stalling under load) or a re-subscription artifact the estimator should treat as join backlog. Its join rule excludes only frames produced before the first arrival. Log per-group arrival at the publisher, relay and player.
2. Decide what a large auto-delay rise does during playback: today it pads silence and freezes video for the whole step, while #8507 meant the ring to slow onto a rise. Coordinate with [audio playout](/quest/m2/audio-playout/README.md), which owns that rule.
3. Explain the 11-17 CI trace: audio reports timestamp 0 and is silent 0.6 s while video plays, then video freezes and jumps 600 ms. Why does `measurePlayback`'s "playback starts" wait pass while the ring has no audio yet?

Reproduce from `test/`: `just media --media-cases rename,pause` under `taskset -c 0-3` with two busy loops on the same cores, and `HARNESS_RUNS` set to a short path (pulse.py's Unix socket path is too long from a worktree's `.tmp`). Without load it passes almost every time (12/13).

Fix the root cause; no retries or looser thresholds.

Public API: none. Wire: none.
15 changes: 15 additions & 0 deletions quest/m0/lite07-stale-reset.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# [L] A lite-07 publisher resets a stale group's stream when it expires

## Goal

A lite-07 publisher aborts every group stream whose group expired, including one whose end it already wrote, so stale video can't fill the connection's send window and stall the streams behind it. The scheduled TS workflow's validation passes again: no audio expires on its 7 Mb/s path.

## Plan

Caused by [moq-dev/moq#8908](https://github.com/moq-dev/moq/pull/8908) (`fb443a5ac7`), found by bisecting the TS validation failure (run [44135366690](https://github.com/moq-dev/moq/actions/runs/44135366690), still failing on 11-18). Under lite-07, a stale group stream whose GROUP_END was already written is no longer reset when the group expires (`poll_stopped` and `closes_expired` in `rs/moq-net/src/lite/publisher.rs`). Stale video groups then fill moq-quic's 8 MiB connection send window, the control stream sending SUBSCRIBE_SKIP stalls 1–10 s behind them, and audio expires: 60–100 s lost out of 120. Removing the `!has_group_end()` guard brings audio loss to zero.

Decided 2026-11-18 (maintainer, /quest-plan): ✅ reset stale groups again, as lite-06 did, in Rust, JS, the draft's Expiration section, and the `verify-subscribe` model (rejected: keep the wire rule and let higher-priority streams get past a full send window in moq-quic, since stale video would still use bottleneck bandwidth and other QUIC stacks would stall the same way; rejected: measure both first). This relaxes #8908's rule that each group is named only once; lite-07 is unpublished `-wip`, so the wire change is free.

Decided 2026-11-18 (maintainer): moq-net never finishes a stream or producer and forgets it; something must abort it eventually. The rule is in `rs/moq-net/AGENTS.md`. As part of this quest, audit moq-net (Rust and `js/net`) for other streams or producers that are finished and then never aborted, such as fetch, datagram, and announce streams, and fix each one found or plan it as a follow-up.

Public API: none. Wire: lite-07-wip expiry resets a finished group's stream again; update the draft in the same PR.
21 changes: 21 additions & 0 deletions quest/m0/nightly/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,24 @@ disk ([moq-dev/moq#9033](https://github.com/moq-dev/moq/pull/9033)), benches
[moq-dev/moq#9021](https://github.com/moq-dev/moq/pull/9021),
[moq-dev/moq#9022](https://github.com/moq-dev/moq/pull/9022)), and mutants
([moq-dev/moq#9019](https://github.com/moq-dev/moq/pull/9019)).

Decided 2026-11-18 (maintainer, /quest-plan), from the triage of runs
[44127032469](https://github.com/moq-dev/moq/actions/runs/44127032469) and
[44342367698](https://github.com/moq-dev/moq/actions/runs/44342367698): four
more children (JS resume window, audio-quality p50 ceilings, publisher drop
cost, resume mutants). Owned elsewhere: the firefox player by
[media resume delay](/quest/m0/interop-green/media-resume-delay.md), OBS by
[moq-dev/moq#9016](https://github.com/moq-dev/moq/pull/9016), and fixes in PRs
from the same triage: benches
([moq-dev/moq#9089](https://github.com/moq-dev/moq/pull/9089),
[moq-dev/moq#9090](https://github.com/moq-dev/moq/pull/9090),
[moq-dev/moq#9098](https://github.com/moq-dev/moq/pull/9098),
[moq-dev/moq#9099](https://github.com/moq-dev/moq/pull/9099)), features
([moq-dev/moq#9091](https://github.com/moq-dev/moq/pull/9091),
[moq-dev/moq#9092](https://github.com/moq-dev/moq/pull/9092)), quic-interop
([moq-dev/moq#9096](https://github.com/moq-dev/moq/pull/9096),
[moq-dev/moq#9097](https://github.com/moq-dev/moq/pull/9097)), fuzz
([moq-dev/moq#9105](https://github.com/moq-dev/moq/pull/9105),
[moq-dev/moq#9106](https://github.com/moq-dev/moq/pull/9106)), and route-sweep
([moq-dev/moq#9108](https://github.com/moq-dev/moq/pull/9108)). The JS audit's
picomatch advisory was withdrawn upstream.
17 changes: 17 additions & 0 deletions quest/m0/nightly/audio-quality-p50.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# [S] Browser audio-quality lag ceilings match what the player corrects

## Goal

The Nightly `Audio quality` job's browser `av_lag_ms_p50` ceilings sit outside the lag the player deliberately leaves alone, so a row fails only when audio really drifts. It failed on 11-13, 11-14, 11-15 and 11-17 on `chromium-aac-44100-congestion: av_lag_ms_p50 2.7 > 1.3`.

## Plan

Cause: the 1.3 ms ceiling was set provisionally in [moq-dev/moq#8459](https://github.com/moq-dev/moq/pull/8459) from a few developer-machine runs. [moq-dev/moq#8722](https://github.com/moq-dev/moq/pull/8722) then made the player leave lag within ±5 ms (half a 10 ms stretch step) uncorrected, so where p50 settles inside that window is chance (runner: 0.4–18.2 ms across 11-13..11-17). #8722 re-measured the replay rows but left the browser rows to the runner.

Decided 2026-11-18 (maintainer, /quest-plan): ✅ set the browser p50 ceilings to the ±5 ms window now, then tighten from runner artifacts (`gh run download <id> -n audio-quality-run`) once a week of nights since #8722 exists (rejected: wait for a week of runner data first and apply `budgets.json`'s worst night + 25% + 20 ms rule, about 24 ms, leaving Nightly red meanwhile).

Also look at the Opus browser rows: several rest 7–11 ms behind on the runner, outside the ±5 ms window, and Opus congestion hit exactly its 8.3 ceiling on 11-17. Decide from the data whether that is a playout bug for [audio playout](/quest/m2/audio-playout/README.md) or a ceiling to set.

A developer machine can't reproduce the runner's numbers (every browser row sits about 30 ms behind locally, as `budgets.json` notes), so use runner artifacts.

Public API: none. Wire: none.
20 changes: 20 additions & 0 deletions quest/m0/nightly/js-resume-window.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# [M] A JS resumed reader hands out only groups in its window

## Goal

`js/net/src/resume.ts` hands a reader only groups inside the ranges it resolved, as Rust's `Reader::poll_lowest` does ("the window decides"), so the Nightly `resume-model` property `readers match the logical track` passes. Done when seeds 1–5 and 1218508351 pass at 4096 cases, plus `just test resume-model`.

## Plan

Caused by [moq-dev/moq#9027](https://github.com/moq-dev/moq/pull/9027) (`0c14c87c49`): it moved JS to time anchors to match Rust's [moq-dev/moq#8638](https://github.com/moq-dev/moq/pull/8638) but left out Rust's guard. A copy that doesn't yet hold the group a range starts at resolves the start to an earlier group, and JS hands that group out. The model passes at #9027's parent on four seeds × 4096 cases and fails at #9027 on all of them, including Nightly's counterexample (seed 1218508351, run [44342367698](https://github.com/moq-dev/moq/actions/runs/44342367698)): `reader 0: group 0 outside ranges 1,4 (resolved from 1) and bounds undefined`. Rust is correct: the failing cases, ported to Rust's regression test, pass.

Partial work: branch `fix/js-resume-window-wip` (`0f1b73d365`) adds the guard and a regression case. It passes the Nightly seed and seeds 3 and 5; seeds 1, 2 and 4 still fail with groups missing or leaking. An earlier attempt broke because `#placed` forces copies to resolve early, and the model's own `servingResolved` check calls it too.

Recommended: port Rust's reader window properly:

- a reader-owned window that keeps its first placement, built from each copy's resolution only where a failover cap doesn't empty that range, and filled from each copy's unclamped resolution when ranges are replaced;
- the guard, and Rust's `owes_below` check for ordered readers;
- `#backfillGaps` starting from the window's floor;
- `#placed` no longer forcing copies to resolve early.

Public API: none. Wire: none.
13 changes: 13 additions & 0 deletions quest/m0/nightly/publisher-drop-cost.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# [L] A dropped group costs a subscription O(1), not O(ranges)

## Goal

Dropping a group from a subscription with many ranges costs about the same as with one, so moq-net's `publisher` bench `wake/8192` finishes in seconds instead of running past 600 s and timing out the Nightly `Rust bench targets` job (55-minute limit). A new bench sweeps drop cost over ranges × drops, so a regression shows in Nightly's measured coverage.

## Plan

Caused by [moq-dev/moq#8638](https://github.com/moq-dev/moq/pull/8638): each dropped group makes the subscription rebuild state over every range, about 2.4 ms per drop at 8193 ranges. Four places rebuild instead of updating incrementally, in `rs/moq-net/src/model/track.rs` and `subscription/{window,runs}.rs`: the window, what is owed, the misses check, and the servable set. Make each incremental, and keep the JS counterpart in step where it has the same shape.

Decided 2026-11-18 (maintainer, /quest-plan): ✅ no stopgap (rejected: having the bench fixture do all its drops before one step, which takes 6 ms but hides the bug from the only bench that catches it). The Rust bench job stays red until this lands.

Public API: none. Wire: none.
21 changes: 21 additions & 0 deletions quest/m0/nightly/resume-mutants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# [M] No resume or rekey mutant survives the Nightly mutation run

## Goal

The Nightly `mutants` shards pass: every mutant of `rs/moq-net/src/model/resume.rs` and of `rekey` in `rs/moq-net/src/route/path.rs` fails a test, and a night's shards finish instead of being cancelled.

## Plan

[moq-dev/moq#9019](https://github.com/moq-dev/moq/pull/9019) killed the previous survivors; new ones followed later changes. Survivors seen:

- 11-16 ([43909101837](https://github.com/moq-dev/moq/actions/runs/43909101837)): 5 in `resume.rs` `Reader::window`, 1 in `route/path.rs` `rekey`.
- 11-17 ([44127032469](https://github.com/moq-dev/moq/actions/runs/44127032469)): all shards cancelled, after more missed in `resume.rs`: `Subscriber::holds`, `previous_start`, `skipped_below`, `largest_finished`, and the `Route::answering` guard.
- 11-18 ([44342367698](https://github.com/moq-dev/moq/actions/runs/44342367698)): shards 2–5 failed; read them for the current list.

Find first why the 11-17 shards were cancelled. Then, for `Reader::window`, add unit tests where a copy's failover cap equals, falls just below, or falls just above a range's start or end, and assert the settled window; add direct tests for the other listed methods and `rekey`. Re-run with `cargo mutants -f <file>`. A mutant that changes no behaviour is excluded with its reason, not tested.

Public API: none. Wire: none.

## Related

- [Mutants of the other modelled code](/quest/m4/mutants-modelled.md) - widens the run to more files
13 changes: 13 additions & 0 deletions quest/m2/cluster-routes/dead-instance-flip.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# [M] A dead instance's stale path never wins a lite-07 route

## Goal

A reader of a path whose route flips C → B → C never sees B when B is an instance already known dead, whenever it polls, so the route-sweep invariants hold with no window for a step back to a dead instance.

## Plan

Found by [moq-dev/moq#9108](https://github.com/moq-dev/moq/pull/9108), which fixed the Nightly `mesh_sweep` failures: when the route flipped C → B → C before the reader polled, the reader got a second restart to C. #9108 merges a restart back to the instance the reader already shows into an Update, but a reader that polls mid-flip can still see B, so the sweep can still flake.

The flip became reachable with the lite-07 route rule in [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987). Decide whether that rule should stop a dead instance's stale path from winning at all, which removes the flip, rather than hiding it at the reader. Recommended: stop it at the rule if the route model (`/verify-failover`, `/verify-warm`) shows no case where the stale path must win; check the draft's route section for the same.

Public API: none. Wire: possibly the lite-07-wip route rule; update the draft in the same PR if so.
1 change: 1 addition & 0 deletions rs/moq-net/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ The wire layer: generic over the transport and media-agnostic.
- Nothing is local-only: every field and behavior survives the wire, so it means the same through a relay as in-process. Local-only knobs belong above moq-net (moq-mux, hang).
- Local and remote paths behave the same: a local close, cancel, or error reaches readers exactly as the remote one would, never as a cleaner end.
- Closes never cascade: aborting a group leaves its track, broadcast, and siblings alone.
- Never finish a stream or producer and forget it: something, such as expiry, a cancel, or the session's close, must abort it eventually, or a stale stream holds connection credit and stalls the streams behind it.
- A subscription stays on the instance it started on while that instance has a route, even after a newer route wins the path. When none is left, it ends with an error, never a clean end, and never stitches onto another instance. A clean end means only a deliberate publisher finish. Apps restart on announcements, never on request errors. See "Stream instances" in `doc/concept/moq-lite.md`.
- The cache is only an optimization: a relay behaves the same warm or cold, reporting upstream state (such as the largest group) rather than what it holds.
- A relay never blocks an answer on other nodes finishing something; it reports what it knows now. One exception: a track listing's synced mark waits for the upstream's, while its tracks still flow live.
Expand Down