From 7529e3d9dbc9e4e963f69c21baa13f6a1f034f99 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 18:17:09 -0800 Subject: [PATCH 1/7] quest: plan the incremental relay receive path and Key.sign's one spelling; fix two dependency notes; unused surface is deleted Co-Authored-By: Claude Opus 5.5 --- .../native-rendition-delay.md | 2 +- quest/m2/api-shape/README.md | 12 +++++-- quest/m2/api-shape/auth-sign.md | 25 +++++++++++++++ quest/m2/cluster-routing/routes/README.md | 1 + .../cluster-routing/routes/relay-receive.md | 32 +++++++++++++++++++ quest/m2/tokio-resolved-options.md | 7 ++-- 6 files changed, 73 insertions(+), 6 deletions(-) create mode 100644 quest/m2/api-shape/auth-sign.md create mode 100644 quest/m2/cluster-routing/routes/relay-receive.md diff --git a/quest/m1/audio-jitter-target/native-rendition-delay.md b/quest/m1/audio-jitter-target/native-rendition-delay.md index f7b4ae4cee3..da8e90055c5 100644 --- a/quest/m1/audio-jitter-target/native-rendition-delay.md +++ b/quest/m1/audio-jitter-target/native-rendition-delay.md @@ -13,7 +13,7 @@ behind it. Planned 2026-11-08 (maintainer, approved follow-up of [moq-dev/moq#8272](https://github.com/moq-dev/moq/pull/8272), which gives moq-mux's playout sync per-track targets and the largest-target delay). Milestone: -m1, an additive fix to shipped playback; start after #8272 merges. +m1, an additive fix to shipped playback. #8272 has merged. On `main`, nothing under `rs/moq-cli/src/play/` reads a rendition's `delay`; the browser does (`js/watch/src/video/playhead.ts`, diff --git a/quest/m2/api-shape/README.md b/quest/m2/api-shape/README.md index 0d12fbe06e0..714fc457569 100644 --- a/quest/m2/api-shape/README.md +++ b/quest/m2/api-shape/README.md @@ -29,17 +29,22 @@ Added 2026-11-07 (maintainer, /quest-plan from the 2026-11-07 API sweep of open what later quests grow (`#[non_exhaustive]` on structs and enums m3 to m5 add to), shapes those quests would otherwise break, and this epic's own work: before the cut, run a "public item used only by its definition" -search over the published crates and remove each hit, starting with +search over the published crates and JS packages and delete each hit, starting with `moq_tokio::Server::serve_both`, `hang::Catalog::to_json_pretty`, and `moq_audio::Format::{is_planar, is_float}`, the shortlist of [the dead code audit](/quest/m3/audit-dead-code.md). The audit itself stays in m3 for private code. +Decided 2026-11-09 (maintainer, on [moq-dev/moq#8302](https://github.com/moq-dev/moq/pull/8302): +"delete anything unused in general"): ✅ unused public surface is deleted, +not made private, and the pre-cut sweep covers the JS packages too +(rejected: a separate audit quest, since this epic already owns the sweep). + Rules every child follows: - Removal beats addition. A public item with no caller in this repository - or a known downstream application goes private, unless an m1 or m2 quest plans a - caller. + or a known downstream application is deleted, unless an m1 or m2 quest plans a + caller; it goes private only when internal code still uses it. - No serde aliases, refusals, or shims for a spelling `release` never shipped. - Each child lists its public API and wire impact in its PR, and its Impact @@ -59,4 +64,5 @@ Rejected in planning: - [Crates re-export no whole crate](/quest/m2/api-shape/reexports.md) - no moq crate re-exports a whole crate, moq's own included, so a dependency bump is no moq break, and moq-mux's container `Guard` gets a name - [Media crates' small surface fixes](/quest/m2/api-shape/media.md) - exporters, the container consumer, catalog color range, rate grant, moq-stats, JSON, and MSF each have one path and one name - [JS packages' small surface fixes](/quest/m2/api-shape/js.md) - track info, connection, qmux, watch, publish, ui, video, boy, and auth exports each have one path and one name +- [JS Key.sign takes exactly Claims](/quest/m2/api-shape/auth-sign.md) - `@moq/auth` signs only `Claims`, with no legacy `put`/`get` input, like Rust - [Bindings' small surface fixes](/quest/m2/api-shape/bindings.md) - Go's read cancel, JSON producer construction, and the Python wheel's test module diff --git a/quest/m2/api-shape/auth-sign.md b/quest/m2/api-shape/auth-sign.md new file mode 100644 index 00000000000..feb70a43a76 --- /dev/null +++ b/quest/m2/api-shape/auth-sign.md @@ -0,0 +1,25 @@ +# [XS] JS Key.sign takes exactly Claims + +## Goal + +`@moq/auth`'s `Key.sign` takes exactly `Claims`, as Rust's +`moq_auth::Key::sign(&Claims)` does: its input no longer accepts the legacy +`put`/`get` role spelling, so each role has one spelling. Tokens it writes +are unchanged: when the legacy form says the same thing, it is still written +that way so older verifiers accept it, and `verify` still reads legacy +tokens. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete follow-up of +[moq-dev/moq#8296](https://github.com/moq-dev/moq/pull/8296), which settles +`Key.sign`'s caps spelling). ✅ Its own small breaking PR in this epic +(rejected: folding it into #8296 mid-iterate; dropping it). + +`sign` in `js/auth/src/key.ts` takes `z.input` plus +the rate fields, so its type still admits whatever the schema parses. Narrow +the parameter to `Claims` (as `verify` returns it), keep the scope check on +the parsed value, and update `js/auth/api/*.api.md`. + +Public API: `@moq/auth` `Key.sign` no longer accepts `put`/`get` input. +Migrate: pass `publish`/`subscribe` roles. Wire: none. diff --git a/quest/m2/cluster-routing/routes/README.md b/quest/m2/cluster-routing/routes/README.md index 3200e53854b..b18b8ec201f 100644 --- a/quest/m2/cluster-routing/routes/README.md +++ b/quest/m2/cluster-routing/routes/README.md @@ -537,6 +537,7 @@ benchmarks. ## Required - [Route session integration](/quest/m2/cluster-routing/routes/session.md) - one router per origin node drives the origin and every lite-07 announce stream +- [A relay folds received routes incrementally](/quest/m2/cluster-routing/routes/relay-receive.md) - a received route batch costs the claims it changes, not the origin's size - [Announcement availability](/quest/m2/cluster-routing/routes/availability.md) - announcements report availability, and a let-go is offline rather than END, in Rust and JS - [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - `js/net` encodes, decodes, and resolves the route layer - [Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) - forwarding fixtures ported and the model deleted, incident fixtures, end-to-end checks, and benchmarks diff --git a/quest/m2/cluster-routing/routes/relay-receive.md b/quest/m2/cluster-routing/routes/relay-receive.md new file mode 100644 index 00000000000..d663c3769df --- /dev/null +++ b/quest/m2/cluster-routing/routes/relay-receive.md @@ -0,0 +1,32 @@ +# [L] A relay folds received routes incrementally + +## Goal + +A relay's work per route batch it receives scales with the claims that +changed, not with the size of the origin they belong to. A benchmark swept +over both axes (claims per origin, claims changed per batch) shows the cost +flat in origin size. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete follow-up of +[moq-dev/moq#8293](https://github.com/moq-dev/moq/pull/8293)). Milestone: ✅ +m2, a child of this epic (rejected: m3, dropping it). #8293 makes the origin +node and each stream's presentation incremental. The receive side still +does linear work per change: + +- each received batch clones the heard view and merges the whole view + (`View::merge`, `retire`); +- `node::State::home` walks every shown claim of the identity. + +So one claim change on a large origin costs a pass over all of its claims at +every relay that hears it. Fold each batch into the held view by the claims +it names, and keep whatever `home` derives up to date per change instead of +recomputing it. #8293's benchmark covers only the origin; extend it to the +receiving relay, so both axes are swept per the repository's fan-out rule. + +Public API: none expected. Wire: none. + +## Required + +- [Route session integration](/quest/m2/cluster-routing/routes/session.md) - the incremental origin and per-stream presentation this builds on diff --git a/quest/m2/tokio-resolved-options.md b/quest/m2/tokio-resolved-options.md index b7fe9e8f198..559573160ba 100644 --- a/quest/m2/tokio-resolved-options.md +++ b/quest/m2/tokio-resolved-options.md @@ -16,8 +16,7 @@ resolve. Planned 2026-11-08 (maintainer, approved follow-up of [moq-dev/moq#8265](https://github.com/moq-dev/moq/pull/8265), which makes moq-net's reconnect policy take `Option` and keeps moq-tokio's config layer -on zero). Milestone: m2, since it breaks moq-tokio's public API; start after -#8265 merges. +on zero). Milestone: m2, since it breaks moq-tokio's public API. On `main`: `Backoff::max()` documents "zero leaves the delay uncapped" and `timeout()` "zero meaning unlimited" (`rs/moq-tokio/src/connection.rs`); @@ -31,3 +30,7 @@ other `Resolved` structs for the same pattern while there. Public API: moq-tokio's resolved fields and accessors become `Option`. Migrate: compare against `None` instead of `Duration::ZERO`. Wire: none. + +## Waiting + +- moq-dev/moq: [moq-dev/moq#8265](https://github.com/moq-dev/moq/pull/8265) merges, since this finishes the change it makes to moq-net. Check with `gh pr view 8265 --json state`; delete this section once it reads `MERGED`. From e4fa01945f480b0a84206aa7a673da78308da24e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 18:22:53 -0800 Subject: [PATCH 2/7] quest: plan four Nightly failures nothing owned Co-Authored-By: Claude Opus 5.5 --- quest/m0/nightly/README.md | 12 ++++++++ quest/m0/nightly/bench-regressions.md | 32 +++++++++++++++++++++ quest/m0/nightly/moxygen-conformance.md | 25 ++++++++++++++++ quest/m0/nightly/qmux-bench-peer.md | 23 +++++++++++++++ quest/m0/nightly/session-bench-handshake.md | 19 ++++++++++++ 5 files changed, 111 insertions(+) create mode 100644 quest/m0/nightly/bench-regressions.md create mode 100644 quest/m0/nightly/moxygen-conformance.md create mode 100644 quest/m0/nightly/qmux-bench-peer.md create mode 100644 quest/m0/nightly/session-bench-handshake.md diff --git a/quest/m0/nightly/README.md b/quest/m0/nightly/README.md index 7baee25f60d..3a82a9d0589 100644 --- a/quest/m0/nightly/README.md +++ b/quest/m0/nightly/README.md @@ -40,6 +40,18 @@ to the PR gate; the full run costs about 17 minutes per PR. Public API: none. Wire: none. +Decided 2026-11-09 (maintainer, /quest-complete), from the triage of runs +[42377867523](https://github.com/moq-dev/moq/actions/runs/42377867523) and +[42487375545](https://github.com/moq-dev/moq/actions/runs/42487375545): four +more children for the causes nothing owned. Dropped: one-night bench moves +near the paired threshold (re-check if one repeats), a one-off +`remote error: 52` in the browser bench smoke, and failures main or +[moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290) already fixes. + ## Required - [Nightly's macOS platform job passes](/quest/m0/nightly/macos.md) - `moq-audio`'s tests compile on macOS again and the websocket version test's timeout is fixed if it recurs +- [The qmux startup benchmark's fake peer sends transport parameters](/quest/m0/nightly/qmux-bench-peer.md) - Audio quality's WebSocket startup step stops timing out +- [The moq-net session bench completes moq-transport-22 at 16 broadcasts](/quest/m0/nightly/session-bench-handshake.md) - the bench workload stops panicking on a handshake timeout +- [Bisect the moq-audio bus and moq-mux parameter_sets regressions](/quest/m0/nightly/bench-regressions.md) - two real benchmark regressions are explained and fixed or accepted +- [Moxygen conformance cells pass again](/quest/m0/nightly/moxygen-conformance.md) - the first-object signal and integer extensions survive, or moxygen's divergence is skipped with an upstream link diff --git a/quest/m0/nightly/bench-regressions.md b/quest/m0/nightly/bench-regressions.md new file mode 100644 index 00000000000..4f66dcbfb98 --- /dev/null +++ b/quest/m0/nightly/bench-regressions.md @@ -0,0 +1,32 @@ +# [M] Bisect the moq-audio bus and moq-mux parameter_sets regressions + +## Goal + +Two benchmark regressions Nightly flagged are explained and either fixed or +accepted with a recorded reason: + +- 13 `moq-audio/bus` cases (`bus_minus`, `bus_mixes`, `bus_sources`), +16 to + +29% together, between `fbef872` and `b5805859`. +- `moq-mux/parameter_sets h264-split-pps/distinct`, +14% / +26% / +46% at + n = 16 / 256 / 4096, so the cost grows with n. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete from the triage of Nightly +runs [42377867523](https://github.com/moq-dev/moq/actions/runs/42377867523) +(the bus set, in "Rust supplemental benchmarks") and +[42487375545](https://github.com/moq-dev/moq/actions/runs/42487375545) (the +parameter sets, in "Rust shared")). Both look real: the bus cases moved +together, and the parameter_sets cost scales with n, which points at a +per-item cost that grew rather than noise. + +Suspects from the triage, to confirm by bisecting with the paired +comparison (`bench/README.md#paired-comparison`), not by reading: +[moq-dev/moq#8121](https://github.com/moq-dev/moq/pull/8121) (encoders take a +join) and [moq-dev/moq#8010](https://github.com/moq-dev/moq/pull/8010) +(comfort noise) for the bus; [moq-dev/moq#8140](https://github.com/moq-dev/moq/pull/8140) +and [moq-dev/moq#8199](https://github.com/moq-dev/moq/pull/8199) for the +parameter sets. Name the culprit PR under `## Caused by`. A regression that +buys correctness may stay, with the trade recorded in the PR. + +Public API: none expected. Wire: none. diff --git a/quest/m0/nightly/moxygen-conformance.md b/quest/m0/nightly/moxygen-conformance.md new file mode 100644 index 00000000000..e7ea1aa9536 --- /dev/null +++ b/quest/m0/nightly/moxygen-conformance.md @@ -0,0 +1,25 @@ +# [M] Moxygen conformance cells pass again + +## Goal + +Nightly's `ietf-conformance / moxygen` job passes. Today its cells fail with +`Missing Begins With First Object Signal for group=0 subgroup=0`, and with +`Extension Error=2 ... Expected size: 1 Actual size: 0` in the Extensions 1/2 +and Object/Group Counts 3/4 cells. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete from the triage of Nightly +runs [42377867523](https://github.com/moq-dev/moq/actions/runs/42377867523) +and [42487375545](https://github.com/moq-dev/moq/actions/runs/42487375545); +also red on 2026-11-06). ✅ Plan it (rejected: drop the lane), since it +keeps Nightly red, though moq-transport compatibility is low priority by the +repository's rules. + +Find which side is wrong before changing anything: our relay dropping the +subgroup's first-object flag and an integer extension's value, or moxygen's +expectations diverging from the draft we implement. Fix ours when it is +ours. When moxygen is wrong, record it upstream and skip just those cells +with a link, rather than adding cost to match it. + +Public API: none expected. Wire: moq-transport only, if ours is wrong. diff --git a/quest/m0/nightly/qmux-bench-peer.md b/quest/m0/nightly/qmux-bench-peer.md new file mode 100644 index 00000000000..e77df5d870d --- /dev/null +++ b/quest/m0/nightly/qmux-bench-peer.md @@ -0,0 +1,23 @@ +# [XS] The qmux startup benchmark's fake peer sends transport parameters + +## Goal + +Nightly's Audio quality job passes its "WebSocket fallback startup benchmark" +step: `just js qmux-bench` completes instead of failing with +`error: evaluate: Error: handshake timeout`. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete from the triage of Nightly +runs [42377867523](https://github.com/moq-dev/moq/actions/runs/42377867523) +and [42487375545](https://github.com/moq-dev/moq/actions/runs/42487375545)). +Failed both nights. + +The Bun server in `js/net/bench/qmux.ts` never sends QMux transport +parameters, and the JS session now waits for them before it opens. +[moq-dev/moq#8290](https://github.com/moq-dev/moq/pull/8290) fixes the same +fake-peer handshake in `js/qmux/bench/datagrams.ts`, but Nightly runs the +`js/net` copy. Make the fake peer send them the way the real server does, +ideally through the same helper as `datagrams.ts` so the two cannot drift. + +Public API: none. Wire: none. diff --git a/quest/m0/nightly/session-bench-handshake.md b/quest/m0/nightly/session-bench-handshake.md new file mode 100644 index 00000000000..2601a6e630e --- /dev/null +++ b/quest/m0/nightly/session-bench-handshake.md @@ -0,0 +1,19 @@ +# [S] The moq-net session bench completes moq-transport-22 at 16 broadcasts + +## Goal + +Nightly's "Rust bench targets" workload runs `moq-net`'s session bench to the +end: `session_delivery_broadcasts/moq-transport-22/.../broadcasts=16` no +longer panics with `server handshake failed: Timeout` +(`harness.rs:233`). + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete from the triage of Nightly +run [42487375545](https://github.com/moq-dev/moq/actions/runs/42487375545), +where it first failed this way). Reproduce it first, then fix the cause: +a handshake that stalls under that load is a bug in the session or the +harness, never a reason to raise the timeout. Check whether the harness +reuses state across cells and whether lite versions pass the same cell. + +Public API: none expected. Wire: none. From fb6ce13fe1b98f47c0191ea88461a1c572ab73e0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 18:23:59 -0800 Subject: [PATCH 3/7] quest: plan the Chromium Opus stalled-share rows Co-Authored-By: Claude Opus 5.5 --- quest/m1/audio-jitter-target/README.md | 1 + .../audio-jitter-target/opus-stalled-share.md | 29 +++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 quest/m1/audio-jitter-target/opus-stalled-share.md diff --git a/quest/m1/audio-jitter-target/README.md b/quest/m1/audio-jitter-target/README.md index 96cfa558533..0238c6dd2bf 100644 --- a/quest/m1/audio-jitter-target/README.md +++ b/quest/m1/audio-jitter-target/README.md @@ -51,6 +51,7 @@ native playback against its budgets. - [Buffered converge](/quest/m1/audio-jitter-target/buffered-converge.md) - a buffered `` time-stretches its audio back onto the shared clock after a stall or drift, so lip sync returns and no audio group is dropped - [Capped budgets](/quest/m1/audio-jitter-target/capped-budgets.md) - the AAC browser budgets are re-measured from nightlies after the TS importer groups each PES - [Native rendition delay](/quest/m1/audio-jitter-target/native-rendition-delay.md) - `moq play` holds each subscribed rendition's catalog `delay` plus its target, as the browser does +- [Chromium Opus stalled share](/quest/m1/audio-jitter-target/opus-stalled-share.md) - the Chromium Opus browser rows stay under the 0.01 stalled-share budget every night ## Closes diff --git a/quest/m1/audio-jitter-target/opus-stalled-share.md b/quest/m1/audio-jitter-target/opus-stalled-share.md new file mode 100644 index 00000000000..fb4932f6685 --- /dev/null +++ b/quest/m1/audio-jitter-target/opus-stalled-share.md @@ -0,0 +1,29 @@ +# [S] Chromium Opus rows stay under the stalled-share budget + +## Goal + +Nightly's Audio quality browser rows for Chromium Opus pass their +`stalled_share` budget (0.01) every night. Today a different row fails each +night at 0.014: `opus-48000-bursty` on 2026-11-08 and `opus-48000-wide` on +2026-11-07. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete from the triage of Nightly +runs [42377867523](https://github.com/moq-dev/moq/actions/runs/42377867523) +and [42487375545](https://github.com/moq-dev/moq/actions/runs/42487375545)). +✅ A child of this epic, m1 (rejected: leaving it to the watch quest's +grading). + +Start after [Watch](/quest/m1/audio-jitter-target/watch.md) lands +([moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) time-stretches +the browser's audio ring), then grade the next scheduled nightlies (never +dispatch one). If the rows pass, delete this quest. Otherwise find what +stalls Opus that the stretch does not cover and fix it there; do not loosen +the budget to pass. + +Public API: none expected. Wire: none. + +## Required + +- [Watch](/quest/m1/audio-jitter-target/watch.md) - the browser time-stretches its audio ring, which may already remove these stalls From 15fc93ce88cd2bfef1904f742487827e2ce3f9ae Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:07:03 -0800 Subject: [PATCH 4/7] quest: plan the two js/net load flakes #8333 quarantines Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 2 ++ .../m0/test-flakes-2/compressed-frame-cap.md | 27 +++++++++++++++++ quest/m0/test-flakes-2/redropped-reoffer.md | 29 +++++++++++++++++++ 3 files changed, 58 insertions(+) create mode 100644 quest/m0/test-flakes-2/compressed-frame-cap.md create mode 100644 quest/m0/test-flakes-2/redropped-reoffer.md diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 3c9d08b8011..6d6d97a086a 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -229,3 +229,5 @@ Public API: none. Wire: none. - [js/net's lite-06 session-close test keeps the end under load](/quest/m0/test-flakes-2/session-close-end.md) - a session close right after a datagram track's end returns the end, not a `remote error: 0`, and its lite-06 skip is removed - [js/net's widened max-age re-drop test ends cleanly, not with a reset](/quest/m0/test-flakes-2/widened-max-age-reset.md) - the test finishes the track only after the re-drop reply arrives, so the end is a FIN, and drops its wall-clock assert - [bench/relay.test.py's stream-transport test gets a port free for TCP and UDP](/quest/m0/test-flakes-2/relay-port-pair.md) - the test binds TCP and UDP to one number free in both, so parallel checks can't collide with it +- [js/net's compressed-frame cap test holds up under load](/quest/m0/test-flakes-2/compressed-frame-cap.md) - the cap is proven without a 32 MiB payload, and its `test.skip` is removed +- [js/net's lite-07 re-dropped re-offer test holds up under load](/quest/m0/test-flakes-2/redropped-reoffer.md) - update 3 is ordered before the re-offer opens by an event, not a yield count, and its `test.skip` is removed diff --git a/quest/m0/test-flakes-2/compressed-frame-cap.md b/quest/m0/test-flakes-2/compressed-frame-cap.md new file mode 100644 index 00000000000..487df9e9340 --- /dev/null +++ b/quest/m0/test-flakes-2/compressed-frame-cap.md @@ -0,0 +1,27 @@ +# [S] js/net's compressed-frame cap test holds up under load + +## Goal + +`js/net/src/group.test.ts`'s "a compressed frame is refused unless its +worst-case slice fits a peer's frame cap" passes on a loaded host, and its +`test.skip` is removed. It still proves the cap: a frame whose worst-case +slice exceeds the peer's cap is refused, and one that fits is accepted. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete), under the repository's +flaky-test rule. Quarantined by +[moq-dev/moq#8333](https://github.com/moq-dev/moq/pull/8333) after the +whole `js/net` package failed it 2 of 30 times under 48 CPU burners on 24 +cores, first seen in [moq-dev/moq#8326](https://github.com/moq-dev/moq/pull/8326)'s +loaded runs. + +The failure is bun's per-test limit ("timed out after 5000ms", at 5840 ms and +10442 ms). Alone it takes about 0.85 s: it deflates and inflates a random +payload of almost 32 MiB, so the CPU work, not a race, runs past the limit +under load. This resembles the flate tail-shed test fixed in +[moq-dev/moq#8137](https://github.com/moq-dev/moq/pull/8137). Prove the cap +with a payload sized relative to a small cap (or a cap the test sets low) +rather than a full-size one; never raise the timeout. + +Public API: none. Wire: none. diff --git a/quest/m0/test-flakes-2/redropped-reoffer.md b/quest/m0/test-flakes-2/redropped-reoffer.md new file mode 100644 index 00000000000..aaec59a6ef2 --- /dev/null +++ b/quest/m0/test-flakes-2/redropped-reoffer.md @@ -0,0 +1,29 @@ +# [S] js/net's lite-07 re-dropped re-offer test holds up under load + +## Goal + +`js/net/src/lite/publisher.test.ts`'s "lite draft-07: a re-offer dropped +again before it opens stays stale at the end" passes on a loaded host, and +its `test.skip` is removed. Update 3 is guaranteed to land before the +re-offer of group 1 opens, so the end still reports group 1 as stale. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete), under the repository's +flaky-test rule. Quarantined by +[moq-dev/moq#8333](https://github.com/moq-dev/moq/pull/8333) after the whole +`js/net` package failed it 1 of 30 times under 48 CPU burners on 24 cores, +first seen in [moq-dev/moq#8326](https://github.com/moq-dev/moq/pull/8326)'s +loaded runs. + +The failure: after 22 ms, `holesToEnd` returned `[]` where +`[["drop", 1n, 1n, 3n]]` was expected (`publisher.test.ts:1572`). Suspected, +not confirmed: the write budget is already stubbed, but the test sends update +2, waits one `yieldNow()`, then sends update 3, assuming update 3 lands +before the re-offer opens; under load the re-offer is served first. Reproduce +it first. Then order the test on an event (the update's reply or the +re-offer's queueing), not a count of yields. If the publisher can serve a +re-offer an already-sent update dropped, that is a publisher bug: fix it there, +mirroring Rust's `a_redropped_reoffer_stays_stale_at_the_end_on_lite07`. + +Public API: none. Wire: none. From 8c5c3e4718d08d4a1023e357c13edf9a7c33d3d7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:21:52 -0800 Subject: [PATCH 5/7] test(moq-net): quarantine the slow random-mesh withdrawal tests and plan their fix Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 1 + .../m0/test-flakes-2/mesh-withdraw-runtime.md | 35 +++++++++++++++++++ .../audio-jitter-target/opus-stalled-share.md | 16 ++++----- rs/moq-net/tests/integration/mesh_withdraw.rs | 2 ++ sh/gh/test-inventory.allow | 2 ++ 5 files changed, 46 insertions(+), 10 deletions(-) create mode 100644 quest/m0/test-flakes-2/mesh-withdraw-runtime.md diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 328ee76af88..030499f2758 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -229,3 +229,4 @@ Public API: none. Wire: none. - [bench/relay.test.py's stream-transport test gets a port free for TCP and UDP](/quest/m0/test-flakes-2/relay-port-pair.md) - the test binds TCP and UDP to one number free in both, so parallel checks can't collide with it - [js/net's compressed-frame cap test holds up under load](/quest/m0/test-flakes-2/compressed-frame-cap.md) - the cap is proven without a 32 MiB payload, and its `test.skip` is removed - [js/net's lite-07 re-dropped re-offer test holds up under load](/quest/m0/test-flakes-2/redropped-reoffer.md) - update 3 is ordered before the re-offer opens by an event, not a yield count, and its `test.skip` is removed +- [moq-net's random-mesh withdrawal tests fit the gate](/quest/m0/test-flakes-2/mesh-withdraw-runtime.md) - the 200-seed mesh tests run in seconds on the test clock, and their `#[ignore]` is removed diff --git a/quest/m0/test-flakes-2/mesh-withdraw-runtime.md b/quest/m0/test-flakes-2/mesh-withdraw-runtime.md new file mode 100644 index 00000000000..821d12a558f --- /dev/null +++ b/quest/m0/test-flakes-2/mesh-withdraw-runtime.md @@ -0,0 +1,35 @@ +# [S] moq-net's random-mesh withdrawal tests fit the gate + +## Goal + +`moq-net`'s `mesh_withdraw::random_meshes_retract_once` and +`capped_random_meshes_retract_once` run in a few seconds on a loaded host, +well under nextest's 60 s kill (`.config/nextest.toml`, whose stated ceiling +for an honest test is about 9 s), and their `#[ignore]` and +`sh/gh/test-inventory.allow` entries are removed. They still check every one +of the 200 seeded meshes retracts exactly once. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete): ✅ quarantine now and plan +this quest (rejected: leave them in the gate and only plan; move them to +nightly without a quest). Reported by +[moq-dev/moq#8247](https://github.com/moq-dev/moq/pull/8247)'s iterate, whose +`just check` saw both hit the 60 s kill. On `main` they take about 40 s and +45 s even alone, so any load pushes them over. CI passed them, so this is cost, +not a race. + +Find where the time goes before changing anything: the 200 seeds in +`withdrawal_counts` each build a mesh, publish, close, and wait out the +withdrawal, so a wall-clock wait per seed (such as the 300 ms hold) costs +minutes in total. Drive the waits on the test clock (`#[kio::test]`'s +manual clock), or run the seeds concurrently, so the work is CPU-bound and +small. Cutting seeds is the last resort: the counts in each test's doc +comment came from all 200. If neither fits, move them to the nightly suite +with a caller instead of ignoring them. Never raise the timeout. + +[Route layer acceptance](/quest/m2/cluster-routing/routes/acceptance.md) +owns the `residual_*` counterexamples in the same file; keep those as they +are. + +Public API: none. Wire: none. diff --git a/quest/m1/audio-jitter-target/opus-stalled-share.md b/quest/m1/audio-jitter-target/opus-stalled-share.md index fb4932f6685..f51a17e7339 100644 --- a/quest/m1/audio-jitter-target/opus-stalled-share.md +++ b/quest/m1/audio-jitter-target/opus-stalled-share.md @@ -15,15 +15,11 @@ and [42487375545](https://github.com/moq-dev/moq/actions/runs/42487375545)). ✅ A child of this epic, m1 (rejected: leaving it to the watch quest's grading). -Start after [Watch](/quest/m1/audio-jitter-target/watch.md) lands -([moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058) time-stretches -the browser's audio ring), then grade the next scheduled nightlies (never -dispatch one). If the rows pass, delete this quest. Otherwise find what -stalls Opus that the stretch does not cover and fix it there; do not loosen -the budget to pass. +[moq-dev/moq#8058](https://github.com/moq-dev/moq/pull/8058), which +time-stretches the browser's audio ring, merged on 2026-11-09 and may already +remove these stalls. Grade the scheduled nightlies after it (never dispatch +one). If the rows pass, delete this quest. Otherwise find what stalls Opus +that the stretch does not cover and fix it there; do not loosen the budget to +pass. Public API: none expected. Wire: none. - -## Required - -- [Watch](/quest/m1/audio-jitter-target/watch.md) - the browser time-stretches its audio ring, which may already remove these stalls diff --git a/rs/moq-net/tests/integration/mesh_withdraw.rs b/rs/moq-net/tests/integration/mesh_withdraw.rs index ffdbb0cb14b..0ac4592e7af 100644 --- a/rs/moq-net/tests/integration/mesh_withdraw.rs +++ b/rs/moq-net/tests/integration/mesh_withdraw.rs @@ -575,6 +575,7 @@ fn random_mesh(seed: u64) -> (Vec, usize) { /// revive, and how many updates they send beyond the one retraction. Hop chains /// counted (231, 3, 290); routed by origin node, none. #[kio::test] +#[ignore = "quarantined: takes 40s+ alone and times out under load; quest/m0/test-flakes-2/mesh-withdraw-runtime.md"] async fn random_meshes_retract_once() { assert_eq!(withdrawal_counts(false).await, (0, 0, 0)); } @@ -583,6 +584,7 @@ async fn random_meshes_retract_once() { /// extra hops, including the publishing client outside the relay graph. Hop /// chains counted (177, 0, 192). #[kio::test] +#[ignore = "quarantined: takes 40s+ alone and times out under load; quest/m0/test-flakes-2/mesh-withdraw-runtime.md"] async fn capped_random_meshes_retract_once() { assert_eq!(withdrawal_counts(true).await, (0, 0, 0)); } diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index c8debe20bb3..ea8961f79e7 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -36,6 +36,8 @@ ignore moq-video::render::export::dx12::tests::exports_an_nv12_texture /quest/m5 ignore moq-video::render::export::dx12::tests::media_foundation_encodes_the_export_in_place /quest/m5/video-hardware.md # Quarantined flakes, re-enabled by the quest that fixes them. +ignore moq-net::mesh_withdraw::random_meshes_retract_once /quest/m0/test-flakes-2/mesh-withdraw-runtime.md +ignore moq-net::mesh_withdraw::capped_random_meshes_retract_once /quest/m0/test-flakes-2/mesh-withdraw-runtime.md # Need another implementation to speak the route wire; the quest ports it. ignore moq-net::test_interop::lite_tracks_interop /quest/m2/cluster-routing/routes/js.md From e679b5151f851c21284ebfd8f0e474d4284a846c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:48:58 -0800 Subject: [PATCH 6/7] quest: plan the mesh tests' CPU cost on the sim clock and date the new flakes Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 11 ++++++++++- quest/m0/test-flakes-2/mesh-withdraw-runtime.md | 14 ++++++++------ 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 030499f2758..4d09077ce3c 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -219,6 +219,15 @@ raw media warmup test, which failed in the merge queue for buffered groups on the FFI runtime's real clock. It is quarantined with `#[ignore]` until its child fixes it. +Added 2026-11-09 (maintainer, /quest-complete): `js/net`'s compressed-frame +cap test and lite-07 re-dropped re-offer test, which failed under load during +the datagram-queues runs and were quarantined with `test.skip` by +[moq-dev/moq#8333](https://github.com/moq-dev/moq/pull/8333); and moq-net's +two random-mesh withdrawal tests, which hit nextest's 60 s kill in +[moq-dev/moq#8247](https://github.com/moq-dev/moq/pull/8247)'s local +`just check` and are quarantined with `#[ignore]` by +[moq-dev/moq#8336](https://github.com/moq-dev/moq/pull/8336). + Public API: none. Wire: none. ## Required @@ -229,4 +238,4 @@ Public API: none. Wire: none. - [bench/relay.test.py's stream-transport test gets a port free for TCP and UDP](/quest/m0/test-flakes-2/relay-port-pair.md) - the test binds TCP and UDP to one number free in both, so parallel checks can't collide with it - [js/net's compressed-frame cap test holds up under load](/quest/m0/test-flakes-2/compressed-frame-cap.md) - the cap is proven without a 32 MiB payload, and its `test.skip` is removed - [js/net's lite-07 re-dropped re-offer test holds up under load](/quest/m0/test-flakes-2/redropped-reoffer.md) - update 3 is ordered before the re-offer opens by an event, not a yield count, and its `test.skip` is removed -- [moq-net's random-mesh withdrawal tests fit the gate](/quest/m0/test-flakes-2/mesh-withdraw-runtime.md) - the 200-seed mesh tests run in seconds on the test clock, and their `#[ignore]` is removed +- [moq-net's random-mesh withdrawal tests fit the gate](/quest/m0/test-flakes-2/mesh-withdraw-runtime.md) - the 200-seed mesh tests cost seconds of CPU, and their `#[ignore]` is removed diff --git a/quest/m0/test-flakes-2/mesh-withdraw-runtime.md b/quest/m0/test-flakes-2/mesh-withdraw-runtime.md index 821d12a558f..d4517a9e1e6 100644 --- a/quest/m0/test-flakes-2/mesh-withdraw-runtime.md +++ b/quest/m0/test-flakes-2/mesh-withdraw-runtime.md @@ -19,12 +19,14 @@ nightly without a quest). Reported by 45 s even alone, so any load pushes them over. CI passed them, so this is cost, not a race. -Find where the time goes before changing anything: the 200 seeds in -`withdrawal_counts` each build a mesh, publish, close, and wait out the -withdrawal, so a wall-clock wait per seed (such as the 300 ms hold) costs -minutes in total. Drive the waits on the test clock (`#[kio::test]`'s -manual clock), or run the seeds concurrently, so the work is CPU-bound and -small. Cutting seeds is the last resort: the counts in each test's doc +Find where the time goes before changing anything. The tests run on +`#[kio::test]`'s single-threaded simulated clock, where every wait is +`kio::clock::sim::idle()`, so the 300 ms hold costs no wall time: the 40 s is +CPU spent across the 200 seeds in `withdrawal_counts`, each building a mesh, +publishing, closing, and settling. Profile one seed (`cargo flamegraph` or +`perf` on the test binary) and cut the per-seed cost it shows, such as mesh +setup the seeds could share or work that grows with the mesh on each route +change. Cutting seeds is the last resort: the counts in each test's doc comment came from all 200. If neither fits, move them to the nightly suite with a caller instead of ignoring them. Never raise the timeout. From a32c75bf721d307b8da53d90f71c0b3d7e35ae97 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 20:49:59 -0800 Subject: [PATCH 7/7] quest: record that #8329 re-enabled the ffi warmup test Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 4d09077ce3c..61b115b76e4 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -216,8 +216,9 @@ OS picked for TCP and failed a local run while other checks held that UDP port. Added 2026-11-08 (maintainer's standing rule that flakes join this epic): moq-ffi's raw media warmup test, which failed in the merge queue for [moq-dev/moq#8272](https://github.com/moq-dev/moq/pull/8272) when its reader skipped one of three -buffered groups on the FFI runtime's real clock. It is quarantined with `#[ignore]` until its -child fixes it. +buffered groups on the FFI runtime's real clock. +[moq-dev/moq#8329](https://github.com/moq-dev/moq/pull/8329) moved it to the paused test +clock, re-enabled it, and deleted its child. Added 2026-11-09 (maintainer, /quest-complete): `js/net`'s compressed-frame cap test and lite-07 re-dropped re-offer test, which failed under load during