Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 16 additions & 15 deletions bench/ffi-gen/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ Each lands its fixes in its own PR and plans only what needs a decision.
- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription
- [Performance audit](/quest/m0/audit-perf.md) - benchmarks that regressed since the last run are explained, fixed, or planned, and unbenched hot paths get coverage
- [Compile time audit](/quest/m0/audit-compile-time.md) - build-time hotspots are measured against a baseline and the worst are fixed or planned
- [Test and CI cost audit](/quest/m0/audit-test-cost.md) - the slowest, flakiest, and least valuable tests and CI jobs are fixed, deleted, or planned
- [Documentation audit](/quest/m0/audit-docs.md) - stale pages, examples, links, and undocumented exports are fixed or planned
- [Dependency and alternatives audit](/quest/m0/audit-deps.md) - dependencies are current, deduplicated, needed, and the best maintained option
- [Artifact size audit](/quest/m0/audit-size.md) - binary, FFI, WASM, and bundle growth since the last run is explained, fixed, or planned
Expand Down
7 changes: 4 additions & 3 deletions quest/m0/audit-perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,10 @@ Audit evidence (2026-10-19, second checkpoint, incomplete):
nights; its -12% on small frames sits inside that band, and both nights
used Dart 3.13.0 (the nixpkgs bump `4109d3893` did not change it). No
binding regression is established. Recheck Dart against `gh-pages`.
- JS: the Nightly JS leg is cancelled at one hour by the forward fixture,
which [the test-cost audit](/quest/m0/audit-test-cost.md) is fixing. Until
it lands there is no JS history. Nightly never runs these JS benches:
- JS: the Nightly JS leg was cancelled at one hour by the forward fixture,
which the test-cost audit fixed in
[#5976](https://github.com/moq-dev/moq/pull/5976). Until a Nightly runs
with it there is no JS history. Nightly never runs these JS benches:
net `observed`; json `codec` and `parse`; hang `slow-group`; watch
`warmup`; clock `timers` (`qmux` and `serve-browser` run, uncharted, in
the audio-quality job). Add them, with an inventory test like the Rust
Expand Down
85 changes: 0 additions & 85 deletions quest/m0/audit-test-cost.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m0/ci/audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Every merge queue and nightly job is timed, and every expensive step either
earns its minutes or is cut, moved or planned. The Rust cache reuses
unchanged crates, and CI compiles nothing it doesn't need. Fixes land in this
quest's PR, one commit each. Ranking individual tests stays with the
recurring [test cost audit](/quest/m0/audit-test-cost.md).
recurring [test cost audit](/quest/m1/audit-test-cost.md).

## Plan

Expand Down
3 changes: 3 additions & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,11 @@ lands before the cut.

- [C++ and OBS package proof](/quest/m1/cpp-package-check.md) - the installed artifacts pass before their first release

- [Nightly media players](/quest/m1/media-engines-nightly.md) - the Firefox and WebKit player rows pass, or an engine that cannot play leaves the nightly matrix

- [Dead code and API surface audit](/quest/m1/audit-dead-code.md) - repeat the removal audit before the next major release
- [Test coverage audit](/quest/m1/audit-test-coverage.md) - repeat the coverage audit against the m0 baselines before the next major release
- [Test and CI cost audit](/quest/m1/audit-test-cost.md) - the slowest, flakiest, and least valuable tests and CI jobs are fixed, deleted, or planned

## Related

Expand Down
73 changes: 73 additions & 0 deletions quest/m1/audit-test-cost.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# [M] Test and CI cost audit

## Goal

The slowest, flakiest, and least valuable tests and CI jobs are sped up, fixed,
deleted, or planned, so the suite costs what it earns.

## Plan

Recurs once per major release (decided 2026-10-19). The PR that finishes
this run also adds the next one with the same file name at the end of the next
milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal
and this paragraph; the rest of its Plan is rewritten with what this run
learned: baselines, results, and what to check first. A successor placed in m2
also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it
waits for the release in flight to ship. Only `/quest-delete` ends the
recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups.
Only a change that needs a maintainer decision (public API, scope) or is
independently landable and too large to review alongside the rest becomes a
follow-up through `/quest-plan`.

Use nextest's normal CI log to rank the slowest tests (`slow-timeout` is 30s).
Flakes are tracked by hand in [test flakes](/quest/m1/test-flakes-2/README.md),
and `retries = 0` is deliberate. A test that sleeps on the wall clock should
mock time (AGENTS.md). Delete tests whose coverage another test already
provides. [CI cost and latency](/quest/m0/ci/README.md) owns runner and cache
infrastructure. Once the one-time [CI pipeline audit](/quest/m0/ci/audit.md)
closes, this audit ranks job costs again.

What the m0 run (2026-10-19) learned; check these first:

- Classify every failed job of the latest Nightly, scheduled Interop, and
Platform runs, not just the slowest. The m0 run's first checkpoint looked
mainly at the Nightly's stalled JS benchmarks, while 28 Nightly jobs had
failed for eight independent causes, some for days. A red Nightly hides the
next regression: the send-depth gate and all six fuzz targets broke on
2026-10-18 and stayed broken unnoticed until this audit.
- Platform (macOS, Windows) runs only in the merge queue and is not required,
so a Windows-only compile error in moq-video stayed red for two days across
dozens of queue runs until
[#5966](https://github.com/moq-dev/moq/pull/5966). Most Platform failures
in the sample were that error, a choco `nasm` install outage, or another
queued PR's breakage. Rank Platform by failure cause, not count.
- A queue failure often belongs to another PR in the same batch: a stale
fixture in the entry itself, or a semantic conflict with a PR merged after it
was queued. Check the entry's head branch and what merged around it before
calling it a flake.
- Fixed by the m0 run: the ACME rate-limit test's released port, the stats
session graph's recursive Send proof (send-depth and fuzz), bench/ffi-gen's
stale wasm-bindgen lock, moq-uring's fixed 1 ms calibration slack, the
Nightly JS benchmark stall, the CI selection test's feature count, and the
stats producer benchmark lint.
- Planned by the m0 run: [redundant failover drift](/quest/m1/test-flakes-2/redundant-failover-drift.md),
[macOS QUIC connects](/quest/m1/test-flakes-2/macos-quic-connect.md), and
[nightly media players](/quest/m1/media-engines-nightly.md). Owned
elsewhere: release builds by [release builds link](/quest/m0/release-link.md),
Rust advisories by the dependency audit, and the Nix wasm-bindgen pin by
[Cargo.lock pins wasm-bindgen](/quest/m0/ci/wasm-bindgen-pin.md).

Baselines from queue run
[39371990135](https://github.com/moq-dev/moq/actions/runs/39371990135):
Test took 17 min and Check 15 min; nextest ran 7,657 tests in 220 s. The
slowest were the three new redundant-importer exports (37, 29 and 15 s,
waiting out a killed importer's 10 s QUIC idle timeout), the CLI
restart/stitch case (18.6 s), same-instance linger (17.5 s), TS source-limit
timing (16.5 s), the impaired flapping-peer drill (16.4 s), and the one-day
jitter drift simulation (14.7 s). Apart from the redundant exports, the
leaders match the earlier sample and cover different behaviors.

Still to measure: a JS net benchmark-only edit selects WASM and the net
dependents, and a stats benchmark-only edit selects relay and CLI tests, docs,
and CLI's optional play build. Compare that cost with the regression signal
before changing selection.
38 changes: 38 additions & 0 deletions quest/m1/media-engines-nightly.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# [S] Nightly media players pass on Firefox and WebKit, or leave the matrix

## Goal

Nightly's `Media (firefox player)` and `Media (webkit player)` jobs pass, or
an engine that cannot play the fixture leaves the matrix with its limit
recorded. Neither has passed since
[#5640](https://github.com/moq-dev/moq/pull/5640) added them on 2026-10-16,
so every nightly is red for a known cause and hides new failures.

## Plan

Found by the 2026-10-19 test cost audit; the same failure each night
(for example [webkit](https://github.com/moq-dev/moq/actions/runs/39266265648/job/118106525118)
and [firefox](https://github.com/moq-dev/moq/actions/runs/39266265648/job/118106525416)):

- WebKit 26.6 and 27.2 fail the capability check: "cannot run player:
missing decode avc1.42001f, decode vp8, decode opus". Every row then fails.
Playwright's Linux WebKit is not Safari, so this looks like a limit of the
build rather than a player bug. #5640 left exactly this call open.
- Firefox 157 fails each row at "gesture starts audio: waiting for the audio
graph to resume after a user gesture". #5640 saw this locally once and
could not reproduce it on a host with an audio server; the arm runner may
have none.

Open decision for WebKit, recommended first:

1. Drop it from the nightly matrix and record the missing decoders, since a
row that cannot pass measures nothing. Real Safari coverage stays with
[WebGPU on Safari](/quest/m1/webgpu-safari.md) and manual runs.
2. Keep it and report the capability gap without failing the job, which
contradicts the harness's fail-by-name rule.

For Firefox, reproduce on an arm runner image first. If the cause is the
missing audio device, give the job one (for example a null PulseAudio sink)
rather than relaxing the gesture check.

Public API: none. Wire: none.
6 changes: 5 additions & 1 deletion quest/m1/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,19 @@ times on a loaded machine, as the first round did. moq-tokio's
known exception, tracked by [Readers see SUBSCRIBE_DROP](/quest/m1/subscribe-drop-errors.md).

Added 2026-10-19 (maintainer): three more, the redial pair first since CI
hit it.
hit it. The same day's test cost audit added the macOS QUIC connects, and the
redundant failover drift first: it may hide a TS export bug and failed two of
three loaded `just check` runs.

Public API: none. Wire: none.

## Required

- [Redundant failover drift](/quest/m1/test-flakes-2/redundant-failover-drift.md) - a stitching TS export switches back to the surviving importer behind bufferbloat instead of failing on a stalled clock
- [moq-tokio redial tests](/quest/m1/test-flakes-2/tokio-redial.md) - the two redial tests stop assuming a dropped port stays free and stop racing 10 s deadlines
- [moq-ffi cancel during connect](/quest/m1/test-flakes-2/ffi-cancel-connect.md) - `client_cancel_aborts_a_pending_connect` orders its cancel deterministically, fixing the code if the race is real
- [Flapping-peer reorder seed](/quest/m1/test-flakes-2/flapping-peer-reorder.md) - replay one drill seed on current main, and fix it or delete the quest
- [macOS QUIC connects](/quest/m1/test-flakes-2/macos-quic-connect.md) - moq-tokio's loopback QUIC tests stop timing out on the macOS Platform job

## Related

Expand Down
Loading