Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ The historical samples locate candidates; they are not controlled benchmarks.
- [Read-only pull-request CI](/quest/m0/ci/read-only-secrets.md) - write secrets and publisher identities reach only protected branches and tags, never a pushed branch
- [Land main through quest land](/quest/m0/ci/land.md) - a host daemon tests small batches of queued PRs once and lands each as its tested squash commit
- [A cache error is a miss](/quest/m0/ci/nix-cache-5xx.md) - a transient `nix.moq.dev` 5xx no longer fails `just ci`
- [Every io_uring test in the io-uring group](/quest/m0/ci/io-uring-test-group.md) - the parallel tests of one `just ci` stop exhausting the shared `RLIMIT_MEMLOCK`, and a too-low limit fails clearly
- [Cheap checks skip the queue](/quest/m0/ci/gate-quick.md) - condition: quest ships `quest gate --quick`; then cheap modules wait at most a minute
- [Gate each diff in under five minutes](/quest/m0/ci/five-minutes.md) - `just check` and `just test` each finish within a worker's 5-minute prompt cache, admission wait included
- [Measure the parallel frontend](/quest/m0/ci/nightly-levers.md) - a pinned nightly's parallel frontend, adopted only if the numbers beat its risks; Cranelift is ruled out
Expand Down
21 changes: 21 additions & 0 deletions quest/m0/ci/io-uring-test-group.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# [XS] Every io_uring test runs in the nextest io-uring group

## Goal

Every test that opens an io_uring, including the relay's `wire_stats::uring_*`, `peer_limits::uring_*`, and `refusals::uring_*` integration tests and the `bitrate_caps::uring_*` tests [moq-dev/moq#7535](https://github.com/moq-dev/moq/pull/7535) adds, runs inside the `io-uring` nextest test group, so the tests of one `just ci` running in parallel no longer exhaust the user's shared `RLIMIT_MEMLOCK`. The group caps rings only within one nextest invocation; separate invocations still share the budget, as `rs/moq-uring/README.md` documents, and this quest does not change that. An io_uring test that still hits too low a limit fails with a message naming `RLIMIT_MEMLOCK` and how to raise it, not a generic setup error or a timeout.

## Plan

Found by [moq-dev/moq#7535](https://github.com/moq-dev/moq/pull/7535) and [moq-dev/moq#7211](https://github.com/moq-dev/moq/pull/7211): #7535's io_uring tests passed serially but could hit the shared limit in parallel, and #7211's `just ci` failed once in `uring_peer_session_widens_quic_limits` on `RLIMIT_MEMLOCK`.
Comment thread
kixelated marked this conversation as resolved.
Resolved

Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m0, since each such failure fails a whole merge-gate batch.

- The group's filter in `.config/nextest.toml` names only `runtime_uring::` and `embed::.*uring` in `moq-relay::integration`. Match every io_uring test instead, for example `test(/uring/)` in that binary plus moq-relay's own unit tests that start ring workers, so a new test joins by its name. Check that the inventory of uring tests (`rg 'fn uring_' rs/moq-relay`) all land in the group with `cargo nextest show-config test-groups`, which lists each group's members; plain `cargo nextest list` shows no group.
- moq-uring already names the limit on a ring's `ENOMEM` (`rs/moq-uring/src/error.rs`). Check that this error reaches the failing test's output on the relay paths (worker startup, `start_relay`) instead of being wrapped or swallowed; if a test sees only a timeout, surface the startup error there.
- Keep the group's `max-threads` unless measured otherwise; the limit is shared with other checkouts, which `rs/moq-uring/README.md` documents.

Public API: none. Wire: none.

## Related

- [More tests under load](/quest/m0/test-flakes-2/README.md) - the flake epic this would otherwise land in
8 changes: 8 additions & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,12 @@ contradict the code. Its other findings already had owners or contradicted
settled decisions (archives break in place; OBS video-only output stays
removed).

Added 2026-11-01 (maintainer, follow-ups from the 2026-11-01 /quest-afk run):
the shared pacer's overshoot under 2 Mbit/s with the BBR3 retune it needs,
since BBR3 is the default controller the cut ships; a truncated MKV that
left its tracks open; and, under release performance, a bisect of the relay's
per-frame CPU growth since #5926.

## Required

- [Address groups by timestamp](/quest/m1/time-addressing/README.md) - group start timestamps replace sequence numbers in the moq-net API and lite-07 wire, while old subscribers, old publishers, and published JS clients keep working
Expand Down Expand Up @@ -169,6 +175,7 @@ removed).
- [Interop coverage](/quest/m1/interop-coverage/README.md) - the riskiest moq-transport features moq-net exposes get cross-implementation tests over drafts 18 and 22
- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - A/A and deliberately slowed Nightly runs set the paired comparison's threshold, so its regression alerts can be trusted
- [Probe scenario seeds randomize the media's start](/quest/m1/sim-media-offset.md) - each seed draws its own media start offset against the handshake and competitor, so probe budgets are not judged against one shared timing alignment
- [The shared pacer holds its rate under 2 Mbit/s](/quest/m1/pacer-low-rate.md) - a rate-paced controller stops sending up to twice its pacing rate at low rates, landed with a BBR3 retune that keeps the probe budgets
- [HLS and DASH segment conformance](/quest/m1/hls-conformance.md) - nightly ffprobe checks that advertised timing, sync flags, and bytes match the media served
- [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh
- [Legacy lite close drain](/quest/m1/lite-legacy-close-drain.md) - a lite-01..06 graceful close never cuts a final group the subscriber has not read
Expand All @@ -180,6 +187,7 @@ removed).
- [HLS export follows a resolution change](/quest/m1/hls-reinit-same-codec.md) - a decoder-identity change under the same codec string starts a new init and segment tag
- [A JS reader finishes after a clean close](/quest/m1/js-close-range-finish.md) - a range ending below the cache delivers what's held and then ends
- [fMP4 decode times keep increasing](/quest/m1/fmp4-bframe-decode-time.md) - fragments shorter than a GOP of B-frame video never step `tfdt` backwards
- [A truncated MKV ends its tracks at finish](/quest/m1/mkv-truncated-finish.md) - `finish()` refusing truncated input still ends the tracks, so subscribers stop waiting
- [moq-quic deletes its remaining 0-RTT code](/quest/m1/quic-zero-rtt-leftovers.md) - only the code that drops a 0-RTT packet remains, with a test
- [Gate the no-em-dash rule](/quest/m1/em-dash-gate.md) - `just check` fails on an em dash in any tracked text file

Expand Down
19 changes: 19 additions & 0 deletions quest/m1/mkv-truncated-finish.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# [XS] A truncated MKV ends its tracks at finish

## Goal

When the MKV importer's `finish()` refuses truncated input (`Mkv(Truncated)`), it still ends its tracks with what was accepted, as it does after a refused `decode`, so a subscriber's `read()` returns instead of waiting until the importer is dropped. `finish()` still returns the error, and the frame cut short still fails its group with the reason.

## Plan

Found by the verification in [moq-dev/moq#7515](https://github.com/moq-dev/moq/pull/7515), which fed the live `opus.webm` truncated at every length and with every byte flipped: 1462 truncation lengths and 29 byte flips ended with tracks neither finished nor aborted after `finish()` returned `Truncated`, at every chunk size. Callers that drop the importer after an error were unaffected.

In `Import::finish` (`rs/moq-mux/src/container/mkv/import.rs`, about lines 838-852), a failed `drain_eof` aborts only the streaming block's group and returns early, skipping the per-track finish. A `decode` that already failed (`self.failed`) skips `drain_eof` and still reaches the track loop, so it ends its tracks.

Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m1, a small fix in an importer the cut ships. End the tracks on the truncation path the way a refused `decode` already does (`decode`'s docs: "`finish` ends the tracks with what was accepted"), so the two refusals behave alike. #7515 also floated aborting every track with the truncation error instead; matching the refused `decode` keeps one rule and keeps the groups that did arrive. Test with #7515's harness shape: a truncated `opus.webm` whose subscriber's read ends after `finish()` while the importer is still held.

Public API: none. Wire: none.

## Related

- [Untrusted media parsers have fuzz targets](/quest/m2/media-fuzz.md) - could assert that every track ends after `finish()`, whatever the input
23 changes: 23 additions & 0 deletions quest/m1/pacer-low-rate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# [S] The shared pacer holds its rate under 2 Mbit/s

## Goal

Under 2 Mbit/s, a rate-paced controller on moq-quic (BBR3 today, and any candidate that sets a pacing rate) sends at its pacing rate, not up to twice it. BBR3 is retuned in the same PR so the nightly probe scenarios, `competing_cubic` included, stay within their late-frame and overhead budgets.

## Plan

Found by [moq-dev/moq#7546](https://github.com/moq-dev/moq/pull/7546) (the GCC candidate). Below 2 Mbit/s the pacer's burst budget is one packet, and moq-quic checks a datagram's pace while the packet before it is still being built, so a batch of two leaves on one packet's credit. The shared pacer (`rs/moq-congestion/src/pacer.rs`) then forgets the overshoot instead of charging it to the next send. #7546 fixed it by carrying the overshoot as debt (its commit `b62e538b94`), which moved BBR3's nightly `competing_cubic` (`rs/moq-quic/src/tests/probe.rs`) to 441.5 late frames against an allowed 439.4, so it reverted the fix rather than retune BBR3 there.

Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m1, because the next release ships BBR3 as moq-quic's default and a 2x overshoot at low rates is a correctness bug in what it sends. Land the pacer fix and BBR3's retune together, so the nightly never runs one without the other.

- Start from `b62e538b94`'s debt approach, with a unit test in `pacer.rs` that a batch outrunning a one-packet budget is charged to the next send, and the existing `aggregate_throughput_matches_controller_pacing_rate` swept below 2 Mbit/s.
- Retune where the change shifts BBR3's probing, not the budgets: the probe scenarios' budgets judge the probe, and loosening them to pass would hide the fix's effect. Report every probe scenario's paired late frames, p99, and overhead before and after, on the scenario's full seed set.
- `competing_cubic`'s 8-seed late budget is fragile on its own ([moq-dev/moq#7538](https://github.com/moq-dev/moq/pull/7538) found 28% of 8-seed windows fail it); its seed count is planned separately in [moq-dev/moq#7440](https://github.com/moq-dev/moq/pull/7440). Judge this change on a seed set large enough to separate it from that noise, not on whether seeds 1-8 pass.
- Re-run the realtime candidates' rows that pace below 2 Mbit/s ([GCC](https://github.com/moq-dev/moq/pull/7546), [BBR3 delay bound](https://github.com/moq-dev/moq/pull/7544), [SCReAMv2](https://github.com/moq-dev/moq/pull/7547)) for whichever have landed, and record the shift in the [epic](/quest/m2/quic-realtime/README.md).

Public API: none. Wire: none.

## Related

- [Realtime uplink congestion control](/quest/m2/quic-realtime/README.md) - every candidate paces through this pacer
- [Probe up to a target bitrate](/quest/m2/probe/README.md) - its scenario budgets are what the retune must keep
21 changes: 20 additions & 1 deletion quest/m1/quic-api/poll.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [S] The QUIC drivers' waits poll
# [M] The QUIC drivers' waits poll

## Goal

Expand Down Expand Up @@ -57,6 +57,25 @@ until a caller needs one. Report the public API:
new poll methods on `Connection`, `send::Stream` and `Endpoint`, with the
async forms unchanged.

Added 2026-11-01 (maintainer, follow-up planning after the 2026-11-01
/quest-afk run), from [moq-dev/moq#7490](https://github.com/moq-dev/moq/pull/7490),
which finishes [waits](/quest/m1/quic-api/waits.md):

- **Remove `ShouldTransmit`.** Once moq-quic wakes the transmit waiter itself,
the `ShouldTransmit` that `Chunks::finalize` and the send lease methods
(`lease`, `poll_lease`, `restore`, `release` in
`rs/moq-quic/src/connection/streams/lease.rs`) return is ignored by every
consumer. Drop it from moq-quic's public API and its callers, as part of
this reshape, since moq-quic is unreleased.
- **Test moq-tokio's close while the socket is blocked.** moq-tokio's driver
skips `poll_transmit` while the socket holds output back, so a local close
could not re-arm the closing period's timer; #7490's round 1 fixed it by
also parking the driver on the non-consuming `poll_closed`. No driver test
has a socket that holds sends back, so nothing covers it.
Add one (a test socket whose sends return `WouldBlock` until released) and
a test that a close behind it is still bounded, mirroring qmux's
`a_close_behind_a_stalled_peer_is_bounded`.

## Required

- [Waits](/quest/m1/quic-api/waits.md) - moq-quic's `poll_*` forms these forward to
Expand Down
7 changes: 4 additions & 3 deletions quest/m1/release-perf/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ Before the next release cut, every benchmark where `main` is slower than
is fixed. Cost that pays for a feature `main` deliberately added is accepted,
and the reason is written here.

This epic's own work: once its child has landed, re-pair the relay `fanout`
workload against `release` and confirm main's relay CPU is no higher in every
round. The sparse rows are accepted (2026-10-30). The browser frame path
This epic's own work: once its children have landed, re-pair the relay
`fanout` workload against `release` and confirm main's relay CPU is no higher
in every round. The sparse rows are accepted (2026-10-30). The browser frame path
needs no repeat; the [performance audit](/quest/m2/audit-perf.md) now pairs
every later cut with `release`.

Expand Down Expand Up @@ -271,6 +271,7 @@ Decided 2026-10-30 (maintainer), in
## Required

- [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) - relay fanout subscribers stop contending one track lock, so main's relay CPU matches release's in every round
- [Relay CPU per frame since #5926](/quest/m1/release-perf/relay-cpu-bisect.md) - the 18-56% rise in relay instructions per delivered frame is bisected, attributed, and its avoidable cost fixed

## Related

Expand Down
33 changes: 33 additions & 0 deletions quest/m1/release-perf/relay-cpu-bisect.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# [M] Find and fix the relay's per-frame CPU growth since #5926

## Goal

The commits that raised the relay's instructions per delivered frame between [`cdff3c8f95`](https://github.com/moq-dev/moq/commit/cdff3c8f95) ([moq-dev/moq#5926](https://github.com/moq-dev/moq/pull/5926) as merged) and `main` are found, each rise is attributed to a cause, and the avoidable costs are fixed. A rise that pays for a deliberate feature is accepted with its reason in the [release-perf epic](/quest/m1/release-perf/README.md).

## Plan

Found by [moq-dev/moq#7505](https://github.com/moq-dev/moq/pull/7505), which profiled the io_uring clock read. At `main` [`30f23c3f8a`](https://github.com/moq-dev/moq/commit/30f23c3f8a), relay instructions per delivered frame had risen 25-56% on io_uring since `cdff3c8f95` and 18-38% on the tokio workers since `1dd9b83d1`, with the same delivery:

| workload | runtime | before, kinstr/frame | `30f23c3f8a`, kinstr/frame |
|---|---|---|---|
| chat | io_uring | 86.3 [84.5-87.4] at `cdff3c8f95` | 134.8 [128.5-151.3] |
| video | io_uring | 108.7 [106.6-116.2] at `cdff3c8f95` | 136.1 [132.6-152.8] |
| chat | tokio workers | 104.0 [101.4-105.0] at `1dd9b83d1` | 143.1 [127.5-205.3] |
| video | tokio workers | 141.6 [139.3-148.3] at `1dd9b83d1` | 167.1 [164.9-173.6] |

Medians of 5 interleaved reps, min-max in brackets. The tokio rows start at [`1dd9b83d1`](https://github.com/moq-dev/moq/commit/1dd9b83d19), #5926's base, since #7505 measured the tokio workers only there and at `main`. Both runtimes moved, so at least one cause is shared code (moq-net's model, kio, or the session layer). That does not rule out a further io_uring-only cost, such as the clock read #7505 profiled. The tokio range also spans #5926 itself, so the two rises can't be subtracted to find one.

Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m1, in this epic, since `main` must not reach the cut carrying accidental per-frame cost.

- Measure as #7505 did: `--profile profiling` builds with frame pointers and `moq-relay/io-uring,moq-bench/uring`, one target directory per revision, `perf stat` `cpu_core` instructions (user and kernel) over the steady window, divided by the frames the generator's 500 ms reports count over the same window. 4 io_uring workers, not 1, so chat does not saturate a worker. Pin the relay and generator cores so a loaded host still reads instruction deltas.
- Bisect `cdff3c8f95..main` on chat over io_uring first (the largest rise and the tightest ranges), with a step size that keeps each build's cost bounded, then confirm the culprits on video and on the tokio workers. A rise that shows on io_uring but not on the tokio workers is attributed on its own, not folded into a shared culprit.
- Attribute each step with `perf record` at a fixed period, as #7505 did, before deciding fix or accept. Record accepted costs in the epic's Plan with the feature they pay for.
- Commit the driver this needs as a deterministic mode of `just bench-profile` or `just bench-runtime` (pinned cores, instructions per delivered frame, fixed-period samples) instead of another throwaway script; #7505's driver on `bench/relay.sh` was not committed.
- [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) covers the bimodal futex cost on the fanout workload; don't fix it twice. A bisect step that lands on the same lock goes there.

Public API: none expected. Wire: none.

## Related

- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - the nightly series that should have caught this growth
- [Performance audit](/quest/m2/audit-perf.md) - counts instructions when bisecting, for the same reason
11 changes: 11 additions & 0 deletions quest/m1/vendor-sync/uniffi.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,14 @@ First run, check these first:
[`d1f766f4`](https://github.com/mozilla/uniffi-rs/commit/d1f766f4) (Kotlin
Throwable handling), and [`f903c36c`](https://github.com/mozilla/uniffi-rs/commit/f903c36c)
(Python prefers the running event loop).

Fork CI, added 2026-11-01 (maintainer, follow-up planning after the
2026-11-01 /quest-afk run): kixelated/uniffi-dart's `Testing`
(`.github/workflows/tests.yml`) and `Test Downstream`
(`.github/workflows/test-downstream.yml`) workflows failed in all 14 of their
runs among the fork's 30 most recent PR runs that day, while `Nix` passed, so
a red check on a fork PR, including a vendor sync's own, says nothing. In `Testing`, `Test Suite (stable)`
failed at `Run cargo nextest` while nightly passed and 1.91 was cancelled.
Fix them, or drop them where the fork's `Nix` job and moq's own binding
checks already cover what they test, and record which in the fork's README.
Keep the fork's CI read-only, as `quest guide`'s Reviews section requires.
9 changes: 9 additions & 0 deletions quest/m1/vendor-sync/vaapi.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,12 @@ at `59384456`, five commits ahead, of which `59384456` (a VP9 struct field for
libva 2.23+) is the only fix and moq-vaapi is H.264-only, and
chromeos/cros-codecs was at `a050c6cd`, one commit ahead. Record the commits
checked in moq-dev/vaapi's README, and copy them here for the next run.

CI, added 2026-11-01 (maintainer, follow-up planning after the 2026-11-01
/quest-afk run): moq-dev/vaapi's only workflow, `.github/workflows/check.yml`,
sets `id-token: write` at the workflow level and runs on `pull_request`, so
every PR's check job can mint an OIDC token without a protected environment.
CI on pull requests must stay read-only. Remove it, leaving
`contents: read`; if a step needs OIDC (`magic-nix-cache-action` is the likely one), confirm
it still works without it, and otherwise move that step to a job limited to
`main` behind a protected environment.
Loading