Repository navigation
quest: plan follow-ups from the 2026-11-01 /quest-afk run #7568
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
6d9f321
quest: plan follow-ups from the 2026-11-01 /quest-afk run
kixelated 2a35da6
Merge remote-tracking branch 'origin/main' into afk-followups-iter
kixelated 685c6ff
quest: address review findings on the 2026-11-01 follow-ups
kixelated e0a4b5d
quest: keep an io_uring-only cost open in relay-cpu-bisect
kixelated bd93ba8
quest land: merge tested base
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # [XS] Every io_uring test runs in the nextest io-uring group | ||
|
|
||
| ## Goal | ||
|
|
||
| Every test that opens an io_uring, including the relay's `wire_stats::uring_*`, `peer_limits::uring_*`, and `refusals::uring_*` integration tests and the `bitrate_caps::uring_*` tests [moq-dev/moq#7535](https://github.com/moq-dev/moq/pull/7535) adds, runs inside the `io-uring` nextest test group, so the tests of one `just ci` running in parallel no longer exhaust the user's shared `RLIMIT_MEMLOCK`. The group caps rings only within one nextest invocation; separate invocations still share the budget, as `rs/moq-uring/README.md` documents, and this quest does not change that. An io_uring test that still hits too low a limit fails with a message naming `RLIMIT_MEMLOCK` and how to raise it, not a generic setup error or a timeout. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found by [moq-dev/moq#7535](https://github.com/moq-dev/moq/pull/7535) and [moq-dev/moq#7211](https://github.com/moq-dev/moq/pull/7211): #7535's io_uring tests passed serially but could hit the shared limit in parallel, and #7211's `just ci` failed once in `uring_peer_session_widens_quic_limits` on `RLIMIT_MEMLOCK`. | ||
|
|
||
| Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m0, since each such failure fails a whole merge-gate batch. | ||
|
|
||
| - The group's filter in `.config/nextest.toml` names only `runtime_uring::` and `embed::.*uring` in `moq-relay::integration`. Match every io_uring test instead, for example `test(/uring/)` in that binary plus moq-relay's own unit tests that start ring workers, so a new test joins by its name. Check that the inventory of uring tests (`rg 'fn uring_' rs/moq-relay`) all land in the group with `cargo nextest show-config test-groups`, which lists each group's members; plain `cargo nextest list` shows no group. | ||
| - moq-uring already names the limit on a ring's `ENOMEM` (`rs/moq-uring/src/error.rs`). Check that this error reaches the failing test's output on the relay paths (worker startup, `start_relay`) instead of being wrapped or swallowed; if a test sees only a timeout, surface the startup error there. | ||
| - Keep the group's `max-threads` unless measured otherwise; the limit is shared with other checkouts, which `rs/moq-uring/README.md` documents. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [More tests under load](/quest/m0/test-flakes-2/README.md) - the flake epic this would otherwise land in | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # [XS] A truncated MKV ends its tracks at finish | ||
|
|
||
| ## Goal | ||
|
|
||
| When the MKV importer's `finish()` refuses truncated input (`Mkv(Truncated)`), it still ends its tracks with what was accepted, as it does after a refused `decode`, so a subscriber's `read()` returns instead of waiting until the importer is dropped. `finish()` still returns the error, and the frame cut short still fails its group with the reason. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found by the verification in [moq-dev/moq#7515](https://github.com/moq-dev/moq/pull/7515), which fed the live `opus.webm` truncated at every length and with every byte flipped: 1462 truncation lengths and 29 byte flips ended with tracks neither finished nor aborted after `finish()` returned `Truncated`, at every chunk size. Callers that drop the importer after an error were unaffected. | ||
|
|
||
| In `Import::finish` (`rs/moq-mux/src/container/mkv/import.rs`, about lines 838-852), a failed `drain_eof` aborts only the streaming block's group and returns early, skipping the per-track finish. A `decode` that already failed (`self.failed`) skips `drain_eof` and still reaches the track loop, so it ends its tracks. | ||
|
|
||
| Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m1, a small fix in an importer the cut ships. End the tracks on the truncation path the way a refused `decode` already does (`decode`'s docs: "`finish` ends the tracks with what was accepted"), so the two refusals behave alike. #7515 also floated aborting every track with the truncation error instead; matching the refused `decode` keeps one rule and keeps the groups that did arrive. Test with #7515's harness shape: a truncated `opus.webm` whose subscriber's read ends after `finish()` while the importer is still held. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Untrusted media parsers have fuzz targets](/quest/m2/media-fuzz.md) - could assert that every track ends after `finish()`, whatever the input |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # [S] The shared pacer holds its rate under 2 Mbit/s | ||
|
|
||
| ## Goal | ||
|
|
||
| Under 2 Mbit/s, a rate-paced controller on moq-quic (BBR3 today, and any candidate that sets a pacing rate) sends at its pacing rate, not up to twice it. BBR3 is retuned in the same PR so the nightly probe scenarios, `competing_cubic` included, stay within their late-frame and overhead budgets. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found by [moq-dev/moq#7546](https://github.com/moq-dev/moq/pull/7546) (the GCC candidate). Below 2 Mbit/s the pacer's burst budget is one packet, and moq-quic checks a datagram's pace while the packet before it is still being built, so a batch of two leaves on one packet's credit. The shared pacer (`rs/moq-congestion/src/pacer.rs`) then forgets the overshoot instead of charging it to the next send. #7546 fixed it by carrying the overshoot as debt (its commit `b62e538b94`), which moved BBR3's nightly `competing_cubic` (`rs/moq-quic/src/tests/probe.rs`) to 441.5 late frames against an allowed 439.4, so it reverted the fix rather than retune BBR3 there. | ||
|
|
||
| Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m1, because the next release ships BBR3 as moq-quic's default and a 2x overshoot at low rates is a correctness bug in what it sends. Land the pacer fix and BBR3's retune together, so the nightly never runs one without the other. | ||
|
|
||
| - Start from `b62e538b94`'s debt approach, with a unit test in `pacer.rs` that a batch outrunning a one-packet budget is charged to the next send, and the existing `aggregate_throughput_matches_controller_pacing_rate` swept below 2 Mbit/s. | ||
| - Retune where the change shifts BBR3's probing, not the budgets: the probe scenarios' budgets judge the probe, and loosening them to pass would hide the fix's effect. Report every probe scenario's paired late frames, p99, and overhead before and after, on the scenario's full seed set. | ||
| - `competing_cubic`'s 8-seed late budget is fragile on its own ([moq-dev/moq#7538](https://github.com/moq-dev/moq/pull/7538) found 28% of 8-seed windows fail it); its seed count is planned separately in [moq-dev/moq#7440](https://github.com/moq-dev/moq/pull/7440). Judge this change on a seed set large enough to separate it from that noise, not on whether seeds 1-8 pass. | ||
| - Re-run the realtime candidates' rows that pace below 2 Mbit/s ([GCC](https://github.com/moq-dev/moq/pull/7546), [BBR3 delay bound](https://github.com/moq-dev/moq/pull/7544), [SCReAMv2](https://github.com/moq-dev/moq/pull/7547)) for whichever have landed, and record the shift in the [epic](/quest/m2/quic-realtime/README.md). | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Realtime uplink congestion control](/quest/m2/quic-realtime/README.md) - every candidate paces through this pacer | ||
| - [Probe up to a target bitrate](/quest/m2/probe/README.md) - its scenario budgets are what the retune must keep |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # [M] Find and fix the relay's per-frame CPU growth since #5926 | ||
|
|
||
| ## Goal | ||
|
|
||
| The commits that raised the relay's instructions per delivered frame between [`cdff3c8f95`](https://github.com/moq-dev/moq/commit/cdff3c8f95) ([moq-dev/moq#5926](https://github.com/moq-dev/moq/pull/5926) as merged) and `main` are found, each rise is attributed to a cause, and the avoidable costs are fixed. A rise that pays for a deliberate feature is accepted with its reason in the [release-perf epic](/quest/m1/release-perf/README.md). | ||
|
|
||
| ## Plan | ||
|
|
||
| Found by [moq-dev/moq#7505](https://github.com/moq-dev/moq/pull/7505), which profiled the io_uring clock read. At `main` [`30f23c3f8a`](https://github.com/moq-dev/moq/commit/30f23c3f8a), relay instructions per delivered frame had risen 25-56% on io_uring since `cdff3c8f95` and 18-38% on the tokio workers since `1dd9b83d1`, with the same delivery: | ||
|
|
||
| | workload | runtime | before, kinstr/frame | `30f23c3f8a`, kinstr/frame | | ||
| |---|---|---|---| | ||
| | chat | io_uring | 86.3 [84.5-87.4] at `cdff3c8f95` | 134.8 [128.5-151.3] | | ||
| | video | io_uring | 108.7 [106.6-116.2] at `cdff3c8f95` | 136.1 [132.6-152.8] | | ||
| | chat | tokio workers | 104.0 [101.4-105.0] at `1dd9b83d1` | 143.1 [127.5-205.3] | | ||
| | video | tokio workers | 141.6 [139.3-148.3] at `1dd9b83d1` | 167.1 [164.9-173.6] | | ||
|
|
||
| Medians of 5 interleaved reps, min-max in brackets. The tokio rows start at [`1dd9b83d1`](https://github.com/moq-dev/moq/commit/1dd9b83d19), #5926's base, since #7505 measured the tokio workers only there and at `main`. Both runtimes moved, so at least one cause is shared code (moq-net's model, kio, or the session layer). That does not rule out a further io_uring-only cost, such as the clock read #7505 profiled. The tokio range also spans #5926 itself, so the two rises can't be subtracted to find one. | ||
|
|
||
| Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m1, in this epic, since `main` must not reach the cut carrying accidental per-frame cost. | ||
|
|
||
| - Measure as #7505 did: `--profile profiling` builds with frame pointers and `moq-relay/io-uring,moq-bench/uring`, one target directory per revision, `perf stat` `cpu_core` instructions (user and kernel) over the steady window, divided by the frames the generator's 500 ms reports count over the same window. 4 io_uring workers, not 1, so chat does not saturate a worker. Pin the relay and generator cores so a loaded host still reads instruction deltas. | ||
| - Bisect `cdff3c8f95..main` on chat over io_uring first (the largest rise and the tightest ranges), with a step size that keeps each build's cost bounded, then confirm the culprits on video and on the tokio workers. A rise that shows on io_uring but not on the tokio workers is attributed on its own, not folded into a shared culprit. | ||
| - Attribute each step with `perf record` at a fixed period, as #7505 did, before deciding fix or accept. Record accepted costs in the epic's Plan with the feature they pay for. | ||
| - Commit the driver this needs as a deterministic mode of `just bench-profile` or `just bench-runtime` (pinned cores, instructions per delivered frame, fixed-period samples) instead of another throwaway script; #7505's driver on `bench/relay.sh` was not committed. | ||
| - [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) covers the bimodal futex cost on the fanout workload; don't fix it twice. A bisect step that lands on the same lock goes there. | ||
|
|
||
| Public API: none expected. Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - the nightly series that should have caught this growth | ||
| - [Performance audit](/quest/m2/audit-perf.md) - counts instructions when bisecting, for the same reason |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.