From f22df581df284207901096e1321601ec7bcff625 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 16:31:36 -0700 Subject: [PATCH 1/5] quest: plan main-vs-release perf regressions and the release resubscribe hang Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 1 + quest/m0/release-resubscribe-hang.md | 25 +++++ quest/m1/README.md | 1 + quest/m1/release-perf/README.md | 108 +++++++++++++++++++++ quest/m1/release-perf/compare-harness.md | 22 +++++ quest/m1/release-perf/fetch-fanout.md | 44 +++++++++ quest/m1/release-perf/gc-idle.md | 26 +++++ quest/m1/release-perf/js-frames.md | 30 ++++++ quest/m1/release-perf/recheck.md | 66 +++++++++++++ quest/m1/release-perf/resume-duplicates.md | 33 +++++++ quest/m1/release-perf/sparse-gaps.md | 37 +++++++ quest/m1/release-perf/stats-report.md | 42 ++++++++ 12 files changed, 435 insertions(+) create mode 100644 quest/m0/release-resubscribe-hang.md create mode 100644 quest/m1/release-perf/README.md create mode 100644 quest/m1/release-perf/compare-harness.md create mode 100644 quest/m1/release-perf/fetch-fanout.md create mode 100644 quest/m1/release-perf/gc-idle.md create mode 100644 quest/m1/release-perf/js-frames.md create mode 100644 quest/m1/release-perf/recheck.md create mode 100644 quest/m1/release-perf/resume-duplicates.md create mode 100644 quest/m1/release-perf/sparse-gaps.md create mode 100644 quest/m1/release-perf/stats-report.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 31a9b165b09..055cc49a114 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -87,6 +87,7 @@ Each lands its fixes in its own PR and plans only what needs a decision. - [qmux 0.5.3](/quest/m0/qmux-0.5.3.md) - crates.io publishes qmux 0.5.3 with the builder session config the release backport of #5609 needs - [Release shutdown backport](/quest/m0/ffi-shutdown-backport.md) - `release` publishes a non-breaking moq-ffi with an async `drain()` and qmux sessions on the QUIC stream limit - [Release fixed delay](/quest/m0/release-fixed-delay.md) - a fixed latency on `release`, which moq.dev runs, ignores the catalog's jitter and delay as `main` already does +- [Release resubscribe hang](/quest/m0/release-resubscribe-hang.md) - a moq-lite-06 subscription that closes and reopens on `release` is proven to reach the relay, or the hang is fixed and backported - [Publisher stall estimate](/quest/m0/publish-stall-estimate.md) - a live browser publish holds its viewers at the Auto target instead of ratcheting their buffer up with every stall - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription - [Performance audit](/quest/m0/audit-perf.md) - benchmarks that regressed since the last run are explained, fixed, or planned, and unbenched hot paths get coverage diff --git a/quest/m0/release-resubscribe-hang.md b/quest/m0/release-resubscribe-hang.md new file mode 100644 index 00000000000..49d4963a948 --- /dev/null +++ b/quest/m0/release-resubscribe-hang.md @@ -0,0 +1,25 @@ +# [S] Release resubscribe hang + +## Goal + +It is settled whether `release` has a moq-lite-06 bug where a subscription that +is closed and reopened never reaches the relay. If the bug is real, a fix with +a regression test is backported to `release`, which moq.pro runs. If it is a +harness artifact, the reason is recorded and this quest is deleted. + +## Plan + +Found while backporting main's benches to `release` in +[#6103](https://github.com/moq-dev/moq/pull/6103): `session_churn_*/moq-lite-06` +hangs on `release`. After one subscription opens and closes, the reopened +subscription never reaches the relay. The same bench passes on main. + +1. Reproduce it on `origin/release` as a session test (one subscriber; open, + close, and reopen the same track through a relay). Mock time, not sleeps. +2. If it hangs, find the cause, and check whether main fixed it on purpose. + Bisect main for the fix and backport it to `release` with the test. + Otherwise, fix it on `release` directly. Either way, keep the test on main + too. +3. If the test passes on `release`, the bench harness's backport is at fault + (release's `tests/support` mock differs from main's). Record that in the PR + and delete this quest. diff --git a/quest/m1/README.md b/quest/m1/README.md index 05d81b035a2..ba573629b02 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -121,6 +121,7 @@ lands before the cut. - [moq-net compiles once](/quest/m1/net-erase-transport.md) - the session drivers stop monomorphizing per transport and per crate, saving several MiB of the relay, FFI library, `libmoq` and CLI - [moq-net internals](/quest/m1/net-internals.md) - no per-group session clones, no transport type parameter in lite and IETF internals, and the `Local` check measured +- [Main is no slower than release](/quest/m1/release-perf/README.md) - every benchmark where main is slower than release is fixed or accepted with a reason, checked by a final paired run before the cut - [Subscribe update churn](/quest/m1/subscribe-update-churn.md) - a viewer sends SUBSCRIBE_UPDATE only on a meaningful change, measured after the publisher stall fix diff --git a/quest/m1/release-perf/README.md b/quest/m1/release-perf/README.md new file mode 100644 index 00000000000..2c6bbfad5fc --- /dev/null +++ b/quest/m1/release-perf/README.md @@ -0,0 +1,108 @@ +# Main is no slower than release + +## Goal + +Before the next release cut, every benchmark where `main` is slower than +`release` is either fixed or accepted with a recorded reason. Accidental cost +is fixed. Cost that pays for a feature `main` deliberately added is accepted, +and the reason is written here. + +This epic's own work: once every child has landed, run the full main against +release comparison again with `just bench origin/release`, and confirm nothing +new regressed. Then add a release-baseline step to the recurring +[performance audit](/quest/m0/audit-perf.md) (its m1 successor, once +[moq-dev/moq#6112](https://github.com/moq-dev/moq/pull/6112) merges), so every +later cut repeats this comparison. + +## Plan + +Decided 2026-10-21, when planning from a paired comparison of `release` +(`0c56e5b4e`, which included the bench backport +[#6103](https://github.com/moq-dev/moq/pull/6103)) against `main` +(`8b0d9952f`). The run used 811 paired Criterion cases, three alternating +confirmation rounds for the largest moves, JS on bun, and the relay load +harness. It ran on a shared 8-core host, so a small wall-time change may be +noise. + +- The epic gates the cut through the [m1 milestone](/quest/m1/README.md), + because `main` must not ship slower than `release` without a decision. +- In scope: the five cases that held through confirmation, plus the large + screen-only moves. The screen-only moves are re-measured with callgrind + instruction counts, and any that hold are fixed. +- Only accidental cost is fixed. Cost from an intended feature is accepted, + and the reason is recorded here. +- The IETF `/fetch` rows of `session_delivery_group_cost` are accepted and + dropped from release pairing. They measure different work on each branch: + release's bare track handle opened a live SUBSCRIBE, while main sends a real + FETCH per viewer ([#5820](https://github.com/moq-dev/moq/pull/5820) made + handles lazy, and [#4974](https://github.com/moq-dev/moq/pull/4974) added + fetch-only demand). The lite `/fetch` cost is different: it is real, and + [FETCH fan-out](/quest/m1/release-perf/fetch-fanout.md) owns it. +- `stats/session-report` is accepted at O(live sessions), after a cheap fix. + Per-label payload bytes ([#5731](https://github.com/moq-dev/moq/pull/5731)) + let an embedder bucket sessions by label and match payload billing per + bucket, and the drain runs once a second, using under 0.01% of a core at + 10,000 sessions. Deleting those bytes was rejected. + +### Release as a baseline + +`release` lacks many of the features the backported benches exercise: +per-reader delivery records, wall-clock aging, the gap ledger, rewind checks, +and epochs. So a ratio against `release` shows where to look, not what to fix. +Main's own sweeps over a size parameter are the better regression signal: a +cost that grows with the sweep is accidental, while a flat per-operation cost +usually pays for a feature. + +### Confirmed (three rounds, ratio is main / release) + +| Bench | release | main | Ratio | Owner | +|---|---|---|---|---| +| `track_gap_flush/4096` | 5.7 us | 551 us | 96x | [Sparse gaps](/quest/m1/release-perf/sparse-gaps.md) | +| `track_gap_flush/512` | 1.8 us | 48 us | 27x | same | +| `track_gap_flush/64` | 1.5 us | 6.2 us | 4.1x | same | +| `stats/session-report/1000x10` | 11.0 us | 86.8 us | 7.9x | [Stats report](/quest/m1/release-perf/stats-report.md) | +| `track_gc/idle/16384` | 11.6 ns | 73.7 ns | 6.3x | [Idle GC](/quest/m1/release-perf/gc-idle.md) | +| `origin/resume_overlap/sequence/1r_64g_32c` | 93 us | 426 us | 4.6x | [Resume duplicates](/quest/m1/release-perf/resume-duplicates.md) | +| `session_delivery_group_cost/moq-transport-22/.../viewers=64/.../fetch` | 1.19 ms | 3.86 ms | 3.2x | accepted (above) | +| `group_write_frames/single/8192` | 424 us | 486 us | 1.17x | [Re-check](/quest/m1/release-perf/recheck.md) | + +Main is much faster on subscriber churn, joins, range demand, and idle +sessions (by 100x to 3000x). The relay load test used 21% less CPU on video +and 12% less on fanout, with the same delivery. + +### Screen only (one run, unconfirmed) + +- moq-mux: `container-export/ts/4x1` 4.7x and `mkv/4x1` 3.9x (median 1.41x + over 27 cases); `annexb-import-{bytes,frames}` median 2.1x (up to 3.6x). +- moq-net track: `track_stamp_groups` median 2.7x (4.7x at `sparse/4096`), + `track_sparse_read/timed/4096` 4.3x, `track_datagram_subscriber_join` + median 2.3x, `track_parallel_write` 1.6x, `track_fanout_frame` 1.42x, + `track_empty_run` median 1.36x. +- moq-net origin and session: `origin/announce_duplicate/30d_1s` 4.3x, + `session_delivery_relays` median 1.56x, `paced` 1.45x, `scale` 1.43x. +- moq-stats `aggregate` at live=16, departed=4096: 2.1x. The other rows were + 1.0 to 1.3x, so this one is probably noise. +- kio `contended_notify` and `contended_log` at 16 threads: 4 to 5x. These + are contention benches on a busy host, so they are likely noise. +- JS `net/bench/frames.ts` on bun: lite rows 1.2 to 1.3x, IETF rows 1.3 to + 2.2x ([JS frames](/quest/m1/release-perf/js-frames.md)). + +The raw report lived in a scratch directory that has since been removed. The +[comparison harness](/quest/m1/release-perf/compare-harness.md) keeps the next +one. + +## Required + +- [FETCH fan-out](/quest/m1/release-perf/fetch-fanout.md) - a relay serves later viewers' lite FETCHes of a group from its cache, so per-viewer cost falls with viewers again +- [Sparse gaps](/quest/m1/release-perf/sparse-gaps.md) - an arrival and a gap flush cost O(log gaps), not O(gaps) +- [Resume duplicates](/quest/m1/release-perf/resume-duplicates.md) - an ordered resume skips copies of an already-delivered group without consuming them +- [Idle GC](/quest/m1/release-perf/gc-idle.md) - an idle track GC skips the wall-clock pass when nothing is due +- [Stats report](/quest/m1/release-perf/stats-report.md) - the stats drain locks each row once and its bench counts the wire walk; per-label bytes stay, at an accepted O(live sessions) +- [Re-check](/quest/m1/release-perf/recheck.md) - the screen-only moves are re-measured with instruction counts, and the ones that hold are fixed or accepted +- [JS frames](/quest/m1/release-perf/js-frames.md) - the js/net frame path is no slower than release in a browser, or the gap is accepted +- [Comparison harness](/quest/m1/release-perf/compare-harness.md) - `just bench ` pairs full Criterion IDs and keeps its report + +## Related + +- [Performance audit](/quest/m0/audit-perf.md) - compares nightly to nightly; this epic compares main to release +- [Benchmark regressions in CI](/quest/m2/bench-ci.md) - PR-time comparison against the base branch diff --git a/quest/m1/release-perf/compare-harness.md b/quest/m1/release-perf/compare-harness.md new file mode 100644 index 00000000000..9ae3dd0aec6 --- /dev/null +++ b/quest/m1/release-perf/compare-harness.md @@ -0,0 +1,22 @@ +# [S] Comparison harness + +## Goal + +`just bench ` (`bench/run.sh`) pairs every Criterion case by its full +ID, and writes its report somewhere that survives the worktree. The +[epic](/quest/m1/release-perf/README.md)'s final main against release run +uses it without hand-written joins. + +## Plan + +`bench/run.sh` discovers cases from `cargo bench -- --list`, which truncates +display titles at 100 characters. That loses long `session_*` IDs. The +2026-10-21 run worked around it by joining the saved `benchmark.json` +`full_id` values instead (`run.sh` already reads them after the run). Discover +and pair cases from those full IDs. + +The same run kept its report in another worktree's `.scratch/`, and lost it +when that worktree was removed. Write the report under a path the caller +chooses. Default to the current worktree's `.scratch/`, and print its path at +the end. A durable copy belongs with the epic's final run, not in the repo. +Extend `bench/compare.test.py` to cover a case ID longer than 100 characters. diff --git a/quest/m1/release-perf/fetch-fanout.md b/quest/m1/release-perf/fetch-fanout.md new file mode 100644 index 00000000000..536d3c4a2b7 --- /dev/null +++ b/quest/m1/release-perf/fetch-fanout.md @@ -0,0 +1,44 @@ +# [M] FETCH fan-out + +## Goal + +When many viewers FETCH the same group through a relay over moq-lite, the +relay fetches it upstream once and serves later viewers from its cache. Cost +per viewer falls as viewers grow, as it does on `release`. + +## Plan + +Measured 2026-10-21 (`session_delivery_group_cost/.../fetch` on lite-06, +cost per viewer): + +| Viewers | release | main nightly | +|---|---|---| +| 1 | 57 us | 67 us | +| 16 | n/a | 61 us | +| 64 | 16.5 us | 59 us | + +Main's non-fetch path still amortizes (55 us down to 34 us), so on main every +viewer's FETCH appears to cost as much as the first. The relay seems to go +upstream or rebuild state for each fetch. Lite's `IDLE_LINGER` should keep +the relay's copy and cache, so a simple "the track was dropped" explanation is +not enough. + +1. Reproduce first. Count the upstream FETCHes the publisher session sees in + each round. One per group means the cache works and the cost is elsewhere. + One per viewer means the cache is bypassed. +2. If it is bypassed, compare callgrind instruction counts at the parents of + the suspects. The likely fix is for the relay's FETCH serve to resolve its + track once and reuse the lingering copy. Suspects: + [#5820](https://github.com/moq-dev/moq/pull/5820) (the relay's FETCH serve + resolves `broadcast.track(..).fetch_group` per request), + [#5809](https://github.com/moq-dev/moq/pull/5809) (a fetch keeps its track + read), [#5680](https://github.com/moq-dev/moq/pull/5680) (fetch age + budgets; the bench advances the cache clock 1 s per round), and + [#6020](https://github.com/moq-dev/moq/pull/6020) (one writer per group). +3. Land a regression test that asserts one upstream FETCH per group across + many viewers. Also check that the `session_delivery_group_cost` lite + `/fetch` sweep over viewers slopes down again. + +A relay that fetches upstream once per viewer turns N viewers into N upstream +requests. If step 1 confirms that, raise the priority with the maintainer, +since m0 bounds what a peer can make the relay do. diff --git a/quest/m1/release-perf/gc-idle.md b/quest/m1/release-perf/gc-idle.md new file mode 100644 index 00000000000..0e4e2d5e7ad --- /dev/null +++ b/quest/m1/release-perf/gc-idle.md @@ -0,0 +1,26 @@ +# [S] Idle GC + +## Goal + +`track::Pool::advance` with nothing due does no clock read, lock, or waker +registration. `track_gc/idle` returns to a few nanoseconds. + +## Plan + +Found by reading code on 2026-10-21 (11.6 ns to 74 ns, flat across sizes, +confirmed). [#5548](https://github.com/moq-dev/moq/pull/5548) (wall-clock +aging of superseded groups) added the following to every `advance` in +`rs/moq-net/src/model/cache.rs`: + +- the `wall_epoch` `OnceLock` and a `wall_micros.fetch_max`; +- `release_due(self.now_wall(), driver)`, which reads `moq_clock::now()`; +- the `inner.wall` lock and `driver.register`. + +Part of the cost is the bench itself: it builds a fresh `kio::Waiter::noop()` +on every iteration, so each `register` allocates and frees an `Arc`. A +real driver reuses its waiter. Hoist the waiter out of `b.iter` first, and +measure what remains. + +Fix: keep the earliest wall deadline in an atomic, as `expiry.rs` already +does with `earliest`. Read `now_wall()` and take the lock only when that +deadline has passed. Time decisions use `max(wall, pts)`; keep that behavior. diff --git a/quest/m1/release-perf/js-frames.md b/quest/m1/release-perf/js-frames.md new file mode 100644 index 00000000000..ee9c0b7f43b --- /dev/null +++ b/quest/m1/release-perf/js-frames.md @@ -0,0 +1,30 @@ +# [M] JS frames + +## Goal + +The js/net frame path (`js/net/bench/frames.ts`) is no slower than +`release` when measured in a browser, or the gap is accepted with a reason +recorded in the [epic](/quest/m1/release-perf/README.md). + +## Plan + +On bun (2026-10-21), the lite rows were 1.2 to 1.3x slower and the IETF rows +1.3 to 2.2x. JS perf is judged in a browser, so measure there first, against +`release` (the bench file is the same on both branches). + +Leads from reading code: + +- Lite and IETF: [#4226](https://github.com/moq-dev/moq/pull/4226) turned + `Signal.race` into the lazy `Changed` class. Each wait allocates a `Changed`, + the `then` getter builds an extra Promise, and `#finish` adds a microtask + hop. Every reader wake in `group.ts` `#changed()` pays for all of that. + Resolve listeners synchronously when the winner is not a thenable, and add a + race-heavy case to `js/signals/bench/effect.ts`. +- IETF: per-group `track.writeGroup` to `ordered.nextGroup` cost. Candidates + are [#5803](https://github.com/moq-dev/moq/pull/5803), + [#5725](https://github.com/moq-dev/moq/pull/5725), + [#5836](https://github.com/moq-dev/moq/pull/5836), + [#5930](https://github.com/moq-dev/moq/pull/5930), and the eager `new Error()` + plus `FinalizationRegistry` per `Effect` from + [#4876](https://github.com/moq-dev/moq/pull/4876), if a path creates + Effects per group. diff --git a/quest/m1/release-perf/recheck.md b/quest/m1/release-perf/recheck.md new file mode 100644 index 00000000000..15f6ef46dba --- /dev/null +++ b/quest/m1/release-perf/recheck.md @@ -0,0 +1,66 @@ +# [M] Re-check + +## Goal + +Every screen-only move in the [epic](/quest/m1/release-perf/README.md) is +re-measured with instruction counts. Each one that holds is fixed in this +quest's PR, one commit per fix, or accepted with a reason recorded in the +epic. + +## Plan + +Use the instruction-count method from the [performance audit](/quest/m0/audit-perf.md). +Wall time on a shared host can't tell these apart. Build each bench at +`origin/release` and at main, and compare instructions per iteration. A +change that needs a maintainer decision, or is too large to review alongside +the rest, becomes a follow-up through `/quest-plan`. + +Leads from reading code on 2026-10-21, strongest first: + +1. moq-mux `container-export/{ts,mkv}/4x1` (4.7x, 3.9x). The worst ratio is at + one frame per track, so look for a fixed cost per export or per track. The + reap cost from [#5942](https://github.com/moq-dev/moq/pull/5942) that the + audit found grows with buffered groups instead. Suspects: every export now + goes through `jitter::Buffer` + ([#4645](https://github.com/moq-dev/moq/pull/4645), + [#5309](https://github.com/moq-dev/moq/pull/5309)), and `container/source.rs` + start-up ([#5947](https://github.com/moq-dev/moq/pull/5947), + [#5854](https://github.com/moq-dev/moq/pull/5854)). If the jitter path + dominates, skip it when the delay is zero. +2. `annexb-import` (median 2.1x). Check `group_*` and `append_group` first. If + they regressed by the same factor, the cost is per group in moq-net + ([#5803](https://github.com/moq-dev/moq/pull/5803) rewind check, + [#5725](https://github.com/moq-dev/moq/pull/5725) largest notifications, + [#6020](https://github.com/moq-dev/moq/pull/6020)), not in the importer. +3. `track_stamp_groups` (median 2.7x). The first frame on a timed track now + takes the track lock and `bounds`, `starts`, and `sequence` mutexes + ([#5803](https://github.com/moq-dev/moq/pull/5803), + [#6074](https://github.com/moq-dev/moq/pull/6074)). Pass the checked start + into `first_frame`, merge `bounds` and `starts` under one lock, and skip + `advance_largest`'s write and wake when nothing changed. +4. `track_datagram_subscriber_join` (median 2.3x). `Cursor::new` scans the + datagram ring twice, and each join builds a placeholder `Published` that is + immediately rebuilt ([#5195](https://github.com/moq-dev/moq/pull/5195), + [#5929](https://github.com/moq-dev/moq/pull/5929)). Track the newest + datagram timestamp on push, and build `Published` once. +5. `track_parallel_write`, `track_fanout_frame`, `track_empty_run`, + `group_write_frames` (1.2 to 1.6x). This is constant per-group and + per-frame cost from several features. Resolve `now` once per poll in + `Drift`, and skip no-op `advance_largest` writes. Accept the rest. +6. `origin/announce_duplicate/30d_1s` (4.3x). Compare against `1d_1s`, to + tell a constant cost per change from a cost per duplicate. Leads: an + epochless winner switch is now a `Restart` + ([#5087](https://github.com/moq-dev/moq/pull/5087), + [#5443](https://github.com/moq-dev/moq/pull/5443), + [#4942](https://github.com/moq-dev/moq/pull/4942), + [#5144](https://github.com/moq-dev/moq/pull/5144)). +7. `session_delivery_{relays,paced,scale}` (1.4 to 1.6x). Expected to be the + per-group cost above plus the flow-controlled mock transport + ([#5530](https://github.com/moq-dev/moq/pull/5530)). Re-measure after the + fixes above, and accept what remains. +8. moq-stats `aggregate` at live=16, departed=4096, and kio + `contended_{notify,log}` at 16 threads: confirm they are noise. If the + aggregate row holds, suspect [#4874](https://github.com/moq-dev/moq/pull/4874) + and [#4904](https://github.com/moq-dev/moq/pull/4904). + +Add a bench sweep wherever a fix removes a slope, so the slope stays visible. diff --git a/quest/m1/release-perf/resume-duplicates.md b/quest/m1/release-perf/resume-duplicates.md new file mode 100644 index 00000000000..86b79a9aca4 --- /dev/null +++ b/quest/m1/release-perf/resume-duplicates.md @@ -0,0 +1,33 @@ +# [M] Resume duplicates + +## Goal + +An ordered resume reader skips a route copy's groups that another copy +already delivered, without consuming them. `origin/resume_overlap/sequence` +costs about one cheap skip per duplicate again. + +## Plan + +Found by reading code on 2026-10-21 (4.6x at `1r_64g_32c`, confirmed). + +- Release: each segment did a non-consuming `poll_seek_group(floor, cap)`, so + a duplicate below the floor cost nothing. +- Main: each copy has its own `track::Subscriber` (from + [#4741](https://github.com/moq-dev/moq/pull/4741)), and the `Delivered` + dedupe comes from [#5446](https://github.com/moq-dev/moq/pull/5446). For + each group handed out, `resume::Reader::poll_lowest` peeks all 32 copies, + and [#6019](https://github.com/moq-dev/moq/pull/6019) added peeking every + copy (its own PR measured about 1.3x of the cost). For each copy whose head + was already delivered, it then fully consumes the head with + `sub.poll_next_group`. That runs the meter, the inner `hand_out`, + `forget_past_budget`, `count_stale`, and builds a group consumer, only to + discard it. + +Fix: in ordered mode, move a copy's cursor up to the floor with a seek that +builds no consumer, meter entry, or `hand_out`, as release did per segment. +Cache each copy's peek across polls and re-peek only the copy that delivered. +#6019 also lists skipping peek-then-take when only one live copy remains; do +that here if it falls out of the same change. + +Before fixing, check whether `origin/arrival/1r_64g_32c` regressed by the +same factor. If it didn't, that confirms the lost seek is the cause. diff --git a/quest/m1/release-perf/sparse-gaps.md b/quest/m1/release-perf/sparse-gaps.md new file mode 100644 index 00000000000..8b7a74e03d1 --- /dev/null +++ b/quest/m1/release-perf/sparse-gaps.md @@ -0,0 +1,37 @@ +# [M] Sparse gaps + +## Goal + +On a track with many open gaps, recording an arrival and flushing gaps cost +O(log gaps). `track_sparse_read/timed` and `track_gap_flush` stop growing +faster than their size parameter. + +## Plan + +Found by reading code on 2026-10-21. Confirm each step with callgrind +instruction counts at two sizes before and after the change. + +- `Ledger` (`rs/moq-net/src/model/ledger.rs`) stores its runs in a `Vec`. + [#5802](https://github.com/moq-dev/moq/pull/5802) moved it from the session + tail ([#4533](https://github.com/moq-dev/moq/pull/4533), which only ever + held a few gaps) onto every track's live arrivals. Each arrival now pays + O(gaps) twice. `account` uses `Vec::splice`: closing the lowest gap moves + every run, about 100 KB at 4096 gaps. And while gaps exist, `expire` runs a + `dedup_by` over all runs plus `snapshot_subscription`. Under the 3600 s + delay in `track_sparse_read/timed`, gaps never fold, so a sparse feed is + quadratic overall. Fix: key the runs in a `BTreeMap`, + as `Delivered` already does. Cache the earliest fold instant (the smallest + `since` plus grace), so `expire` returns in O(1) when nothing is due. +- `track_gap_flush` (4.1x at 64 gaps, 96x at 4096): `Cursor::forget_past_budget` + in `track.rs` loops once per gap. Each pass calls `delivered.forget_below` + (a `BTreeMap::split_off` plus an insert) and reads the wall clock again in + `state.aged()`. Fix: read `now` once. Every gap below the shared successor + gets the same verdict, so look up the largest run start below it with + `runs.range(..s).next_back()` and call `forget_below` once. + `Delivered` comes from [#5446](https://github.com/moq-dev/moq/pull/5446) and + the wall-clock check from [#5548](https://github.com/moq-dev/moq/pull/5548). + Both are intended features, so only the cost per gap is wrong. + +Release forgot nothing here because it had no per-reader delivery record. +Don't aim for its absolute numbers. Aim for flat cost per gap across the +`track_gap_flush` and `track_sparse_read/timed` sweeps (16, 256, 4096). diff --git a/quest/m1/release-perf/stats-report.md b/quest/m1/release-perf/stats-report.md new file mode 100644 index 00000000000..45cb960aa30 --- /dev/null +++ b/quest/m1/release-perf/stats-report.md @@ -0,0 +1,42 @@ +# [S] Stats report + +## Goal + +`stats::Registry::report` takes each row's lock once and reads live sessions +without a compare-and-swap. The `stats/session-report` bench registers +transport `Wire`s, so the per-connection wire walk shows up in the measured +drain. The remaining O(live sessions) cost is accepted, and the reason is +recorded in the [epic](/quest/m1/release-perf/README.md). + +## Plan + +Decided 2026-10-21. `stats/session-report/1000x10` went from 11 us to 87 us +(7.9x, confirmed). The drain runs once a second, so at 10,000 live sessions +this is under 0.01% of a core. + +- Cause: [#5731](https://github.com/moq-dev/moq/pull/5731) added per-label + `sent_bytes`/`received_bytes`, kept by `SessionBytes`/`Arrival`. That keeps + the write path to one session-local atomic, and the cost moves to the drain: + `SessionBytes::read` does an `Arc::get_mut` per live session, and the row's + bytes mutex is taken twice. [#5659](https://github.com/moq-dev/moq/pull/5659) + (refusals, failures) and [#5773](https://github.com/moq-dev/moq/pull/5773) + (wire counters, `collect_wire` per report) add smaller fixed costs. +- Labels and per-label bytes stay. Labels are how an embedder buckets + sessions (per project, per key, or per customer-chosen bucket). Payload + bytes per bucket are the figure that matches payload billing, which wire + bytes can't give. Deleting them was considered and rejected. +- Cheap fix: in `read()`, use `Arc::strong_count(&arrival.traffic) == 1` plus + an `Acquire` fence instead of `Arc::get_mut`. This is valid only while no + `Weak` exists, so check that first. Return + `live.is_empty()` from the same guard as `read()`, instead of locking a + second time for the closed check. +- Bench: register a `Wire` per session in `session-report`. `collect_wire` + walks every live connection (a `Weak` upgrade plus two mutexes) on every + report and every relay `/metrics` scrape, which walks the registry a second + time for refusals. Wire counters are billed by embedders, so this walk is + needed by design. If it turns out significant at realistic scale, apply the + light fix: keep `WireInner.last` in atomics, and skip the `current` lock + unless the generation changed. A push design, where writers add straight to + the label row so the drain costs O(labels), needs its own `/quest-plan`. + +No API or wire change. From 28bed4e27ed6bebd866d9b767e70532ed357c753 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 16:44:43 -0700 Subject: [PATCH 2/5] quest: record why per-label stats bytes stay Co-Authored-By: Claude Opus 5.5 --- quest/m1/release-perf/stats-report.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/quest/m1/release-perf/stats-report.md b/quest/m1/release-perf/stats-report.md index 45cb960aa30..8222dc0a8e8 100644 --- a/quest/m1/release-perf/stats-report.md +++ b/quest/m1/release-perf/stats-report.md @@ -22,9 +22,12 @@ this is under 0.01% of a core. (refusals, failures) and [#5773](https://github.com/moq-dev/moq/pull/5773) (wire counters, `collect_wire` per report) add smaller fixed costs. - Labels and per-label bytes stay. Labels are how an embedder buckets - sessions (per project, per key, or per customer-chosen bucket). Payload - bytes per bucket are the figure that matches payload billing, which wire - bytes can't give. Deleting them was considered and rejected. + sessions, such as per project or per signing key. Payload bytes per label + are what matches payload billing, and wire bytes can't give that. Nothing + reads them yet. Deleting them was rejected anyway: the cost is negligible, + and removing them would be a breaking change to the stats wire that would + have to be redone once a per-key or per-bucket view wants them. Letting + customers choose their own labels was considered and set aside for now. - Cheap fix: in `read()`, use `Arc::strong_count(&arrival.traffic) == 1` plus an `Acquire` fence instead of `Arc::get_mut`. This is valid only while no `Weak` exists, so check that first. Return From 41e01a23da5f426c79654549c69a574a689d120c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 16:44:56 -0700 Subject: [PATCH 3/5] quest: align the epic's stats rationale Co-Authored-By: Claude Opus 5.5 --- quest/m1/release-perf/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/quest/m1/release-perf/README.md b/quest/m1/release-perf/README.md index 2c6bbfad5fc..3888fff117e 100644 --- a/quest/m1/release-perf/README.md +++ b/quest/m1/release-perf/README.md @@ -40,9 +40,9 @@ noise. [FETCH fan-out](/quest/m1/release-perf/fetch-fanout.md) owns it. - `stats/session-report` is accepted at O(live sessions), after a cheap fix. Per-label payload bytes ([#5731](https://github.com/moq-dev/moq/pull/5731)) - let an embedder bucket sessions by label and match payload billing per - bucket, and the drain runs once a second, using under 0.01% of a core at - 10,000 sessions. Deleting those bytes was rejected. + match payload billing per label, such as per signing key. The drain runs + once a second, using under 0.01% of a core at 10,000 sessions. Deleting + those bytes was rejected. ### Release as a baseline From 3df83745282172208e3cf05fc0902cd9381a2198 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 16:51:16 -0700 Subject: [PATCH 4/5] quest: keep the stats rationale to per-key and per-bucket uses Co-Authored-By: Claude Opus 5.5 --- quest/m1/release-perf/stats-report.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/quest/m1/release-perf/stats-report.md b/quest/m1/release-perf/stats-report.md index 8222dc0a8e8..a256acda04a 100644 --- a/quest/m1/release-perf/stats-report.md +++ b/quest/m1/release-perf/stats-report.md @@ -26,8 +26,7 @@ this is under 0.01% of a core. are what matches payload billing, and wire bytes can't give that. Nothing reads them yet. Deleting them was rejected anyway: the cost is negligible, and removing them would be a breaking change to the stats wire that would - have to be redone once a per-key or per-bucket view wants them. Letting - customers choose their own labels was considered and set aside for now. + have to be redone once a per-key or per-bucket view wants them. - Cheap fix: in `read()`, use `Arc::strong_count(&arrival.traffic) == 1` plus an `Acquire` fence instead of `Arc::get_mut`. This is valid only while no `Weak` exists, so check that first. Return From fa13b78f0ffbb4997ebf8b74bd675a2025ca87f5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 22:43:29 -0700 Subject: [PATCH 5/5] quest: land the resubscribe test on main before porting to release Co-Authored-By: Claude Opus 5.5 --- quest/m0/release-resubscribe-hang.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/quest/m0/release-resubscribe-hang.md b/quest/m0/release-resubscribe-hang.md index 49d4963a948..cb2cf4338ee 100644 --- a/quest/m0/release-resubscribe-hang.md +++ b/quest/m0/release-resubscribe-hang.md @@ -16,10 +16,11 @@ subscription never reaches the relay. The same bench passes on main. 1. Reproduce it on `origin/release` as a session test (one subscriber; open, close, and reopen the same track through a relay). Mock time, not sleeps. -2. If it hangs, find the cause, and check whether main fixed it on purpose. - Bisect main for the fix and backport it to `release` with the test. - Otherwise, fix it on `release` directly. Either way, keep the test on main - too. +2. If it hangs, find the cause. Land the test on main first, per the + main-first backport rule in `CONTRIBUTING.md`, then port it to `release` + in a separate PR. If main fixed the bug, bisect for that fix and port it + with the test. If main never had it (the code diverged), the `release` PR + carries the minimal fix for code that no longer exists on main. 3. If the test passes on `release`, the bench harness's backport is at fault (release's `tests/support` mock differs from main's). Record that in the PR and delete this quest.