From 659b66d349954416a8ed2c969bc1e0b0d44ac95f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 22 Oct 2026 08:20:53 -0700 Subject: [PATCH] quest: scope benchmark-first performance audit findings Co-Authored-By: Codex --- quest/m1/audit-perf.md | 40 ++++-- quest/m2/README.md | 1 + quest/m2/performance-survey/README.md | 135 ++++++++++++++++++ .../performance-survey/announced-snapshots.md | 32 +++++ quest/m2/performance-survey/audio-batches.md | 32 +++++ .../m2/performance-survey/bandwidth-grants.md | 50 +++++++ .../browser-media-copies.md | 32 +++++ .../performance-survey/datagram-delivery.md | 32 +++++ quest/m2/performance-survey/gst-buffers.md | 31 ++++ quest/m2/performance-survey/hang-buffered.md | 30 ++++ quest/m2/performance-survey/hls-serving.md | 33 +++++ .../m2/performance-survey/js-cached-groups.md | 31 ++++ .../m2/performance-survey/jwt-verification.md | 32 +++++ quest/m2/performance-survey/mkv-fragments.md | 32 +++++ .../performance-survey/quic-schedule-churn.md | 30 ++++ .../performance-survey/quic-send-fragments.md | 33 +++++ .../performance-survey/session-selection.md | 30 ++++ quest/m2/performance-survey/task-capacity.md | 33 +++++ 18 files changed, 654 insertions(+), 15 deletions(-) create mode 100644 quest/m2/performance-survey/README.md create mode 100644 quest/m2/performance-survey/announced-snapshots.md create mode 100644 quest/m2/performance-survey/audio-batches.md create mode 100644 quest/m2/performance-survey/bandwidth-grants.md create mode 100644 quest/m2/performance-survey/browser-media-copies.md create mode 100644 quest/m2/performance-survey/datagram-delivery.md create mode 100644 quest/m2/performance-survey/gst-buffers.md create mode 100644 quest/m2/performance-survey/hang-buffered.md create mode 100644 quest/m2/performance-survey/hls-serving.md create mode 100644 quest/m2/performance-survey/js-cached-groups.md create mode 100644 quest/m2/performance-survey/jwt-verification.md create mode 100644 quest/m2/performance-survey/mkv-fragments.md create mode 100644 quest/m2/performance-survey/quic-schedule-churn.md create mode 100644 quest/m2/performance-survey/quic-send-fragments.md create mode 100644 quest/m2/performance-survey/session-selection.md create mode 100644 quest/m2/performance-survey/task-capacity.md diff --git a/quest/m1/audit-perf.md b/quest/m1/audit-perf.md index 4179cf10e5a..8d13c6a7d04 100644 --- a/quest/m1/audit-perf.md +++ b/quest/m1/audit-perf.md @@ -103,27 +103,37 @@ large to review alongside the rest; those become `/quest-plan` follow-ups: Each needs a bench, swept over both axes where it fans out, or its own follow-up quest: -1. HLS/DASH serving (`rs/moq-hls/src/export/rendition.rs` `segment()`, - `media_playlist()`, `mpd.rs`): every request refetches and remuxes; sweep - viewers x renditions x window segments. -2. Session-level datagram delivery (`lite/publisher.rs` `serve_datagram`, - `ietf/datagram.rs`): per object per subscriber; add a - `session_delivery_datagram` group sweeping publishers x viewers. +1. [HLS request fan-out](/quest/m2/performance-survey/hls-serving.md) owns + repeated segment fetch/remux and playlist rendering, swept over viewers, + renditions and window segments. +2. [Session datagrams](/quest/m2/performance-survey/datagram-delivery.md) owns + complete adapter delivery across publishers and viewers. 3. LOC and CMAF wire containers in moq-mux: `containers` and `export` only use `Container::Legacy`; sweep container x frame size x frames. -4. JS hang container `Consumer` walks every buffered group per frame - (`consumer.ts` `#updateBuffered`, `#checkMaxDelay`); browser bench. -5. JS per-sample copies in CMAF decode and LOC encode; `js/loc`, `js/msf`, - `js/flate` have no benches. +4. [Buffered browser groups](/quest/m2/performance-survey/hang-buffered.md) + owns per-frame scans and bulk skip work in the Hang consumer. +5. [Browser media copies](/quest/m2/performance-survey/browser-media-copies.md) + owns CMAF sample copies and LOC encode allocation. Generic `js/msf` and + `js/flate` benchmark coverage remains here. 6. moq-quic loss recovery and real congestion controllers: every bench pins `Unlimited`; sweep controller x flight x loss x streams. -7. Relay `/announced` copies the route table per request (`web.rs`); sweep - routes x pollers. -8. JWT verification per connect and fetch (`moq-auth`, no benches); sweep - algorithm x key-set size. +7. [Announcement snapshots](/quest/m2/performance-survey/announced-snapshots.md) + owns HTTP snapshot/rendering costs across routes and pollers. +8. [JWT verification](/quest/m2/performance-survey/jwt-verification.md) owns + key-source I/O, import, lookup, parsing and crypto attribution. 9. MPEG-TS import with real repeated media (the CLI bench mostly skips null padding), and MKV, FLV, H.265, AV1 importers. -10. JS watch audio ring buffer and publish `Fanout` (lower impact). +10. JS watch audio ring buffer and publish `Fanout` (lower impact). Include + capture-to-encoder staging: `audio/framer.ts` copies planes before + `audio/encoder.ts` joins them. AudioData already uses a transfer list; + do not count that transfer as another mandatory copy. Sweep publishers + x renditions, channels and frame duration in a browser. + +The [2026-10-22 source survey](/quest/m2/performance-survey/README.md) owns +the extracted items above and additional algorithmic hypotheses, with paired +benchmarks required before optimization. This audit still owns nightly +regression attribution and the remaining coverage; the source survey does not +complete this run. Its fragmented-MKV child specializes item 9's MKV work. Single-axis fan-out benches to widen: `session_delivery_paced` (viewers only), `track_fanout_*` (subscribers only), `memory` (lite-06 only), and diff --git a/quest/m2/README.md b/quest/m2/README.md index d5d9f8cc95f..f7e17e835e2 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -67,6 +67,7 @@ ranked just below deterministic TS legs, which it requires. - [Media stats](/quest/m2/stats/README.md) - publishers announce a stats track in the catalog, viewers answer a soliciting catalog through a per-catalog `.echo` broadcast, and one model turns both into a health verdict and a preflight report - [Group storage benchmark](/quest/m2/group-storage-bench.md) - numbers pick the append-only log, a ring buffer, or sorted runs for `GroupState` before either lands - [Perf](/quest/m2/perf/README.md) - eliminate measured hot-path costs across moq-uring, kio, and the moq-net model +- [Performance survey follow-ups](/quest/m2/performance-survey/README.md) - benchmark the source audit's transport, browser, media and serving risks before optimizing - [Nightly benchmark regressions](/quest/m2/bench-ci.md) - each nightly compares the Criterion benches against the previous nightly's commit, publishes it to the git.moq.dev report, and alerts on a regression - [Relay session bench](/quest/m2/bench-relay.md) - the same scenario through moq-relay's own connection handling - [Transport x version benches](/quest/m2/bench-transport-sweep.md) - the session benches sweep transports as well as versions diff --git a/quest/m2/performance-survey/README.md b/quest/m2/performance-survey/README.md new file mode 100644 index 00000000000..de7cc34cd6c --- /dev/null +++ b/quest/m2/performance-survey/README.md @@ -0,0 +1,135 @@ +# Performance survey follow-ups + +## Goal + +Validate the performance risks found in the 2026-10-22 source audit with +benchmarks, then remove costs that matter to real consumers. A measured +no-win closes a quest with evidence; a suspicious loop alone does not justify +an optimization. Each child is independently landable. + +This epic's remaining work is to reconcile the children's measurements into +the recurring performance audit: what mattered, what did not, and which +nightly series now protect the relevant workload. + +## Plan + +Requested on 2026-10-22: an extensive performance audit and quests requiring +benchmarks to validate findings. Audited main +[`1d43c096e044`](https://github.com/moq-dev/moq/commit/1d43c096e04431bfc87d807ce47a546df8017475). +These are source-supported hypotheses, not new measured regressions or +promised speedups. General optimization stays in m2 under the existing +milestone policy. Do not change API, wire, scheduling policy, or retention +contracts just to improve a number; bring a necessary contract change to the +maintainer with the measurement first. No new user guide is needed for this +planning change; implementations update documentation they make stale. + +### Benchmark contract for every child + +- Reproduce the suspected cost in the production path before changing it. + Sweep the dimensions named in the child independently, including both + publisher and subscriber axes for fan-out. Keep useful delivered work + comparable and verify payloads, ordering, lifecycle, and timing semantics. +- Retain baseline and candidate revisions, exact commands, raw samples, + toolchain, machine, load, and build configuration. Use paired alternating + runs and an A/A control to distinguish a gain from measurement noise. + Report absolute cost and normalized cost, not only percentage changes. +- Attribute CPU work with profiles or instruction/operation counts; measure + allocations, copied bytes, retained memory, and tail latency where relevant. + Instruction counts do not measure blocking or prove end-to-end throughput. + Setup and teardown boundaries must match between variants. +- Measure JS changes in a real browser with the existing browser harness. + Bun operation counts can demonstrate an algorithmic slope but cannot stand + in for browser latency, GC, native copies, or hardware behavior. +- Network performance uses a rate below offered media bitrate behind a deep, + finite queue whose overflow also drops packets. Random loss is useful for + correctness only. Report delivery and latency beside CPU so doing less work + cannot look like an improvement. +- Extend the existing benchmark harnesses and schedule each new benchmark + at least nightly. Keep timing out of the merge gate. Fast deterministic + correctness/complexity checks may gate; no arbitrary speedup threshold. + Link the nightly command and retained report in the implementation PR. +- Land an optimization only when the measured consumer benefit justifies its + complexity and memory tradeoffs. Otherwise retain the useful benchmark and + record why the implementation was left alone. + +Reuse [the benchmark harness](/bench/README.md). Criterion's +[timing-loop guidance](https://bheisler.github.io/criterion.rs/book/user_guide/timing_loops.html) +explains setup/drop boundaries; the +[WebCodecs resource model](https://www.w3.org/TR/webcodecs/#resources) +means removing a JS copy still requires measuring native resource lifetime. +Neither source establishes a bottleneck in this repository. + +### Evidence and coverage + +The audit inspected Rust model/runtime/QUIC paths, JS transport/playback/ +publish paths, media import/export, auth, storage, bindings and benchmark +orchestration. It did not run the full release comparison, real-browser +performance suite, GPU/platform codecs, or every generated wrapper. Those +remain with their existing owners. The recurring audit is not completed by +this source survey. + +An operation-count probe imported the actual JS `bandwidth.allocate`, counted +`Want.priority` getter reads, and called it for every reservation, as +`Allocator.#publish` does. With one tier, 16/64/256 reservations caused +512/8,192/131,072 reads, exactly 2N². With distinct tiers, 16/64 caused +6,952/438,944 reads. Nix Bun 1.4.2 was used; no browser speedup is claimed. +The bandwidth child must retain a reproducible benchmark, including the +complete allocator and Rust consumers, before selecting a fix. + +`python3 sh/gh/nightly.test.py NightlyTest.test_every_cargo_bench_target_runs_nightly` +reproduced the missing `moq-archive/producer` selection. The existing +[CI audit PR](https://github.com/moq-dev/moq/pull/6266) already fixes it. +No duplicate quest or workflow edit belongs here. + +### Existing ownership and deliberately deferred work + +- [Release performance](/quest/m1/release-perf/README.md) owns sparse gaps, + resume duplicates, idle GC, stats drain, FETCH fan-out, JS frame regressions, + and paired comparison correctness. Active PRs already address these. +- [Perf](/quest/m2/perf/README.md) owns measured channel contention, relay + allocations, ingest batching, worker quiescence and runtime selection. +- [Release audit PR](https://github.com/moq-dev/moq/pull/6311) owns QMux + copying/resource growth, CMAF allocation abuse, media reorder queues, + watch teardown, and HLS contract investigations. MKV fragmentation below + concerns parser replay before those queues; recheck that PR before coding. +- [Binding bytes](/quest/m2/ffi-bytes.md), + [foreign polling](/quest/m2/ffi-poll-protocol.md), + [Kotlin calls](/quest/m2/kt-ffi-calls.md), + [NVENC pooling](/quest/m2/nvenc-pool.md), and + [watch worker planning](/quest/m2/plan-watch-worker.md) retain their scopes. +- [Route wakes](/quest/m2/route-wakes.md), + [keyed prefix polling](/quest/m2/keyed-prefix-polling.md), and + [front parking](/quest/m2/origin-front-parks.md) own discovery scans/wakes. +- The [recurring audit](/quest/m1/audit-perf.md) retains real congestion- + controller/loss-recovery coverage, broader importer/container coverage, + and publish Fanout/audio-ring profiling. QUIC protocol microbenchmarks use + `Unlimited`; their numbers cannot establish controller performance. Expand + reassembly flight/reordering coverage there before proposing another index. + [BBR study](/quest/m2/quic-bbr-natural-drain.md) owns controller behavior. +- Archive reader internals are being replaced by the + [archive consumer](/quest/m1/archive/consumer.md). Do not optimize the + outgoing timeline reader. Benchmark the landed replacement when revisiting it. + +## Required + +- [Fragmented QUIC sends](/quest/m2/performance-survey/quic-send-fragments.md) - packetization cost under a retained fragmented flight +- [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - polling and memory after peak task occupancy subsides +- [Bandwidth grants](/quest/m2/performance-survey/bandwidth-grants.md) - repeated full allocation in Rust and JS +- [Cached JS group joins](/quest/m2/performance-survey/js-cached-groups.md) - repeated sorting while replaying retained groups +- [PCM batch processing](/quest/m2/performance-survey/audio-batches.md) - repeated tail movement during encoding and resampling +- [Fragmented MKV import](/quest/m2/performance-survey/mkv-fragments.md) - accumulated-buffer copies and parser replay +- [QUIC scheduling churn](/quest/m2/performance-survey/quic-schedule-churn.md) - priority and retirement work across unrelated groups +- [HLS request fan-out](/quest/m2/performance-survey/hls-serving.md) - repeated fetch, remux and playlist construction per viewer +- [Buffered browser groups](/quest/m2/performance-survey/hang-buffered.md) - per-frame scans and bulk skip work +- [Session datagrams](/quest/m2/performance-survey/datagram-delivery.md) - complete per-object delivery across publishers and viewers +- [JWT verification](/quest/m2/performance-survey/jwt-verification.md) - key loading, lookup, parsing and crypto attribution +- [Announcement snapshots](/quest/m2/performance-survey/announced-snapshots.md) - HTTP polling cost versus result size and unrelated routes +- [Selective session operations](/quest/m2/performance-survey/session-selection.md) - exact-ID lookups and registry lock hold time +- [Browser media copies](/quest/m2/performance-survey/browser-media-copies.md) - CMAF sample and LOC framing copies versus retention +- [GStreamer ingest buffers](/quest/m2/performance-survey/gst-buffers.md) - copy avoidance versus upstream pool retention + +## Related + +- [Performance audit](/quest/m1/audit-perf.md) - nightly regression attribution and remaining coverage +- [Benchmark sources](/quest/m2/bench-sources.md) - publishing existing measurements to the trend table +- [Nightly comparisons](/quest/m2/bench-ci.md) - paired nightly regression reporting diff --git a/quest/m2/performance-survey/announced-snapshots.md b/quest/m2/performance-survey/announced-snapshots.md new file mode 100644 index 00000000000..3c9157fd07e --- /dev/null +++ b/quest/m2/performance-survey/announced-snapshots.md @@ -0,0 +1,32 @@ +# [M] Measure HTTP announcement snapshot fan-out + +## Goal + +Repeated announcement-list polling has measured cost proportional to useful +results, with avoidable copying or unrelated-route work removed if worthwhile. + +## Plan + +[serve_announced](/rs/moq-relay/src/web.rs) creates an announcement consumer, +collects prefixes, creates another Vec of strings and joins them per request +(lines 1011-1044). Some output-size cost is necessary; repeated snapshot and +formatting work is the hypothesis, not proof of a bottleneck. + +Sweep registered routes × HTTP pollers, independently varying matched prefix +size, route churn and path length. Compare an empty selective prefix, a small +result among many unrelated routes and the full listing. Measure CPU/request, +allocations/copied bytes, response latency and route-update latency while +polling. Exercise the real authorization/view path, and attribute its cost +separately from rendering. + +Preserve admission, tenant isolation, hidden-prefix rules, response contents +and visibility during changes. Any shared snapshot must follow authorization +and content identity; avoid a global credential-blind cache. Prefer removing +intermediate allocations over a second route index unless measurements earn it. +This takes the recurring audit's HTTP announcement item; discovery wake/index +quests retain ownership of the underlying routing machinery. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/audio-batches.md b/quest/m2/performance-survey/audio-batches.md new file mode 100644 index 00000000000..eb91dfc7794 --- /dev/null +++ b/quest/m2/performance-survey/audio-batches.md @@ -0,0 +1,32 @@ +# [M] PCM batching avoids repeated tail movement + +## Goal + +Encoding and resampling larger PCM batches does not repeatedly move the +same remaining samples, while preserving audio and timestamp behavior. + +## Plan + +[Producer::publish_full_frames](/rs/moq-audio/src/encode/producer.rs) +front-drains into a new Vec for each packet after appending the entire input +(lines 379-389). [Resampler::convert](/rs/moq-audio/src/resample.rs) likewise +front-drains after every chunk (line 291). N chunks in a call can move a +triangular number of trailing samples. Capture and binding producers use these +paths; small normal capture periods may hide the cost. + +Benchmark a constant duration of audio delivered in different batch sizes, +channels and input/output rate pairs, also sweeping independent producers. +Separate buffering/resampling from codec CPU with a cheap supported codec +control. Measure instructions/sample, allocations, copied bytes, throughput +and worst call duration, including tiny steady batches and large bursts. +Verify PCM or decoded output as appropriate, sample counts, packet boundaries, +timestamps, partial tails, flush and discontinuities. + +Processing slices and compacting only the remainder is a candidate; do not +prescribe a queue type before measuring. This does not change capture period +policy or the [Opus grant](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md). + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/bandwidth-grants.md b/quest/m2/performance-survey/bandwidth-grants.md new file mode 100644 index 00000000000..0728874abf2 --- /dev/null +++ b/quest/m2/performance-survey/bandwidth-grants.md @@ -0,0 +1,50 @@ +# [M] Compute bandwidth grants without repeated whole-table work + +## Goal + +Rust and browser publishers receive the same fair bandwidth grants with +less repeated allocation work, when measured across real observers. + +## Plan + +[JS allocation](/js/net/src/bandwidth.ts) computes each reservation's grant +separately. Each call scans all wants, filters/sorts each priority tier, and +searches again for the next tier. Synchronous peek repeats it. The +[Rust sibling](/rs/moq-net/src/model/bandwidth.rs) also snapshots claims and +polls demand per share. One estimate change can duplicate work across every +observer. The survey records deterministic JS priority-read counts, including +2N² reads for all grants in one tier, before sorting. + +Reproduce that operation count with the repository's Bun (not a latency test): + +```bash +bun -e ' +import { allocate } from "./js/net/src/bandwidth.ts"; +for (const tiers of [1, 8, 64]) for (const n of [16, 64, 256]) { + let reads = 0; + const wants = Array.from({ length: n }, (_, id) => ({ + id, get priority() { reads++; return id % tiers; }, max: 1000, + })); + let sum = 0; + for (let id = 0; id < n; id++) sum += allocate(n * 500, wants, id) ?? 0; + console.log({ n, tiers: Math.min(tiers, n), reads, sum }); +}' +``` + +Extend the Rust bandwidth target beyond its existing unlimited-allocator +probe-target sum; add a real-browser counterpart. Sweep connections × +reservations, priority diversity and observer count, with demanded/idle mixes. +Measure estimate changes, demand transitions, ceiling changes and per-frame +peek separately: CPU, allocations, waker/listener registrations and browser +main-thread delay. Assert every grant, not merely their sum. + +Consider calculating a shared allocation once per relevant change, but retain +strict priority, max-min fairness, rounding, ties, immediate peek behavior and +closed-demand removal. Cover both languages in this quest. Coordinate with +[viewer targets](/quest/m2/probe/viewer-target.md), which changes how the same +allocator feeds probes. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/browser-media-copies.md b/quest/m2/performance-survey/browser-media-copies.md new file mode 100644 index 00000000000..1bf171afcc6 --- /dev/null +++ b/quest/m2/performance-survey/browser-media-copies.md @@ -0,0 +1,32 @@ +# [M] Measure browser media framing copies and retention + +## Goal + +CMAF decode and LOC encode retain only copies that earn their lifetime or +API benefits under representative browser media workloads. + +## Plan + +[CMAF decode](/js/hang/src/cmaf/decode.ts) slices each sample from mdat +(line 437). [LOC encoding](/js/loc/src/index.ts) allocates contiguous framed +output and copies the payload (lines 181-195). These are concrete copy sites, +not proof that views are cheaper: a small sample view may pin a large fragment, +and a WebCodecs copyTo may be necessary to obtain bytes at all. + +Extend the existing browser CMAF case and add LOC framing measurements. +Sweep tracks × consumers, sample size × samples per fragment, and held-sample +lifetime. Report JS-visible allocation/copy volume, CPU, GC and retained heap +for immediate consumption and a slow reader. Separate format cost from codec +and transport work; do not infer native copy counts from typed-array hooks. +Check bytes, codec configuration, timestamps and ownership through close. + +A view/owned-buffer change must preserve mutation isolation and release large +backing allocations promptly. Keep current copies if that tradeoff wins. +This extracts the recurring audit's sample-copy item; missing generic +`js/msf`/`js/flate` coverage remains with that audit. CMAF malformed-count +handling stays with the release audit. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/datagram-delivery.md b/quest/m2/performance-survey/datagram-delivery.md new file mode 100644 index 00000000000..4a39d457819 --- /dev/null +++ b/quest/m2/performance-survey/datagram-delivery.md @@ -0,0 +1,32 @@ +# [M] Benchmark complete datagram delivery fan-out + +## Goal + +Session datagram costs are measured across independent publishers and +viewers, and any demonstrated avoidable per-object work is removed. + +## Plan + +The [lite publisher](/rs/moq-net/src/lite/publisher.rs) serializes and +schedules each delivered datagram in `serve_datagram`; the +[IETF adapter](/rs/moq-net/src/ietf/datagram.rs) has its own framing path. +Model-only datagram benches cannot attribute complete adapter delivery. +No regression is claimed merely because framing happens per subscriber. + +Extend session benchmarks with publishers × viewers, payload sizes and +object cadence, covering supported lite and IETF datagram paths. Measure +CPU/delivered object, allocations/copies and queue residence, with complete +sequence/payload validation. Separate the in-memory protocol cost from real +QUIC scheduling and congestion. Under bufferbloat record offered, received, +dropped and expired objects and slow-viewer isolation; best-effort loss must +not make lower CPU look like a gain. + +Keep framing, priority, expiry and memory-accounting behavior intact. The +[release audit](https://github.com/moq-dev/moq/pull/6311) owns the decoded +memory-accounting defect. This quest owns the recurring audit's missing +session-level datagram measurement and any independent measured optimization. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/gst-buffers.md b/quest/m2/performance-survey/gst-buffers.md new file mode 100644 index 00000000000..7e096d0f781 --- /dev/null +++ b/quest/m2/performance-survey/gst-buffers.md @@ -0,0 +1,31 @@ +# [M] Measure GStreamer ingest copies against pool retention + +## Goal + +GStreamer ingest uses the least costly safe buffer ownership strategy, +proven with both fast and slow consumers. + +## Plan + +[Sink::forward_buffer](/rs/moq-gst/src/sink/imp.rs) receives an owned +GStreamer buffer but maps and copies its full encoded payload into Bytes +(lines 714-737). Egress already wraps Bytes in a GStreamer buffer. The ingest +copy could be avoidable, but retaining a mapped upstream buffer can exhaust a +finite producer pool or pin more memory than copying. + +Add ingest benchmarks beyond the existing catalog target. Sweep streams × +subscribers, payload size/cadence, and independently the number of upstream +pool buffers and reader retention. Measure copy bandwidth, allocations, +CPU/frame, delivery tail latency, peak memory and upstream stalls. Include +multi-memory/noncontiguous buffers, map failures, flush, cancellation and +teardown; compare identical delivered payloads and timestamps. + +Evaluate owned mapped backing only if end-to-end results justify it and its +Send/lifetime properties hold. Do not claim a win from removing memcpy alone, +or introduce an unbounded pool to make the experiment pass. Retaining the +current copy with a benchmark explaining why is a complete outcome. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/hang-buffered.md b/quest/m2/performance-survey/hang-buffered.md new file mode 100644 index 00000000000..59baf68ad66 --- /dev/null +++ b/quest/m2/performance-survey/hang-buffered.md @@ -0,0 +1,30 @@ +# [M] Browser buffered-group bookkeeping scales with changed data + +## Goal + +Playing or skipping buffered groups avoids avoidable repeated backlog scans +without changing the buffering and max-delay contract. + +## Plan + +[Hang Consumer](/js/hang/src/container/consumer.ts) rebuilds buffered ranges +by walking groups in `#updateBuffered` (line 696). `#checkMaxDelay` scans +all groups inside a loop that shifts old groups (line 377), so bulk skipping +can repeat the scan. These paths execute while frames arrive and are read. + +Add real-browser cases sweeping buffered groups × frames per group, then +players × independent publishers. Compare ordinary steady playback, sparse +out-of-order groups, large lookahead and a burst that must be skipped. +Measure operation counts, main-thread CPU, allocations/GC and frame delay. +Verify exact buffered ranges, gaps, timestamp order, skip decisions and closure +of discarded frames. Incremental extrema/ranges are candidates only if their +maintenance is cheaper across normal and adversarial arrival orders. + +This extracts the recurring audit's buffered-consumer finding. +[Watch worker planning](/quest/m2/plan-watch-worker.md) owns main-thread +isolation; moving the work does not discharge this algorithmic measurement. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/hls-serving.md b/quest/m2/performance-survey/hls-serving.md new file mode 100644 index 00000000000..d0f56f0efce --- /dev/null +++ b/quest/m2/performance-survey/hls-serving.md @@ -0,0 +1,33 @@ +# [L] Measure and reduce repeated HLS serving work + +## Goal + +HLS/DASH viewer fan-out avoids redundant fetch/remux/render work where +measurements justify sharing, without retaining stale or unauthorized output. + +## Plan + +[Rendition::fetch](/rs/moq-hls/src/export/rendition.rs) creates a muxer, +fetches group ranges, gathers frames and constructs a fragment on each segment +request (lines 1147-1217). Init construction already has caching; do not count +it as missing. Media playlists snapshot/render on each call, and DASH builds +representations from the same rendition state. + +Sweep viewers × renditions with independent window length, frame size and +concurrent same-segment requests. Measure CPU/request, upstream fetch count, +allocations, retained bytes and response tail latency, both cold and warm. +Separate unavoidable response serialization from duplicate content work. +Validate all bytes/durations and include gaps, eviction, restarts, reference +handover, concurrent first requests and authorization differences. + +Shared in-flight work or bounded immutable output caching are candidates only +after profiling. Keys and ownership must include the actual content instance; +never let a reused row number cross epochs or cache one viewer's credentials +in another's playlist. The archive/HLS contract quests and +[release audit](https://github.com/moq-dev/moq/pull/6311) retain policy decisions. +This takes ownership of the recurring audit's HLS serving coverage item. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/js-cached-groups.md b/quest/m2/performance-survey/js-cached-groups.md new file mode 100644 index 00000000000..f9b524b0435 --- /dev/null +++ b/quest/m2/performance-survey/js-cached-groups.md @@ -0,0 +1,31 @@ +# [M] Joining a cached JS track avoids repeated full sorts + +## Goal + +A subscriber joining retained groups, or falling behind while groups arrive, +does no unnecessary whole-backlog sorting. + +## Plan + +[Track replay](/js/net/src/track.ts) mirrors every cached group on join +(line 1061); each mirror pushes and sorts the entire unread group array +(lines 1096-1098). Even ordered replay sorts arrays of sizes 1 through G. +The separate timeline index already has an append fast path and ordered +insertion. Existing track timing drains immediately and misses growing queues. + +Benchmark retained groups × joining subscribers, then unread backlog × active +subscribers, in a real browser. Hold payload work fixed where possible and +include monotonic, reverse and mixed arrivals. Report comparator visits, +join/append latency, allocation/GC cost and event-loop delay. Prove replay +contents, ascending delivery, sparse arrivals, eviction, close and retained +mirror lifetimes remain identical. Reuse existing ordering machinery if it +makes the result simpler. + +This is distinct from subscription update traffic in +[moq-dev/moq#6305](https://github.com/moq-dev/moq/pull/6305) and the storage +contract in [unbounded JS groups](/quest/m2/unbounded-groups/js.md). + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/jwt-verification.md b/quest/m2/performance-survey/jwt-verification.md new file mode 100644 index 00000000000..295b36a9b3a --- /dev/null +++ b/quest/m2/performance-survey/jwt-verification.md @@ -0,0 +1,32 @@ +# [M] Attribute authentication verification cost before caching + +## Goal + +Authentication cost is explained across key sources and concurrent clients, +and avoidable work is reduced without weakening key rotation or validation. + +## Plan + +[Serve::verify](/rs/moq-auth/src/serve.rs) rereads a key or whole key-set +file for each request, then imports it (lines 249-271). A Key's cached crypto +material is therefore not shared across these requests. The +[KeySet lookup](/rs/moq-auth/src/set.rs) is linear, while header parsing also +occurs in [Key::decode](/rs/moq-auth/src/key.rs). Existing session admission +and relay lease-refresh benchmarks do not isolate actual JWT verification. + +Benchmark clients × keys, each supported algorithm, first/last/missing kid, +file/directory/set sources and cold/warm material separately. Attribute file +I/O, JSON/JWT parsing, lookup, key import and signature verification; record +CPU, allocations, throughput and p95/p99 request latency. Include concurrent +key replacement, removal, invalid keys, expired tokens and refusal bursts. +Use deterministic time in validity checks. + +Do not prescribe a TTL or token-result cache. Preserve current fresh-key and +revocation behavior; if sharing requires a new freshness contract, present +that decision with measurements before implementing it. This extracts and +sharpens the recurring audit's auth benchmark gap. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/mkv-fragments.md b/quest/m2/performance-survey/mkv-fragments.md new file mode 100644 index 00000000000..f58c70c87fb --- /dev/null +++ b/quest/m2/performance-survey/mkv-fragments.md @@ -0,0 +1,32 @@ +# [M] Fragmented MKV import avoids replaying accumulated input + +## Goal + +Streaming MKV/WebM import costs remain reasonable as the same input is +split into smaller reads, without changing accepted media or parse errors. + +## Plan + +[Import::drain](/rs/moq-mux/src/container/mkv/import.rs) clones the retained +BytesMut, constructs a fresh WebmIterator and skips handled tags only after +parsing them (lines 227-252). A B-byte partial block arriving in c-byte +chunks can copy O(B²/c) bytes before completion; replay can add parse work. +This is before the reorder queues covered by the +[release audit](https://github.com/moq-dev/moq/pull/6311); recheck its final +scope before implementation. + +Benchmark block size × input chunk size with identical complete media, +including large keyframes, lacing, bounded BlockGroups and unknown-size live +containers. Measure instructions/input byte, allocation and copy volume, +latency to first frame and peak memory. Separate cold catalog startup from +steady parsing. Include fragmented malformed/truncated tags in correctness +checks and ensure bytes, timestamps and errors match unsplit input. + +Prefer retained parser progress or bounded incremental staging if it wins. +Do not hide the slope with a minimum read size or an arbitrary refusal cap. +This specializes the recurring audit's generic MKV importer coverage. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/quic-schedule-churn.md b/quest/m2/performance-survey/quic-schedule-churn.md new file mode 100644 index 00000000000..7d7e412a182 --- /dev/null +++ b/quest/m2/performance-survey/quic-schedule-churn.md @@ -0,0 +1,30 @@ +# [M] QUIC priority changes touch only relevant scheduling state + +## Goal + +Changing or retiring queued scheduling groups does not impose unnecessary +work on unrelated groups sharing a connection. + +## Plan + +[The QUIC scheduler](/rs/moq-quic/src/connection/streams/schedule.rs) +linearly searches round-robin VecDeques in `unring_group` and retirement +(lines 786-821). Priority changes call that path. On shared relay sessions, +repeated updates or unfavorable retirement order can multiply changed groups +by unrelated queued groups. Existing dispatch/stream-churn benches do not +isolate scheduling-group reprioritization and retirement. + +Sweep queued groups × changed/retired groups, and domains × retired domains. +Measure front/middle/tail and mixed-order operations, priority-update CPU, +allocations and packet-dispatch tail latency during churn. Confirm through +publisher × subscriber traffic on a shared session. Preserve byte fairness, +priority ordering and eventual progress rather than changing scheduler policy. + +Coordinate with [priority scope](/quest/m1/track-priority-scope.md); benchmark +the current semantics after that work lands if it changes these paths. No new +priority contract is selected by this performance quest. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/quic-send-fragments.md b/quest/m2/performance-survey/quic-send-fragments.md new file mode 100644 index 00000000000..de657c7125a --- /dev/null +++ b/quest/m2/performance-survey/quic-send-fragments.md @@ -0,0 +1,33 @@ +# [M] Fragmented QUIC sends cost proportional to the bytes touched + +## Goal + +Native relay egress avoids repeated traversal of unrelated retained write +chunks, if benchmarks establish a material cost. + +## Plan + +At the audit revision, [SendBuffer::get](/rs/moq-quic/src/connection/send_buffer.rs) +starts at the first unacknowledged segment for each lookup (lines 146-162). +[Packetization](/rs/moq-quic/src/connection/streams/state.rs) calls it again +for every piece of a range. N small retained segments can incur a triangular +number of visits while sending sequentially; delayed ACKs or a prefix hole +keep earlier segments in the scan. + +Extend `send_write_size` in the QUIC stream benchmarks. Sweep write chunk +size against retained flight size independently, with fixed total payload +controls, normal sequential sending, old-offset retransmission and ACK holes. +Measure segment visits, instructions/byte, CPU, allocations and retained memory. +The existing 64 KiB stream case does not isolate retained-flight growth. +Confirm any win through a relay with publishers × viewers and finite-queue +bufferbloat using a real congestion controller. Preserve ACK, retransmission, +reset and truncation semantics. An offset index or cursor is a candidate, +not a predetermined design. + +Related: [egress profiling](/quest/m2/quic-egress-profile.md) owns syscall, +copy and pacing attribution; this quest owns segment lookup complexity. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/session-selection.md b/quest/m2/performance-survey/session-selection.md new file mode 100644 index 00000000000..39397275dfc --- /dev/null +++ b/quest/m2/performance-survey/session-selection.md @@ -0,0 +1,30 @@ +# [S] Selective session operations avoid full registry scans + +## Goal + +Exact-session listing and revalidation avoid unnecessary work across other +sessions while maintaining the existing filter semantics. + +## Plan + +[Registry::list and revalidate](/rs/moq-relay/src/session.rs) iterate every +entry while holding the registry mutex (lines 114-132), even though +`Filter::id` is exact and the table is keyed by that ID. Views/IDs are cloned +and matching leases woken while the lock is held. Operators and concurrent +connection registration/drop are the consumers. + +Sweep sessions × concurrent queries/revalidations, comparing exact-ID hits +and misses, selective multi-field filters and all-session requests. Measure +CPU/query, allocation volume, lock hold/wait, response latency, and concurrent +registration/drop latency. Verify an ID plus other filters still means all +fields must match, and only matching sessions are nudged. + +A direct keyed lookup is the minimal candidate. Broad listing necessarily +scales with output; do not add secondary indexes for it without independent +measurement. Existing relay auth-refresh benches do not measure this +administrative selection path. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR. diff --git a/quest/m2/performance-survey/task-capacity.md b/quest/m2/performance-survey/task-capacity.md new file mode 100644 index 00000000000..b28be054b30 --- /dev/null +++ b/quest/m2/performance-survey/task-capacity.md @@ -0,0 +1,33 @@ +# [M] Task polling recovers after a burst + +## Goal + +Long-lived task sets do not pay an unjustified steady-state CPU or memory +cost for a burst that has already drained. + +## Plan + +[kio::Tasks](/rs/kio/src/task.rs) grows slots and per-slot wakers, retains +them after completion, and snapshots every allocated wake-bit word each poll +(lines 270-298, 364-370). Dispatch scans those words again. Work can depend on +historical peak/64 even when only one task remains. The io_uring worker and +session serving loops are current consumers. + +Extend [task benchmarks](/rs/kio/benches/tasks.rs) with peak slots × current +live/ready slots: burst then one survivor, sparse live slots, full occupancy, +and empty-but-polled. Report instructions and CPU per owner poll/per ready +task, allocations and retained bytes after draining. Compare a long-lived +worker or session before and after connection/subscription churn. + +The present sparse-wake case fixes 16 ready tasks among 1,000 residents; it +does not isolate historical capacity. Any reclamation or ready-word index +must preserve stale-waker safety, concurrent wakeups, coalescing and snapshot +fairness. Do not trade cheap burst reuse for churn without measuring both. + +Related: [worker quiescence](/quest/m2/perf/uring-quiescence.md) and +[relay allocations](/quest/m2/perf/relay-allocs.md) own other costs. + +Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): +measure the baseline before optimizing, retain paired evidence, verify equivalent +work, schedule coverage nightly, and accept a measured no-win. Expected public +API and wire impact: none; report the actual impact in the implementation PR.