Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 25 additions & 15 deletions quest/m1/audit-perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,27 +103,37 @@ large to review alongside the rest; those become `/quest-plan` follow-ups:
Each needs a bench, swept over both axes where it fans out, or its own
follow-up quest:

1. HLS/DASH serving (`rs/moq-hls/src/export/rendition.rs` `segment()`,
`media_playlist()`, `mpd.rs`): every request refetches and remuxes; sweep
viewers x renditions x window segments.
2. Session-level datagram delivery (`lite/publisher.rs` `serve_datagram`,
`ietf/datagram.rs`): per object per subscriber; add a
`session_delivery_datagram` group sweeping publishers x viewers.
1. [HLS request fan-out](/quest/m2/performance-survey/hls-serving.md) owns
repeated segment fetch/remux and playlist rendering, swept over viewers,
renditions and window segments.
2. [Session datagrams](/quest/m2/performance-survey/datagram-delivery.md) owns
complete adapter delivery across publishers and viewers.
3. LOC and CMAF wire containers in moq-mux: `containers` and `export` only use
`Container::Legacy`; sweep container x frame size x frames.
4. JS hang container `Consumer` walks every buffered group per frame
(`consumer.ts` `#updateBuffered`, `#checkMaxDelay`); browser bench.
5. JS per-sample copies in CMAF decode and LOC encode; `js/loc`, `js/msf`,
`js/flate` have no benches.
4. [Buffered browser groups](/quest/m2/performance-survey/hang-buffered.md)
owns per-frame scans and bulk skip work in the Hang consumer.
5. [Browser media copies](/quest/m2/performance-survey/browser-media-copies.md)
owns CMAF sample copies and LOC encode allocation. Generic `js/msf` and
`js/flate` benchmark coverage remains here.
6. moq-quic loss recovery and real congestion controllers: every bench pins
`Unlimited`; sweep controller x flight x loss x streams.
7. Relay `/announced` copies the route table per request (`web.rs`); sweep
routes x pollers.
8. JWT verification per connect and fetch (`moq-auth`, no benches); sweep
algorithm x key-set size.
7. [Announcement snapshots](/quest/m2/performance-survey/announced-snapshots.md)
owns HTTP snapshot/rendering costs across routes and pollers.
8. [JWT verification](/quest/m2/performance-survey/jwt-verification.md) owns
key-source I/O, import, lookup, parsing and crypto attribution.
9. MPEG-TS import with real repeated media (the CLI bench mostly skips null
padding), and MKV, FLV, H.265, AV1 importers.
10. JS watch audio ring buffer and publish `Fanout` (lower impact).
10. JS watch audio ring buffer and publish `Fanout` (lower impact). Include
capture-to-encoder staging: `audio/framer.ts` copies planes before
`audio/encoder.ts` joins them. AudioData already uses a transfer list;
do not count that transfer as another mandatory copy. Sweep publishers
x renditions, channels and frame duration in a browser.

The [2026-10-22 source survey](/quest/m2/performance-survey/README.md) owns
the extracted items above and additional algorithmic hypotheses, with paired
benchmarks required before optimization. This audit still owns nightly
regression attribution and the remaining coverage; the source survey does not
complete this run. Its fragmented-MKV child specializes item 9's MKV work.

Single-axis fan-out benches to widen: `session_delivery_paced` (viewers only),
`track_fanout_*` (subscribers only), `memory` (lite-06 only), and
Expand Down
1 change: 1 addition & 0 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ ranked just below deterministic TS legs, which it requires.
- [Media stats](/quest/m2/stats/README.md) - publishers announce a stats track in the catalog, viewers answer a soliciting catalog through a per-catalog `.echo` broadcast, and one model turns both into a health verdict and a preflight report
- [Group storage benchmark](/quest/m2/group-storage-bench.md) - numbers pick the append-only log, a ring buffer, or sorted runs for `GroupState` before either lands
- [Perf](/quest/m2/perf/README.md) - eliminate measured hot-path costs across moq-uring, kio, and the moq-net model
- [Performance survey follow-ups](/quest/m2/performance-survey/README.md) - benchmark the source audit's transport, browser, media and serving risks before optimizing
- [Nightly benchmark regressions](/quest/m2/bench-ci.md) - each nightly compares the Criterion benches against the previous nightly's commit, publishes it to the git.moq.dev report, and alerts on a regression
- [Relay session bench](/quest/m2/bench-relay.md) - the same scenario through moq-relay's own connection handling
- [Transport x version benches](/quest/m2/bench-transport-sweep.md) - the session benches sweep transports as well as versions
Expand Down
135 changes: 135 additions & 0 deletions quest/m2/performance-survey/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Performance survey follow-ups

## Goal

Validate the performance risks found in the 2026-10-22 source audit with
benchmarks, then remove costs that matter to real consumers. A measured
no-win closes a quest with evidence; a suspicious loop alone does not justify
an optimization. Each child is independently landable.

This epic's remaining work is to reconcile the children's measurements into
the recurring performance audit: what mattered, what did not, and which
nightly series now protect the relevant workload.

## Plan

Requested on 2026-10-22: an extensive performance audit and quests requiring
benchmarks to validate findings. Audited main
[`1d43c096e044`](https://github.com/moq-dev/moq/commit/1d43c096e04431bfc87d807ce47a546df8017475).
These are source-supported hypotheses, not new measured regressions or
promised speedups. General optimization stays in m2 under the existing
milestone policy. Do not change API, wire, scheduling policy, or retention
contracts just to improve a number; bring a necessary contract change to the
maintainer with the measurement first. No new user guide is needed for this
planning change; implementations update documentation they make stale.

### Benchmark contract for every child

- Reproduce the suspected cost in the production path before changing it.
Sweep the dimensions named in the child independently, including both
publisher and subscriber axes for fan-out. Keep useful delivered work
comparable and verify payloads, ordering, lifecycle, and timing semantics.
- Retain baseline and candidate revisions, exact commands, raw samples,
toolchain, machine, load, and build configuration. Use paired alternating
runs and an A/A control to distinguish a gain from measurement noise.
Report absolute cost and normalized cost, not only percentage changes.
- Attribute CPU work with profiles or instruction/operation counts; measure
allocations, copied bytes, retained memory, and tail latency where relevant.
Instruction counts do not measure blocking or prove end-to-end throughput.
Setup and teardown boundaries must match between variants.
- Measure JS changes in a real browser with the existing browser harness.
Bun operation counts can demonstrate an algorithmic slope but cannot stand
in for browser latency, GC, native copies, or hardware behavior.
- Network performance uses a rate below offered media bitrate behind a deep,
finite queue whose overflow also drops packets. Random loss is useful for
correctness only. Report delivery and latency beside CPU so doing less work
cannot look like an improvement.
- Extend the existing benchmark harnesses and schedule each new benchmark
at least nightly. Keep timing out of the merge gate. Fast deterministic
correctness/complexity checks may gate; no arbitrary speedup threshold.
Link the nightly command and retained report in the implementation PR.
- Land an optimization only when the measured consumer benefit justifies its
complexity and memory tradeoffs. Otherwise retain the useful benchmark and
record why the implementation was left alone.

Reuse [the benchmark harness](/bench/README.md). Criterion's
[timing-loop guidance](https://bheisler.github.io/criterion.rs/book/user_guide/timing_loops.html)
explains setup/drop boundaries; the
[WebCodecs resource model](https://www.w3.org/TR/webcodecs/#resources)
means removing a JS copy still requires measuring native resource lifetime.
Neither source establishes a bottleneck in this repository.

### Evidence and coverage

The audit inspected Rust model/runtime/QUIC paths, JS transport/playback/
publish paths, media import/export, auth, storage, bindings and benchmark
orchestration. It did not run the full release comparison, real-browser
performance suite, GPU/platform codecs, or every generated wrapper. Those
remain with their existing owners. The recurring audit is not completed by
this source survey.

An operation-count probe imported the actual JS `bandwidth.allocate`, counted
`Want.priority` getter reads, and called it for every reservation, as
`Allocator.#publish` does. With one tier, 16/64/256 reservations caused
512/8,192/131,072 reads, exactly 2N². With distinct tiers, 16/64 caused
6,952/438,944 reads. Nix Bun 1.4.2 was used; no browser speedup is claimed.
The bandwidth child must retain a reproducible benchmark, including the
complete allocator and Rust consumers, before selecting a fix.

`python3 sh/gh/nightly.test.py NightlyTest.test_every_cargo_bench_target_runs_nightly`
reproduced the missing `moq-archive/producer` selection. The existing
[CI audit PR](https://github.com/moq-dev/moq/pull/6266) already fixes it.
No duplicate quest or workflow edit belongs here.

### Existing ownership and deliberately deferred work

- [Release performance](/quest/m1/release-perf/README.md) owns sparse gaps,
resume duplicates, idle GC, stats drain, FETCH fan-out, JS frame regressions,
and paired comparison correctness. Active PRs already address these.
- [Perf](/quest/m2/perf/README.md) owns measured channel contention, relay
allocations, ingest batching, worker quiescence and runtime selection.
- [Release audit PR](https://github.com/moq-dev/moq/pull/6311) owns QMux
copying/resource growth, CMAF allocation abuse, media reorder queues,
watch teardown, and HLS contract investigations. MKV fragmentation below
concerns parser replay before those queues; recheck that PR before coding.
- [Binding bytes](/quest/m2/ffi-bytes.md),
[foreign polling](/quest/m2/ffi-poll-protocol.md),
[Kotlin calls](/quest/m2/kt-ffi-calls.md),
[NVENC pooling](/quest/m2/nvenc-pool.md), and
[watch worker planning](/quest/m2/plan-watch-worker.md) retain their scopes.
- [Route wakes](/quest/m2/route-wakes.md),
[keyed prefix polling](/quest/m2/keyed-prefix-polling.md), and
[front parking](/quest/m2/origin-front-parks.md) own discovery scans/wakes.
- The [recurring audit](/quest/m1/audit-perf.md) retains real congestion-
controller/loss-recovery coverage, broader importer/container coverage,
and publish Fanout/audio-ring profiling. QUIC protocol microbenchmarks use
`Unlimited`; their numbers cannot establish controller performance. Expand
reassembly flight/reordering coverage there before proposing another index.
[BBR study](/quest/m2/quic-bbr-natural-drain.md) owns controller behavior.
- Archive reader internals are being replaced by the
[archive consumer](/quest/m1/archive/consumer.md). Do not optimize the
outgoing timeline reader. Benchmark the landed replacement when revisiting it.

## Required

- [Fragmented QUIC sends](/quest/m2/performance-survey/quic-send-fragments.md) - packetization cost under a retained fragmented flight
- [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - polling and memory after peak task occupancy subsides
- [Bandwidth grants](/quest/m2/performance-survey/bandwidth-grants.md) - repeated full allocation in Rust and JS
- [Cached JS group joins](/quest/m2/performance-survey/js-cached-groups.md) - repeated sorting while replaying retained groups
- [PCM batch processing](/quest/m2/performance-survey/audio-batches.md) - repeated tail movement during encoding and resampling
- [Fragmented MKV import](/quest/m2/performance-survey/mkv-fragments.md) - accumulated-buffer copies and parser replay
- [QUIC scheduling churn](/quest/m2/performance-survey/quic-schedule-churn.md) - priority and retirement work across unrelated groups
- [HLS request fan-out](/quest/m2/performance-survey/hls-serving.md) - repeated fetch, remux and playlist construction per viewer
- [Buffered browser groups](/quest/m2/performance-survey/hang-buffered.md) - per-frame scans and bulk skip work
- [Session datagrams](/quest/m2/performance-survey/datagram-delivery.md) - complete per-object delivery across publishers and viewers
- [JWT verification](/quest/m2/performance-survey/jwt-verification.md) - key loading, lookup, parsing and crypto attribution
- [Announcement snapshots](/quest/m2/performance-survey/announced-snapshots.md) - HTTP polling cost versus result size and unrelated routes
- [Selective session operations](/quest/m2/performance-survey/session-selection.md) - exact-ID lookups and registry lock hold time
- [Browser media copies](/quest/m2/performance-survey/browser-media-copies.md) - CMAF sample and LOC framing copies versus retention
- [GStreamer ingest buffers](/quest/m2/performance-survey/gst-buffers.md) - copy avoidance versus upstream pool retention

## Related

- [Performance audit](/quest/m1/audit-perf.md) - nightly regression attribution and remaining coverage
- [Benchmark sources](/quest/m2/bench-sources.md) - publishing existing measurements to the trend table
- [Nightly comparisons](/quest/m2/bench-ci.md) - paired nightly regression reporting
32 changes: 32 additions & 0 deletions quest/m2/performance-survey/announced-snapshots.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# [M] Measure HTTP announcement snapshot fan-out

## Goal

Repeated announcement-list polling has measured cost proportional to useful
results, with avoidable copying or unrelated-route work removed if worthwhile.

## Plan

[serve_announced](/rs/moq-relay/src/web.rs) creates an announcement consumer,
collects prefixes, creates another Vec of strings and joins them per request
(lines 1011-1044). Some output-size cost is necessary; repeated snapshot and
formatting work is the hypothesis, not proof of a bottleneck.

Sweep registered routes × HTTP pollers, independently varying matched prefix
size, route churn and path length. Compare an empty selective prefix, a small
result among many unrelated routes and the full listing. Measure CPU/request,
allocations/copied bytes, response latency and route-update latency while
polling. Exercise the real authorization/view path, and attribute its cost
separately from rendering.

Preserve admission, tenant isolation, hidden-prefix rules, response contents
and visibility during changes. Any shared snapshot must follow authorization
and content identity; avoid a global credential-blind cache. Prefer removing
intermediate allocations over a second route index unless measurements earn it.
This takes the recurring audit's HTTP announcement item; discovery wake/index
quests retain ownership of the underlying routing machinery.

Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md):
measure the baseline before optimizing, retain paired evidence, verify equivalent
work, schedule coverage nightly, and accept a measured no-win. Expected public
API and wire impact: none; report the actual impact in the implementation PR.
32 changes: 32 additions & 0 deletions quest/m2/performance-survey/audio-batches.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# [M] PCM batching avoids repeated tail movement

## Goal

Encoding and resampling larger PCM batches does not repeatedly move the
same remaining samples, while preserving audio and timestamp behavior.

## Plan

[Producer::publish_full_frames](/rs/moq-audio/src/encode/producer.rs)
front-drains into a new Vec for each packet after appending the entire input
(lines 379-389). [Resampler::convert](/rs/moq-audio/src/resample.rs) likewise
front-drains after every chunk (line 291). N chunks in a call can move a
triangular number of trailing samples. Capture and binding producers use these
paths; small normal capture periods may hide the cost.

Benchmark a constant duration of audio delivered in different batch sizes,
channels and input/output rate pairs, also sweeping independent producers.
Separate buffering/resampling from codec CPU with a cheap supported codec
control. Measure instructions/sample, allocations, copied bytes, throughput
and worst call duration, including tiny steady batches and large bursts.
Verify PCM or decoded output as appropriate, sample counts, packet boundaries,
timestamps, partial tails, flush and discontinuities.

Processing slices and compacting only the remainder is a candidate; do not
prescribe a queue type before measuring. This does not change capture period
policy or the [Opus grant](/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md).

Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md):
measure the baseline before optimizing, retain paired evidence, verify equivalent
work, schedule coverage nightly, and accept a measured no-win. Expected public
API and wire impact: none; report the actual impact in the implementation PR.
50 changes: 50 additions & 0 deletions quest/m2/performance-survey/bandwidth-grants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# [M] Compute bandwidth grants without repeated whole-table work

## Goal

Rust and browser publishers receive the same fair bandwidth grants with
less repeated allocation work, when measured across real observers.

## Plan

[JS allocation](/js/net/src/bandwidth.ts) computes each reservation's grant
separately. Each call scans all wants, filters/sorts each priority tier, and
searches again for the next tier. Synchronous peek repeats it. The
[Rust sibling](/rs/moq-net/src/model/bandwidth.rs) also snapshots claims and
polls demand per share. One estimate change can duplicate work across every
observer. The survey records deterministic JS priority-read counts, including
2N² reads for all grants in one tier, before sorting.

Reproduce that operation count with the repository's Bun (not a latency test):

```bash
bun -e '
import { allocate } from "./js/net/src/bandwidth.ts";
for (const tiers of [1, 8, 64]) for (const n of [16, 64, 256]) {
let reads = 0;
const wants = Array.from({ length: n }, (_, id) => ({
id, get priority() { reads++; return id % tiers; }, max: 1000,
}));
let sum = 0;
for (let id = 0; id < n; id++) sum += allocate(n * 500, wants, id) ?? 0;
console.log({ n, tiers: Math.min(tiers, n), reads, sum });
}'
```

Extend the Rust bandwidth target beyond its existing unlimited-allocator
probe-target sum; add a real-browser counterpart. Sweep connections ×
reservations, priority diversity and observer count, with demanded/idle mixes.
Measure estimate changes, demand transitions, ceiling changes and per-frame
peek separately: CPU, allocations, waker/listener registrations and browser
main-thread delay. Assert every grant, not merely their sum.

Consider calculating a shared allocation once per relevant change, but retain
strict priority, max-min fairness, rounding, ties, immediate peek behavior and
closed-demand removal. Cover both languages in this quest. Coordinate with
[viewer targets](/quest/m2/probe/viewer-target.md), which changes how the same
allocator feeds probes.

Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md):
measure the baseline before optimizing, retain paired evidence, verify equivalent
work, schedule coverage nightly, and accept a measured no-win. Expected public
API and wire impact: none; report the actual impact in the implementation PR.
Loading
Loading