Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions quest/m2/perf/shard-default.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,18 @@ way until a measurement justifies the flip.
runtimes.
- Flip the default only if sharding raises the throughput ceiling or cuts
CPU per connection at saturation. A measured no-win deletes this quest.
- Include physical-core, SMT and NIC-local placement on multi-NUMA hardware
against current pinning and an unpinned control. CPU selection currently
walks allowed logical CPU IDs; it does not coordinate NIC RSS queues or IRQ
affinity. Workers already pin before ring/pool allocation, so do not assume
all pool memory starts on the main thread's node. Sweep workers × flows and
publishers × subscribers, with quotas and multiple relay processes. Record
per-core softirq time, migrations, remote memory/cache misses, CPU per useful
byte, memory per worker and tail latency. Keep CID ownership intact while
evaluating RSS/RPS/XPS alignment; the Linux
[network scaling guide](https://docs.kernel.org/networking/scaling.html)
explains their separate roles. Hardware-specific results must not silently
become a universal pinning default.
- Shard only where a group serves the same listener the shared runtime would
open. A stream-only relay, or one with a QUIC-LB `lb_id`, stays shared
unless a count is set. A count sized from `available_parallelism`, capped at
Expand Down Expand Up @@ -51,3 +63,5 @@ Update `doc/bin/relay/` for the default change.

- [kio channel contention](/quest/m2/perf/kio-channel-contention/README.md) - its per-thread spikes run on the workers
- [UDP demux](/quest/m2/p2p/udp-demux.md) - demuxes each shard's socket
- [Performance survey](/quest/m2/performance-survey/README.md) - paired benchmark evidence and memory/settings controls
- [Socket queues](/quest/m2/performance-survey/socket-queues.md) - queue sizing and local drop attribution
67 changes: 67 additions & 0 deletions quest/m2/performance-survey/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ contracts just to improve a number; bring a necessary contract change to the
maintainer with the measurement first. No new user guide is needed for this
planning change; implementations update documentation they make stale.

The follow-up audit on main
[`d44813cb703b`](https://github.com/moq-dev/moq/commit/d44813cb703bf0d5366266ae8206f794856c69f8)
adds memory lifetime, idle/churn scaling, kernel mechanisms and settings.
These additional findings are source-supported; no new memory or kernel
benchmark result is claimed.

### Benchmark contract for every child

- Reproduce the suspected cost in the production path before changing it.
Expand All @@ -37,6 +43,16 @@ planning change; implementations update documentation they make stale.
allocations, copied bytes, retained memory, and tail latency where relevant.
Instruction counts do not measure blocking or prove end-to-end throughput.
Setup and teardown boundaries must match between variants.
- For memory, include cold start, burst, quiet, repeat burst and teardown.
Separate logical cache/flow-control charges, live allocations, allocator
active/resident pages, virtual retained mappings and kernel socket memory.
RSS alone cannot establish a leak or identify its owner. Record allocator,
profiling mode and cgroup memory pressure. Only investigate allocator decay,
arenas or background purging after application-owned retention is attributed;
compare reclaim latency and CPU/fault cost on reuse. Reuse the phases in
[heap profiling](/bench/profile.py). The
[jemalloc statistics definitions](https://jemalloc.net/jemalloc.3.html)
distinguish allocated, active, resident and retained memory.
- Measure JS changes in a real browser with the existing browser harness.
Bun operation counts can demonstrate an algorithmic slope but cannot stand
in for browser latency, GC, native copies, or hardware behavior.
Expand All @@ -52,6 +68,20 @@ planning change; implementations update documentation they make stale.
complexity and memory tradeoffs. Otherwise retain the useful benchmark and
record why the implementation was left alone.

For settings sweeps, record effective values, not only requested ones.
[Resolved client defaults](/rs/moq-tokio/src/quic.rs) currently allow 10,000
streams per direction, 64 MiB connection receive credit, 4 MiB per-stream

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the client receive-window default to 128 MiB

At the reviewed commit, rs/moq-tokio/src/quic.rs:119 defines DEFAULT_RECEIVE_WINDOW as 128 << 20, not 64 MiB. Because this paragraph establishes the effective baseline for subsequent settings sweeps, the incorrect value could cause experiments to compare against or explicitly configure the wrong baseline; update it to 128 MiB.

AGENTS.md reference: AGENTS.md:L32-L32

Useful? React with 👍 / 👎.

credit and 16 MiB send credit. [Relay peer overrides](/rs/moq-relay/src/limits.rs)
allow 100,000 streams and 640 MiB each of connection receive and send credit.
Standalone backend defaults differ again; benchmark the effective settings
of the actual consumer. These are limits, not per-connection allocations.
Include idle connections and stalled
readers as well as high-bandwidth, high-RTT relay links before recommending
smaller windows. Receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)) own budget
correctness; settings must preserve useful bandwidth-delay product and
failover behavior. [Keepalive](/quest/m2/quic-keep-alive.md) already owns timer
policy. Do not add a generic tuning surface without a measured consumer.

Reuse [the benchmark harness](/bench/README.md). Criterion's
[timing-loop guidance](https://bheisler.github.io/criterion.rs/book/user_guide/timing_loops.html)
explains setup/drop boundaries; the
Expand Down Expand Up @@ -81,6 +111,36 @@ reproduced the missing `moq-archive/producer` selection. The existing
[CI audit PR](https://github.com/moq-dev/moq/pull/6266) already fixes it.
No duplicate quest or workflow edit belongs here.

### Kernel and placement assessment

- Already implemented in [moq-uring](/rs/moq-uring/README.md): multishot
receive, provided buffers, GRO/GSO, deferred/cooperative task running,
coalesced futex wakes and per-worker CID-steered sockets. Measure these
with existing ablations before adding another mechanism.
- [Ring submission setup](/quest/m2/3199-moq-uring-remove-sq-indirection-and-per-enter-ring-fd.md)
and [completion batching](/quest/m2/3200-moq-uring-batch-completion-wakeups-with-min-timeout.md)
already own the small ring experiments. Batching must preserve deadlines
and latency tails, not merely reduce syscalls.
- [Zero-copy sends](/quest/m3/3201-moq-uring-use-sendmsg-zc-for-large-udp-gso-trains.md)
need physical-NIC evidence: the previous loopback prototype lost 6-9%
while exercising copy fallback. Deferred completion ownership, pinned
memory and CQ pressure are costs to measure, not free benefits.
[QUIC I/O ownership](/quest/m2/quic-io-boundary.md) owns the buffer seam.
- Extend [worker defaults](/quest/m2/perf/shard-default.md) with NUMA, SMT,
NIC queues and CPU quotas. Application CID steering does not establish
matching NIC/softirq locality; see the Linux
[network scaling guide](https://docs.kernel.org/networking/scaling.html).
- Linux [io_uring zero-copy receive](https://docs.kernel.org/networking/iou-zcrx.html)
describes a TCP receive path with hardware header/data split and dedicated
queues. It is not evidence of a drop-in optimization for this UDP path.
Confirm transport, hardware and minimum-kernel support before a future trial.
- NAPI busy polling and registered wait arguments were explicitly dropped in
[Perf](/quest/m2/perf/README.md). Do not reopen them without new profiling
evidence. SQPOLL, huge pages and blanket page locking likewise have no
demonstrated need here; they can exchange CPU or reclaimable memory for
lower latency. Incremental provided-buffer consumption is deliberately
unsuitable for the current multishot `recvmsg` tail behavior.

### Existing ownership and deliberately deferred work

- [Release performance](/quest/m1/release-perf/README.md) owns sparse gaps,
Expand Down Expand Up @@ -113,13 +173,20 @@ No duplicate quest or workflow edit belongs here.
## Required

- [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - polling and memory after peak task occupancy subsides
- [QUIC receive-state reuse](/quest/m2/performance-survey/quic-receive-reuse.md) - reassembly metadata retained by recycled completed streams
- [Cache metadata](/quest/m2/performance-survey/cache-metadata.md) - bookkeeping capacity after cached groups expire
- [Cache backing amplification](/quest/m2/performance-survey/cache-backing.md) - small retained samples pinning large shared fragments
- [Bandwidth grants](/quest/m2/performance-survey/bandwidth-grants.md) - repeated full allocation in Rust and JS
- [Cached JS group joins](/quest/m2/performance-survey/js-cached-groups.md) - repeated sorting while replaying retained groups
- [PCM batch processing](/quest/m2/performance-survey/audio-batches.md) - repeated tail movement during encoding and resampling
- [Fragmented MKV import](/quest/m2/performance-survey/mkv-fragments.md) - accumulated-buffer copies and parser replay
- [QUIC scheduling churn](/quest/m2/performance-survey/quic-schedule-churn.md) - priority and retirement work across unrelated groups
- [HLS request fan-out](/quest/m2/performance-survey/hls-serving.md) - repeated fetch, remux and playlist construction per viewer
- [Buffered browser groups](/quest/m2/performance-survey/hang-buffered.md) - per-frame scans and bulk skip work
- [Idle browser stats](/quest/m2/performance-survey/browser-idle-stats.md) - local transport polling without active publishing demand
- [Transport preference history](/quest/m2/performance-survey/transport-history.md) - full URL retention across retired sessions
- [Resume identity history](/quest/m2/performance-survey/resume-history.md) - historical source graphs held by first-seen track metadata
- [UDP socket queues](/quest/m2/performance-survey/socket-queues.md) - drop attribution and queue-size tradeoffs across workers and load
- [Session datagrams](/quest/m2/performance-survey/datagram-delivery.md) - complete per-object delivery across publishers and viewers
- [JWT verification](/quest/m2/performance-survey/jwt-verification.md) - key loading, lookup, parsing and crypto attribution
- [Announcement snapshots](/quest/m2/performance-survey/announced-snapshots.md) - HTTP polling cost versus result size and unrelated routes
Expand Down
26 changes: 26 additions & 0 deletions quest/m2/performance-survey/browser-idle-stats.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# [M] Local bandwidth sampling follows demand

## Goal

Browser connections avoid unnecessary local transport-stat work while
preserving useful send-rate estimates when publishing needs them.

## Plan

[Reload](/js/net/src/connection/reload.ts) samples `connection.stats()` every
100 ms whenever established, independently of active bandwidth reservations.
This includes receive-only connections and pooled sessions during linger.
[QMux stats](/js/net/src/connection/stats.ts) still allocates an empty result
when the transport has no `getStats`. Remote PROBE observation is separate.

In the real browser harness, sweep distinct connections × active publishing
reservations. Include zero reservations, receive-only playback, active/idle
transitions, pool linger and QMux. Identical pooled handles are a control,
not distinct network connections. Count timers, real `getStats` calls,
allocations/GC and main-thread CPU; measure first useful estimate latency
after publishing resumes. Keep current cadence as the baseline before
choosing demand gating or any changed sampling policy.

Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md):
baseline first, paired evidence, equivalent work and nightly coverage. A
measured no-win is valid. No API or wire change is expected; report actual impact.
33 changes: 33 additions & 0 deletions quest/m2/performance-survey/cache-backing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# [M] Measure shared cache backing amplification

## Goal

Small retained media samples do not pin disproportionately large backing
allocations when a measured alternative earns its copying cost.

## Plan

[Group storage](/rs/moq-net/src/model/group.rs) bills `Bytes::len()` and stores
the supplied view. [fMP4 import](/rs/moq-mux/src/container/fmp4/import.rs)
slices samples from a fragment. Selective expiry across groups or tracks can
leave a small sample holding a large fragment after most logical charges
disappear. Cache capacity is not a promised hard RSS limit; this is a backing
amplification hypothesis, not proof of a broken capacity contract.

Benchmark the actual importer and cache, sweeping fragment size × retained
sample fraction × tracks. Include mixed audio/video, several keyframe groups
in a fragment, selective expiry and independently allocated samples as a
control. Measure uniquely owned backing bytes versus `Pool::used()`, copy
volume, allocations, CPU and delivery latency. Do not double count shared
backing or assume each slice owns a fresh allocation. Compare continued
sharing with copying/compaction only where the workload warrants it.

This covers completed reliable groups. Coordinate with
receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)),
[browser media copies](/quest/m2/performance-survey/browser-media-copies.md),
and the [release audit](https://github.com/moq-dev/moq/pull/6311), which owns
decoded datagrams and incomplete/reordered media storage.

Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md):
baseline first, paired evidence, equivalent work and nightly coverage. A
measured no-win is valid. No API or wire change is expected; report actual impact.
30 changes: 30 additions & 0 deletions quest/m2/performance-survey/cache-metadata.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# [M] Cache metadata follows live occupancy

## Goal

Tracks that expire a large cache do not retain unjustified bookkeeping
capacity after their payload charges fall.

## Plan

[Track expiry](/rs/moq-net/src/model/track.rs) pops or retains entries in
`arrival` and `evict` without shrinking backing storage. The `CACHE_OVERHEAD`
rationale assumes a deque holds at most twice its current entries; draining
does not preserve that assumption. Long-lived relay tracks can therefore
retain historical queue capacity after `Pool::used()` falls.

Sweep tracks × peak cached groups, expire to the protected live group, then
refill. Record queue capacities, logical charges, live allocation bytes and
RSS separately. Use simulated time for expiry correctness and repeated
grow/drain cycles to expose reclamation CPU, copies and allocation churn.
Preserve arrival cursors, live-edge protection and useful cache retention.
Correct the accounting explanation with the chosen implementation.

Coordinate with [group storage](/quest/m2/group-storage-bench.md) and
[log adoption](/quest/m2/perf/kio-channel-contention/log-adoption.md) if their
representation has changed. [Idle GC](/quest/m1/release-perf/gc-idle.md) owns
unnecessary clock/lock work, not retained queue backing.

Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md):
baseline first, paired evidence, equivalent work and nightly coverage. A
measured no-win is valid. No API or wire change is expected; report actual impact.
27 changes: 27 additions & 0 deletions quest/m2/performance-survey/quic-receive-reuse.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# [M] QUIC receive state recovers after bursts

## Goal

Long-lived QUIC connections do not retain unjustified reassembly metadata
from completed streams, without losing the benefit of cheap stream reuse.

## Plan

[Stream recycling](/rs/moq-quic/src/connection/streams/state.rs) puts freed
receive state in `free_recv`. [Assembler reset](/rs/moq-quic/src/connection/assembler.rs)
preserves its deque allocation and clears entries. Payload release therefore
does not release historical chunk capacity. Relay connections that once
received many fragmented streams are the consumer; this is separate from
unread payload credit, which receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)) bound.

Sweep connections × peak concurrent streams × fragments, followed by stream
completion and an idle phase. Include ordered traffic, gaps/reordering,
cancellation, and another burst. Measure live allocated metadata, recycled
state counts/capacities, allocation rate, CPU and latency when traffic resumes.
Distinguish application-owned allocations from allocator-resident pages.
Any reclamation must preserve stream credit and avoid repeated allocation
thrashing. Build on the receive leases from [moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878).

Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md):
baseline first, paired evidence, equivalent work and nightly coverage. A
measured no-win is valid. No API or wire change is expected; report actual impact.
31 changes: 31 additions & 0 deletions quest/m2/performance-survey/resume-history.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# [M] Resume identity retains only necessary state

## Goal

Long-lived browser fronts preserve track identity across routes without
unnecessarily retaining historical source object graphs.

## Plan

[Front metadata](/js/net/src/resume.ts) records the first `{info, source}` for
each track. A `Source` owns broadcast handles and a wire view; releasing its
last hold closes the cloned consumer but leaves those strong references.
This establishes a retained JS path, not that a closed socket or its native
buffers remain live. A fixed track set pins first-answer identities, not a
new source for every reconnect. Track-name churn plus route turnover is the
important adverse case.

Sweep fronts × distinct first-seen track names, independently varying route
replacements. Compare one stable track across many routes against new names
first answered by each route. Close readers after lookup, fetch and subscribe;
after cache expiry and browser GC, inspect heap dominators, retained source/
subscriber/transport objects and payload bytes. Measure transition CPU too.

Keep the metadata proof: the same name must still mean compatible content.
Preserve incompatibility refusal, concurrent cancellation and current-source
reuse. Coordinate with [route health](/quest/m2/route-health/README.md) before
changing identity representation; do not delete history just to reduce heap.

Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md):
baseline first, paired evidence, equivalent work and nightly coverage. A
measured no-win is valid. No API or wire change is expected; report actual impact.
36 changes: 36 additions & 0 deletions quest/m2/performance-survey/socket-queues.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# [L] Attribute UDP queue loss before tuning buffers

## Goal

Relay socket-buffer settings follow measured loss, queue delay and memory
cost across useful load, with local starvation separated from packet drops.

## Plan

[Socket binding](/rs/moq-sock/src/bind.rs) requests 8 MiB receive and send
buffers and already reads back OS clamping. These are ceilings, not eager
allocations. The io_uring `rx_enobufs` metric counts provided-buffer
starvation, not confirmed packet loss. Neither UDP receive stack currently
parses `SO_RXQ_OVFL` for socket-drop attribution.

Extend benchmark instrumentation to record granted queue sizes and actual
local drops. Linux [socket documentation](https://man7.org/linux/man-pages/man7/socket.7.html)
describes `SO_RXQ_OVFL`; account for ancillary space, counter rollover, GRO,
and the fact that missing socket-drop evidence cannot locate every loss.
Keep instrumentation private unless a production consumer needs it.

Sweep workers × offered flows and publishers × subscribers, with kernel
queue size and userspace pool depth varied independently. Include CPU
contention, bursts and rate-limited deep finite queues with overflow drops.
Report useful goodput, latency, CPU, memory and local versus path loss. Check
both shared and io_uring receive paths. Retain fixed defaults if measurements
do not justify a new configuration surface; larger queues can worsen delay.

Coordinate with the UDP pool memory quest planned in
[moq-dev/moq#6350](https://github.com/moq-dev/moq/pull/6350) and
[runtime selection](/quest/m2/perf/shard-default.md), sharing fixtures and
reports without making independent experiments block each other.

Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md):
baseline first, paired evidence, equivalent work and nightly coverage. A
measured no-win is valid. Report actual API, configuration and wire impact.
Loading
Loading