diff --git a/quest/m2/perf/shard-default.md b/quest/m2/perf/shard-default.md index 076c07004f0..4adea1eafdd 100644 --- a/quest/m2/perf/shard-default.md +++ b/quest/m2/perf/shard-default.md @@ -22,6 +22,18 @@ way until a measurement justifies the flip. runtimes. - Flip the default only if sharding raises the throughput ceiling or cuts CPU per connection at saturation. A measured no-win deletes this quest. +- Include physical-core, SMT and NIC-local placement on multi-NUMA hardware + against current pinning and an unpinned control. CPU selection currently + walks allowed logical CPU IDs; it does not coordinate NIC RSS queues or IRQ + affinity. Workers already pin before ring/pool allocation, so do not assume + all pool memory starts on the main thread's node. Sweep workers × flows and + publishers × subscribers, with quotas and multiple relay processes. Record + per-core softirq time, migrations, remote memory/cache misses, CPU per useful + byte, memory per worker and tail latency. Keep CID ownership intact while + evaluating RSS/RPS/XPS alignment; the Linux + [network scaling guide](https://docs.kernel.org/networking/scaling.html) + explains their separate roles. Hardware-specific results must not silently + become a universal pinning default. - Shard only where a group serves the same listener the shared runtime would open. A stream-only relay, or one with a QUIC-LB `lb_id`, stays shared unless a count is set. A count sized from `available_parallelism`, capped at @@ -51,3 +63,5 @@ Update `doc/bin/relay/` for the default change. - [kio channel contention](/quest/m2/perf/kio-channel-contention/README.md) - its per-thread spikes run on the workers - [UDP demux](/quest/m2/p2p/udp-demux.md) - demuxes each shard's socket +- [Performance survey](/quest/m2/performance-survey/README.md) - paired benchmark evidence and memory/settings controls +- [Socket queues](/quest/m2/performance-survey/socket-queues.md) - queue sizing and local drop attribution diff --git a/quest/m2/performance-survey/README.md b/quest/m2/performance-survey/README.md index 1efe689c87c..587754fa520 100644 --- a/quest/m2/performance-survey/README.md +++ b/quest/m2/performance-survey/README.md @@ -23,6 +23,12 @@ contracts just to improve a number; bring a necessary contract change to the maintainer with the measurement first. No new user guide is needed for this planning change; implementations update documentation they make stale. +The follow-up audit on main +[`d44813cb703b`](https://github.com/moq-dev/moq/commit/d44813cb703bf0d5366266ae8206f794856c69f8) +adds memory lifetime, idle/churn scaling, kernel mechanisms and settings. +These additional findings are source-supported; no new memory or kernel +benchmark result is claimed. + ### Benchmark contract for every child - Reproduce the suspected cost in the production path before changing it. @@ -37,6 +43,16 @@ planning change; implementations update documentation they make stale. allocations, copied bytes, retained memory, and tail latency where relevant. Instruction counts do not measure blocking or prove end-to-end throughput. Setup and teardown boundaries must match between variants. +- For memory, include cold start, burst, quiet, repeat burst and teardown. + Separate logical cache/flow-control charges, live allocations, allocator + active/resident pages, virtual retained mappings and kernel socket memory. + RSS alone cannot establish a leak or identify its owner. Record allocator, + profiling mode and cgroup memory pressure. Only investigate allocator decay, + arenas or background purging after application-owned retention is attributed; + compare reclaim latency and CPU/fault cost on reuse. Reuse the phases in + [heap profiling](/bench/profile.py). The + [jemalloc statistics definitions](https://jemalloc.net/jemalloc.3.html) + distinguish allocated, active, resident and retained memory. - Measure JS changes in a real browser with the existing browser harness. Bun operation counts can demonstrate an algorithmic slope but cannot stand in for browser latency, GC, native copies, or hardware behavior. @@ -52,6 +68,20 @@ planning change; implementations update documentation they make stale. complexity and memory tradeoffs. Otherwise retain the useful benchmark and record why the implementation was left alone. +For settings sweeps, record effective values, not only requested ones. +[Resolved client defaults](/rs/moq-tokio/src/quic.rs) currently allow 10,000 +streams per direction, 64 MiB connection receive credit, 4 MiB per-stream +credit and 16 MiB send credit. [Relay peer overrides](/rs/moq-relay/src/limits.rs) +allow 100,000 streams and 640 MiB each of connection receive and send credit. +Standalone backend defaults differ again; benchmark the effective settings +of the actual consumer. These are limits, not per-connection allocations. +Include idle connections and stalled +readers as well as high-bandwidth, high-RTT relay links before recommending +smaller windows. Receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)) own budget +correctness; settings must preserve useful bandwidth-delay product and +failover behavior. [Keepalive](/quest/m2/quic-keep-alive.md) already owns timer +policy. Do not add a generic tuning surface without a measured consumer. + Reuse [the benchmark harness](/bench/README.md). Criterion's [timing-loop guidance](https://bheisler.github.io/criterion.rs/book/user_guide/timing_loops.html) explains setup/drop boundaries; the @@ -81,6 +111,36 @@ reproduced the missing `moq-archive/producer` selection. The existing [CI audit PR](https://github.com/moq-dev/moq/pull/6266) already fixes it. No duplicate quest or workflow edit belongs here. +### Kernel and placement assessment + +- Already implemented in [moq-uring](/rs/moq-uring/README.md): multishot + receive, provided buffers, GRO/GSO, deferred/cooperative task running, + coalesced futex wakes and per-worker CID-steered sockets. Measure these + with existing ablations before adding another mechanism. +- [Ring submission setup](/quest/m2/3199-moq-uring-remove-sq-indirection-and-per-enter-ring-fd.md) + and [completion batching](/quest/m2/3200-moq-uring-batch-completion-wakeups-with-min-timeout.md) + already own the small ring experiments. Batching must preserve deadlines + and latency tails, not merely reduce syscalls. +- [Zero-copy sends](/quest/m3/3201-moq-uring-use-sendmsg-zc-for-large-udp-gso-trains.md) + need physical-NIC evidence: the previous loopback prototype lost 6-9% + while exercising copy fallback. Deferred completion ownership, pinned + memory and CQ pressure are costs to measure, not free benefits. + [QUIC I/O ownership](/quest/m2/quic-io-boundary.md) owns the buffer seam. +- Extend [worker defaults](/quest/m2/perf/shard-default.md) with NUMA, SMT, + NIC queues and CPU quotas. Application CID steering does not establish + matching NIC/softirq locality; see the Linux + [network scaling guide](https://docs.kernel.org/networking/scaling.html). +- Linux [io_uring zero-copy receive](https://docs.kernel.org/networking/iou-zcrx.html) + describes a TCP receive path with hardware header/data split and dedicated + queues. It is not evidence of a drop-in optimization for this UDP path. + Confirm transport, hardware and minimum-kernel support before a future trial. +- NAPI busy polling and registered wait arguments were explicitly dropped in + [Perf](/quest/m2/perf/README.md). Do not reopen them without new profiling + evidence. SQPOLL, huge pages and blanket page locking likewise have no + demonstrated need here; they can exchange CPU or reclaimable memory for + lower latency. Incremental provided-buffer consumption is deliberately + unsuitable for the current multishot `recvmsg` tail behavior. + ### Existing ownership and deliberately deferred work - [Release performance](/quest/m1/release-perf/README.md) owns sparse gaps, @@ -113,6 +173,9 @@ No duplicate quest or workflow edit belongs here. ## Required - [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - polling and memory after peak task occupancy subsides +- [QUIC receive-state reuse](/quest/m2/performance-survey/quic-receive-reuse.md) - reassembly metadata retained by recycled completed streams +- [Cache metadata](/quest/m2/performance-survey/cache-metadata.md) - bookkeeping capacity after cached groups expire +- [Cache backing amplification](/quest/m2/performance-survey/cache-backing.md) - small retained samples pinning large shared fragments - [Bandwidth grants](/quest/m2/performance-survey/bandwidth-grants.md) - repeated full allocation in Rust and JS - [Cached JS group joins](/quest/m2/performance-survey/js-cached-groups.md) - repeated sorting while replaying retained groups - [PCM batch processing](/quest/m2/performance-survey/audio-batches.md) - repeated tail movement during encoding and resampling @@ -120,6 +183,10 @@ No duplicate quest or workflow edit belongs here. - [QUIC scheduling churn](/quest/m2/performance-survey/quic-schedule-churn.md) - priority and retirement work across unrelated groups - [HLS request fan-out](/quest/m2/performance-survey/hls-serving.md) - repeated fetch, remux and playlist construction per viewer - [Buffered browser groups](/quest/m2/performance-survey/hang-buffered.md) - per-frame scans and bulk skip work +- [Idle browser stats](/quest/m2/performance-survey/browser-idle-stats.md) - local transport polling without active publishing demand +- [Transport preference history](/quest/m2/performance-survey/transport-history.md) - full URL retention across retired sessions +- [Resume identity history](/quest/m2/performance-survey/resume-history.md) - historical source graphs held by first-seen track metadata +- [UDP socket queues](/quest/m2/performance-survey/socket-queues.md) - drop attribution and queue-size tradeoffs across workers and load - [Session datagrams](/quest/m2/performance-survey/datagram-delivery.md) - complete per-object delivery across publishers and viewers - [JWT verification](/quest/m2/performance-survey/jwt-verification.md) - key loading, lookup, parsing and crypto attribution - [Announcement snapshots](/quest/m2/performance-survey/announced-snapshots.md) - HTTP polling cost versus result size and unrelated routes diff --git a/quest/m2/performance-survey/browser-idle-stats.md b/quest/m2/performance-survey/browser-idle-stats.md new file mode 100644 index 00000000000..6ab8cfd8e2b --- /dev/null +++ b/quest/m2/performance-survey/browser-idle-stats.md @@ -0,0 +1,26 @@ +# [M] Local bandwidth sampling follows demand + +## Goal + +Browser connections avoid unnecessary local transport-stat work while +preserving useful send-rate estimates when publishing needs them. + +## Plan + +[Reload](/js/net/src/connection/reload.ts) samples `connection.stats()` every +100 ms whenever established, independently of active bandwidth reservations. +This includes receive-only connections and pooled sessions during linger. +[QMux stats](/js/net/src/connection/stats.ts) still allocates an empty result +when the transport has no `getStats`. Remote PROBE observation is separate. + +In the real browser harness, sweep distinct connections × active publishing +reservations. Include zero reservations, receive-only playback, active/idle +transitions, pool linger and QMux. Identical pooled handles are a control, +not distinct network connections. Count timers, real `getStats` calls, +allocations/GC and main-thread CPU; measure first useful estimate latency +after publishing resumes. Keep current cadence as the baseline before +choosing demand gating or any changed sampling policy. + +Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): +baseline first, paired evidence, equivalent work and nightly coverage. A +measured no-win is valid. No API or wire change is expected; report actual impact. diff --git a/quest/m2/performance-survey/cache-backing.md b/quest/m2/performance-survey/cache-backing.md new file mode 100644 index 00000000000..0afb507802d --- /dev/null +++ b/quest/m2/performance-survey/cache-backing.md @@ -0,0 +1,33 @@ +# [M] Measure shared cache backing amplification + +## Goal + +Small retained media samples do not pin disproportionately large backing +allocations when a measured alternative earns its copying cost. + +## Plan + +[Group storage](/rs/moq-net/src/model/group.rs) bills `Bytes::len()` and stores +the supplied view. [fMP4 import](/rs/moq-mux/src/container/fmp4/import.rs) +slices samples from a fragment. Selective expiry across groups or tracks can +leave a small sample holding a large fragment after most logical charges +disappear. Cache capacity is not a promised hard RSS limit; this is a backing +amplification hypothesis, not proof of a broken capacity contract. + +Benchmark the actual importer and cache, sweeping fragment size × retained +sample fraction × tracks. Include mixed audio/video, several keyframe groups +in a fragment, selective expiry and independently allocated samples as a +control. Measure uniquely owned backing bytes versus `Pool::used()`, copy +volume, allocations, CPU and delivery latency. Do not double count shared +backing or assume each slice owns a fresh allocation. Compare continued +sharing with copying/compaction only where the workload warrants it. + +This covers completed reliable groups. Coordinate with +receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)), +[browser media copies](/quest/m2/performance-survey/browser-media-copies.md), +and the [release audit](https://github.com/moq-dev/moq/pull/6311), which owns +decoded datagrams and incomplete/reordered media storage. + +Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): +baseline first, paired evidence, equivalent work and nightly coverage. A +measured no-win is valid. No API or wire change is expected; report actual impact. diff --git a/quest/m2/performance-survey/cache-metadata.md b/quest/m2/performance-survey/cache-metadata.md new file mode 100644 index 00000000000..5d90ec737fc --- /dev/null +++ b/quest/m2/performance-survey/cache-metadata.md @@ -0,0 +1,30 @@ +# [M] Cache metadata follows live occupancy + +## Goal + +Tracks that expire a large cache do not retain unjustified bookkeeping +capacity after their payload charges fall. + +## Plan + +[Track expiry](/rs/moq-net/src/model/track.rs) pops or retains entries in +`arrival` and `evict` without shrinking backing storage. The `CACHE_OVERHEAD` +rationale assumes a deque holds at most twice its current entries; draining +does not preserve that assumption. Long-lived relay tracks can therefore +retain historical queue capacity after `Pool::used()` falls. + +Sweep tracks × peak cached groups, expire to the protected live group, then +refill. Record queue capacities, logical charges, live allocation bytes and +RSS separately. Use simulated time for expiry correctness and repeated +grow/drain cycles to expose reclamation CPU, copies and allocation churn. +Preserve arrival cursors, live-edge protection and useful cache retention. +Correct the accounting explanation with the chosen implementation. + +Coordinate with [group storage](/quest/m2/group-storage-bench.md) and +[log adoption](/quest/m2/perf/kio-channel-contention/log-adoption.md) if their +representation has changed. [Idle GC](/quest/m1/release-perf/gc-idle.md) owns +unnecessary clock/lock work, not retained queue backing. + +Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): +baseline first, paired evidence, equivalent work and nightly coverage. A +measured no-win is valid. No API or wire change is expected; report actual impact. diff --git a/quest/m2/performance-survey/quic-receive-reuse.md b/quest/m2/performance-survey/quic-receive-reuse.md new file mode 100644 index 00000000000..97e9b75cce5 --- /dev/null +++ b/quest/m2/performance-survey/quic-receive-reuse.md @@ -0,0 +1,27 @@ +# [M] QUIC receive state recovers after bursts + +## Goal + +Long-lived QUIC connections do not retain unjustified reassembly metadata +from completed streams, without losing the benefit of cheap stream reuse. + +## Plan + +[Stream recycling](/rs/moq-quic/src/connection/streams/state.rs) puts freed +receive state in `free_recv`. [Assembler reset](/rs/moq-quic/src/connection/assembler.rs) +preserves its deque allocation and clears entries. Payload release therefore +does not release historical chunk capacity. Relay connections that once +received many fragmented streams are the consumer; this is separate from +unread payload credit, which receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)) bound. + +Sweep connections × peak concurrent streams × fragments, followed by stream +completion and an idle phase. Include ordered traffic, gaps/reordering, +cancellation, and another burst. Measure live allocated metadata, recycled +state counts/capacities, allocation rate, CPU and latency when traffic resumes. +Distinguish application-owned allocations from allocator-resident pages. +Any reclamation must preserve stream credit and avoid repeated allocation +thrashing. Build on the receive leases from [moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878). + +Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): +baseline first, paired evidence, equivalent work and nightly coverage. A +measured no-win is valid. No API or wire change is expected; report actual impact. diff --git a/quest/m2/performance-survey/resume-history.md b/quest/m2/performance-survey/resume-history.md new file mode 100644 index 00000000000..ef5bc47230e --- /dev/null +++ b/quest/m2/performance-survey/resume-history.md @@ -0,0 +1,31 @@ +# [M] Resume identity retains only necessary state + +## Goal + +Long-lived browser fronts preserve track identity across routes without +unnecessarily retaining historical source object graphs. + +## Plan + +[Front metadata](/js/net/src/resume.ts) records the first `{info, source}` for +each track. A `Source` owns broadcast handles and a wire view; releasing its +last hold closes the cloned consumer but leaves those strong references. +This establishes a retained JS path, not that a closed socket or its native +buffers remain live. A fixed track set pins first-answer identities, not a +new source for every reconnect. Track-name churn plus route turnover is the +important adverse case. + +Sweep fronts × distinct first-seen track names, independently varying route +replacements. Compare one stable track across many routes against new names +first answered by each route. Close readers after lookup, fetch and subscribe; +after cache expiry and browser GC, inspect heap dominators, retained source/ +subscriber/transport objects and payload bytes. Measure transition CPU too. + +Keep the metadata proof: the same name must still mean compatible content. +Preserve incompatibility refusal, concurrent cancellation and current-source +reuse. Coordinate with [route health](/quest/m2/route-health/README.md) before +changing identity representation; do not delete history just to reduce heap. + +Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): +baseline first, paired evidence, equivalent work and nightly coverage. A +measured no-win is valid. No API or wire change is expected; report actual impact. diff --git a/quest/m2/performance-survey/socket-queues.md b/quest/m2/performance-survey/socket-queues.md new file mode 100644 index 00000000000..7a6d5814d4e --- /dev/null +++ b/quest/m2/performance-survey/socket-queues.md @@ -0,0 +1,36 @@ +# [L] Attribute UDP queue loss before tuning buffers + +## Goal + +Relay socket-buffer settings follow measured loss, queue delay and memory +cost across useful load, with local starvation separated from packet drops. + +## Plan + +[Socket binding](/rs/moq-sock/src/bind.rs) requests 8 MiB receive and send +buffers and already reads back OS clamping. These are ceilings, not eager +allocations. The io_uring `rx_enobufs` metric counts provided-buffer +starvation, not confirmed packet loss. Neither UDP receive stack currently +parses `SO_RXQ_OVFL` for socket-drop attribution. + +Extend benchmark instrumentation to record granted queue sizes and actual +local drops. Linux [socket documentation](https://man7.org/linux/man-pages/man7/socket.7.html) +describes `SO_RXQ_OVFL`; account for ancillary space, counter rollover, GRO, +and the fact that missing socket-drop evidence cannot locate every loss. +Keep instrumentation private unless a production consumer needs it. + +Sweep workers × offered flows and publishers × subscribers, with kernel +queue size and userspace pool depth varied independently. Include CPU +contention, bursts and rate-limited deep finite queues with overflow drops. +Report useful goodput, latency, CPU, memory and local versus path loss. Check +both shared and io_uring receive paths. Retain fixed defaults if measurements +do not justify a new configuration surface; larger queues can worsen delay. + +Coordinate with the UDP pool memory quest planned in +[moq-dev/moq#6350](https://github.com/moq-dev/moq/pull/6350) and +[runtime selection](/quest/m2/perf/shard-default.md), sharing fixtures and +reports without making independent experiments block each other. + +Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): +baseline first, paired evidence, equivalent work and nightly coverage. A +measured no-win is valid. Report actual API, configuration and wire impact. diff --git a/quest/m2/performance-survey/transport-history.md b/quest/m2/performance-survey/transport-history.md new file mode 100644 index 00000000000..0630eb0579c --- /dev/null +++ b/quest/m2/performance-survey/transport-history.md @@ -0,0 +1,30 @@ +# [M] Transport preference has a justified lifetime + +## Goal + +Connection racing retains useful fallback knowledge without accumulating +unneeded URL history for the whole page lifetime. + +## Plan + +[Connection racing](/js/net/src/connection/connect.ts) stores each WebSocket +winning full URL in module-level `websocketWon`, with no deletion. A later +WebTransport win does not clear it. Unique paths or rotating query strings +remain after pooled connections expire. Comments call this the last race, +while current behavior remembers whether WebSocket ever won. + +Sweep unique URLs × reconnects per URL, varying URL length and synthetic +query rotation. Close sessions, advance beyond pool linger and force GC in +the browser harness. Measure retained strings/heap, connection latency and +transport selection. Include WebSocket then WebTransport wins, alternate +fallback URLs and disabled fallback. Never put real credentials in reports. + +Establish the useful ownership lifetime before changing it. Do not silently +collapse endpoint-specific behavior into origin-only keys or pick an +arbitrary TTL; bring a necessary racing-policy decision to the maintainer. +Recheck the [release audit](https://github.com/moq-dev/moq/pull/6311) for overlap +before implementation; its current QMux queue scope does not own this set. + +Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): +baseline first, paired evidence, equivalent work and nightly coverage. A +measured no-win is valid. No API or wire change is expected; report actual impact.