Repository navigation
quest: audit memory retention and kernel settings #6332
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
659b66d
quest: scope benchmark-first performance audit findings
kixelated 32c7c7f
Merge remote-tracking branch 'origin/main' into codex/performance-aud…
kixelated f82b2c7
quest: extend performance audit to memory and kernel settings
kixelated b27b05c
Merge remote-tracking branch 'origin/main' into fix-6332
kixelated 1b829d2
quest: defer UDP pool memory to #6350's uring-pool-memory
kixelated 4869af5
Merge remote-tracking branch 'origin/main' into fix-6332
kixelated f7405e7
quest: point receive-memory links at the landed receive leases
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # [M] Local bandwidth sampling follows demand | ||
|
|
||
| ## Goal | ||
|
|
||
| Browser connections avoid unnecessary local transport-stat work while | ||
| preserving useful send-rate estimates when publishing needs them. | ||
|
|
||
| ## Plan | ||
|
|
||
| [Reload](/js/net/src/connection/reload.ts) samples `connection.stats()` every | ||
| 100 ms whenever established, independently of active bandwidth reservations. | ||
| This includes receive-only connections and pooled sessions during linger. | ||
| [QMux stats](/js/net/src/connection/stats.ts) still allocates an empty result | ||
| when the transport has no `getStats`. Remote PROBE observation is separate. | ||
|
|
||
| In the real browser harness, sweep distinct connections × active publishing | ||
| reservations. Include zero reservations, receive-only playback, active/idle | ||
| transitions, pool linger and QMux. Identical pooled handles are a control, | ||
| not distinct network connections. Count timers, real `getStats` calls, | ||
| allocations/GC and main-thread CPU; measure first useful estimate latency | ||
| after publishing resumes. Keep current cadence as the baseline before | ||
| choosing demand gating or any changed sampling policy. | ||
|
|
||
| Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): | ||
| baseline first, paired evidence, equivalent work and nightly coverage. A | ||
| measured no-win is valid. No API or wire change is expected; report actual impact. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # [M] Measure shared cache backing amplification | ||
|
|
||
| ## Goal | ||
|
|
||
| Small retained media samples do not pin disproportionately large backing | ||
| allocations when a measured alternative earns its copying cost. | ||
|
|
||
| ## Plan | ||
|
|
||
| [Group storage](/rs/moq-net/src/model/group.rs) bills `Bytes::len()` and stores | ||
| the supplied view. [fMP4 import](/rs/moq-mux/src/container/fmp4/import.rs) | ||
| slices samples from a fragment. Selective expiry across groups or tracks can | ||
| leave a small sample holding a large fragment after most logical charges | ||
| disappear. Cache capacity is not a promised hard RSS limit; this is a backing | ||
| amplification hypothesis, not proof of a broken capacity contract. | ||
|
|
||
| Benchmark the actual importer and cache, sweeping fragment size × retained | ||
| sample fraction × tracks. Include mixed audio/video, several keyframe groups | ||
| in a fragment, selective expiry and independently allocated samples as a | ||
| control. Measure uniquely owned backing bytes versus `Pool::used()`, copy | ||
| volume, allocations, CPU and delivery latency. Do not double count shared | ||
| backing or assume each slice owns a fresh allocation. Compare continued | ||
| sharing with copying/compaction only where the workload warrants it. | ||
|
|
||
| This covers completed reliable groups. Coordinate with | ||
| receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)), | ||
| [browser media copies](/quest/m2/performance-survey/browser-media-copies.md), | ||
| and the [release audit](https://github.com/moq-dev/moq/pull/6311), which owns | ||
| decoded datagrams and incomplete/reordered media storage. | ||
|
|
||
| Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): | ||
| baseline first, paired evidence, equivalent work and nightly coverage. A | ||
| measured no-win is valid. No API or wire change is expected; report actual impact. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # [M] Cache metadata follows live occupancy | ||
|
|
||
| ## Goal | ||
|
|
||
| Tracks that expire a large cache do not retain unjustified bookkeeping | ||
| capacity after their payload charges fall. | ||
|
|
||
| ## Plan | ||
|
|
||
| [Track expiry](/rs/moq-net/src/model/track.rs) pops or retains entries in | ||
| `arrival` and `evict` without shrinking backing storage. The `CACHE_OVERHEAD` | ||
| rationale assumes a deque holds at most twice its current entries; draining | ||
| does not preserve that assumption. Long-lived relay tracks can therefore | ||
| retain historical queue capacity after `Pool::used()` falls. | ||
|
|
||
| Sweep tracks × peak cached groups, expire to the protected live group, then | ||
| refill. Record queue capacities, logical charges, live allocation bytes and | ||
| RSS separately. Use simulated time for expiry correctness and repeated | ||
| grow/drain cycles to expose reclamation CPU, copies and allocation churn. | ||
| Preserve arrival cursors, live-edge protection and useful cache retention. | ||
| Correct the accounting explanation with the chosen implementation. | ||
|
|
||
| Coordinate with [group storage](/quest/m2/group-storage-bench.md) and | ||
| [log adoption](/quest/m2/perf/kio-channel-contention/log-adoption.md) if their | ||
| representation has changed. [Idle GC](/quest/m1/release-perf/gc-idle.md) owns | ||
| unnecessary clock/lock work, not retained queue backing. | ||
|
|
||
| Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): | ||
| baseline first, paired evidence, equivalent work and nightly coverage. A | ||
| measured no-win is valid. No API or wire change is expected; report actual impact. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # [M] QUIC receive state recovers after bursts | ||
|
|
||
| ## Goal | ||
|
|
||
| Long-lived QUIC connections do not retain unjustified reassembly metadata | ||
| from completed streams, without losing the benefit of cheap stream reuse. | ||
|
|
||
| ## Plan | ||
|
|
||
| [Stream recycling](/rs/moq-quic/src/connection/streams/state.rs) puts freed | ||
| receive state in `free_recv`. [Assembler reset](/rs/moq-quic/src/connection/assembler.rs) | ||
| preserves its deque allocation and clears entries. Payload release therefore | ||
| does not release historical chunk capacity. Relay connections that once | ||
| received many fragmented streams are the consumer; this is separate from | ||
| unread payload credit, which receive leases ([moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878)) bound. | ||
|
|
||
| Sweep connections × peak concurrent streams × fragments, followed by stream | ||
| completion and an idle phase. Include ordered traffic, gaps/reordering, | ||
| cancellation, and another burst. Measure live allocated metadata, recycled | ||
| state counts/capacities, allocation rate, CPU and latency when traffic resumes. | ||
| Distinguish application-owned allocations from allocator-resident pages. | ||
| Any reclamation must preserve stream credit and avoid repeated allocation | ||
| thrashing. Build on the receive leases from [moq-dev/moq#5878](https://github.com/moq-dev/moq/pull/5878). | ||
|
|
||
| Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): | ||
| baseline first, paired evidence, equivalent work and nightly coverage. A | ||
| measured no-win is valid. No API or wire change is expected; report actual impact. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # [M] Resume identity retains only necessary state | ||
|
|
||
| ## Goal | ||
|
|
||
| Long-lived browser fronts preserve track identity across routes without | ||
| unnecessarily retaining historical source object graphs. | ||
|
|
||
| ## Plan | ||
|
|
||
| [Front metadata](/js/net/src/resume.ts) records the first `{info, source}` for | ||
| each track. A `Source` owns broadcast handles and a wire view; releasing its | ||
| last hold closes the cloned consumer but leaves those strong references. | ||
| This establishes a retained JS path, not that a closed socket or its native | ||
| buffers remain live. A fixed track set pins first-answer identities, not a | ||
| new source for every reconnect. Track-name churn plus route turnover is the | ||
| important adverse case. | ||
|
|
||
| Sweep fronts × distinct first-seen track names, independently varying route | ||
| replacements. Compare one stable track across many routes against new names | ||
| first answered by each route. Close readers after lookup, fetch and subscribe; | ||
| after cache expiry and browser GC, inspect heap dominators, retained source/ | ||
| subscriber/transport objects and payload bytes. Measure transition CPU too. | ||
|
|
||
| Keep the metadata proof: the same name must still mean compatible content. | ||
| Preserve incompatibility refusal, concurrent cancellation and current-source | ||
| reuse. Coordinate with [route health](/quest/m2/route-health/README.md) before | ||
| changing identity representation; do not delete history just to reduce heap. | ||
|
|
||
| Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): | ||
| baseline first, paired evidence, equivalent work and nightly coverage. A | ||
| measured no-win is valid. No API or wire change is expected; report actual impact. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # [L] Attribute UDP queue loss before tuning buffers | ||
|
|
||
| ## Goal | ||
|
|
||
| Relay socket-buffer settings follow measured loss, queue delay and memory | ||
| cost across useful load, with local starvation separated from packet drops. | ||
|
|
||
| ## Plan | ||
|
|
||
| [Socket binding](/rs/moq-sock/src/bind.rs) requests 8 MiB receive and send | ||
| buffers and already reads back OS clamping. These are ceilings, not eager | ||
| allocations. The io_uring `rx_enobufs` metric counts provided-buffer | ||
| starvation, not confirmed packet loss. Neither UDP receive stack currently | ||
| parses `SO_RXQ_OVFL` for socket-drop attribution. | ||
|
|
||
| Extend benchmark instrumentation to record granted queue sizes and actual | ||
| local drops. Linux [socket documentation](https://man7.org/linux/man-pages/man7/socket.7.html) | ||
| describes `SO_RXQ_OVFL`; account for ancillary space, counter rollover, GRO, | ||
| and the fact that missing socket-drop evidence cannot locate every loss. | ||
| Keep instrumentation private unless a production consumer needs it. | ||
|
|
||
| Sweep workers × offered flows and publishers × subscribers, with kernel | ||
| queue size and userspace pool depth varied independently. Include CPU | ||
| contention, bursts and rate-limited deep finite queues with overflow drops. | ||
| Report useful goodput, latency, CPU, memory and local versus path loss. Check | ||
| both shared and io_uring receive paths. Retain fixed defaults if measurements | ||
| do not justify a new configuration surface; larger queues can worsen delay. | ||
|
|
||
| Coordinate with the UDP pool memory quest planned in | ||
| [moq-dev/moq#6350](https://github.com/moq-dev/moq/pull/6350) and | ||
| [runtime selection](/quest/m2/perf/shard-default.md), sharing fixtures and | ||
| reports without making independent experiments block each other. | ||
|
|
||
| Follow the [survey benchmark contract](/quest/m2/performance-survey/README.md): | ||
| baseline first, paired evidence, equivalent work and nightly coverage. A | ||
| measured no-win is valid. Report actual API, configuration and wire impact. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At the reviewed commit,
rs/moq-tokio/src/quic.rs:119definesDEFAULT_RECEIVE_WINDOWas128 << 20, not 64 MiB. Because this paragraph establishes the effective baseline for subsequent settings sweeps, the incorrect value could cause experiments to compare against or explicitly configure the wrong baseline; update it to 128 MiB.AGENTS.md reference: AGENTS.md:L32-L32
Useful? React with 👍 / 👎.