From 19d13f93e2ba52518c427ffe19a2329b5bae7acc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 01:24:57 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m2/perf/quic-borrow-free.md Quest-Claim: acb9ab45fc80b599 From 8c4e1e6cee932afd464a3f6cea56950cb658c50d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 01:55:27 -0800 Subject: [PATCH 2/3] quest: record the borrow-free mechanism measurements The prototypes live on proto/quic-borrow-free. On a pinned core a Mutex costs 1.1-3.5% over context passing, a RefCell 0.2-1.0%, and the driver-owned channel 5-29%. The driver-level prototypes and the fanout sweep remain, pending whether they are still worth building. Co-Authored-By: Claude Opus 5.5 --- quest/m2/perf/quic-borrow-free.md | 33 +++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/quest/m2/perf/quic-borrow-free.md b/quest/m2/perf/quic-borrow-free.md index 865c946d9c0..5f86cc99446 100644 --- a/quest/m2/perf/quic-borrow-free.md +++ b/quest/m2/perf/quic-borrow-free.md @@ -49,6 +49,39 @@ moq-net's `transport::poll` traits. Only context passing would revisit their Public API: none; prototypes stay on a branch. The adoption is its own quest. +### Mechanism prototypes (2026-11-01) + +[moq-dev/moq#7531](https://github.com/moq-dev/moq/pull/7531) has the full +report. The prototypes are on the `proto/quic-borrow-free` branch +([c368f72](https://github.com/moq-dev/moq/commit/c368f72c0b)), not to land: +`rs/moq-quic/benches/share.rs` runs a relay that writes new uni streams to N +subscribers over real moq-quic connections (plaintext, virtual time). Every +shape makes the same moq-quic calls through the [waits](/quest/m1/quic-api/waits.md) +`poll_*` forms. The four shapes are context passing, `RefCell`, `Mutex`, and a +driver-owned connection. In the driver-owned shape, each send stream is a +`kio::log` the handle appends to against published credit, and each received +lease goes through a cell to the reader and comes back for release. + +Instructions per stream (`instructions:u`, pinned core, runs differ by under +0.6%): + +- **`RefCell`** costs 0.2–1.0% more than context passing, and **`Mutex`** + costs 1.1–3.5% more (about 290 instructions per stream, plus about 100 per + extra write). +- **Channel** costs 5–29% more: about 4.2–4.8k instructions per stream plus + about 320 per extra write. The cost is highest for one write per stream, + which is fanout's shape. About 1.3k of the extra comes from `kio::log`'s + epoch reclamation, about 1.1k from allocations (the cells, log segments, + tasks, and lease boxes), and the rest from credit, the cell stacks, and the + extra wake hop. +- **Threaded** (noisy host): with the relay's writes spread over 1–7 threads, + `Mutex` had the lower median wall time in 13 of 16 cases. Channel was ahead + only when two or four writers shared a single connection. + +What remains: the driver-level prototypes in moq-tokio and moq-uring, the +fanout-bench sweep, and the recommendation. Whether they are still worth +building is open on the PR. + ## Related - [QUIC lock contention](/quest/m2/perf/quic-lock-contention.md) - measures the lock this would remove; its staging queue is a partial form of the channel approach From 303d25daed61ef545a46e12e2444155521e811dc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 14:20:20 -0800 Subject: [PATCH 3/3] quest: finish borrow-free QUIC state with the mechanism report The maintainer chose to stop at the mechanism report (2026-11-01). Delete the quest; the report stays in the PR body, and adoption is planned separately. The lock-contention quest now cites the measured costs. Co-Authored-By: Claude Opus 5.5 --- quest/m2/perf/README.md | 1 - quest/m2/perf/quic-borrow-free.md | 88 --------------------------- quest/m2/perf/quic-lock-contention.md | 10 ++- 3 files changed, 7 insertions(+), 92 deletions(-) delete mode 100644 quest/m2/perf/quic-borrow-free.md diff --git a/quest/m2/perf/README.md b/quest/m2/perf/README.md index 8e3bf71344f..7c56483147a 100644 --- a/quest/m2/perf/README.md +++ b/quest/m2/perf/README.md @@ -87,7 +87,6 @@ revisions on the same host and worker count, and profiles with - [Stream transports](/quest/m2/perf/stream/README.md) - qmux drives its own WebSocket and TCP sockets in one task, measured end to end, then serves them from the io_uring workers - [Relay allocations](/quest/m2/perf/relay-allocs.md) - fewer allocations per delivered frame, starting with per-stream session clones - [QUIC lock contention](/quest/m2/perf/quic-lock-contention.md) - a measured verdict on the tokio driver's connection-wide mutex and per-packet endpoint-to-connection hop, with levers built only on evidence -- [Borrow-free QUIC state](/quest/m2/perf/quic-borrow-free.md) - a prototyped report on sharing connection state with stream handles without `RefCell` or `Mutex`: context passing vs driver-owned state with per-stream channels - [Run to quiescence](/quest/m2/perf/uring-quiescence.md) - a received packet's reply is staged in the same turn, under a pass and train budget that keeps the fairness rule - [Ingest batch](/quest/m2/perf/ingest-batch.md) - relay ingest pays one lock, wake, and clock read per burst of whole frames instead of per frame - [Streaming frame read owners](/quest/m2/perf/frame-read-owners.md) - readers of a frame still arriving share one byte owner per publication instead of allocating one each per read diff --git a/quest/m2/perf/quic-borrow-free.md b/quest/m2/perf/quic-borrow-free.md deleted file mode 100644 index 5f86cc99446..00000000000 --- a/quest/m2/perf/quic-borrow-free.md +++ /dev/null @@ -1,88 +0,0 @@ -# [M] QUIC connection state without runtime borrow checks - -## Goal - -A design report, with prototypes, on how a QUIC connection and its stream -handles can share state without a runtime borrow check (`RefCell`, `Mutex`) -on the data path, and a recommendation to adopt one or keep the lock. Each -prototype covers one stream's write and read path in moq-quic and moq-tokio's -driver, enough to run the fanout bench. - -## Plan - -Decided with the maintainer on 2026-10-25: `Rc>` and -`Arc>` are API warts, a sign the borrow checker could not do the job -so it happens at runtime. Today moq-quic's `Connection` is mutated by the -driver (packets, timers, transmit) and by stream handles in other tasks, so -moq-tokio locks it and moq-uring borrows it from a `RefCell` per core. - -Compare: - -- **Context passing.** Stream handles are owned, non-`Clone` tokens, and every - operation takes `&mut Connection` (`conn.write(&mut stream, buf)`). One owner - task holds the connection and passes it into its children's polls - (`kio::Tasks`), like Zig's explicit `std.Io`. Cost to measure: moq-net's - per-subscription tasks become children of the session task, and - moq-net's `transport::poll` `&self` shape flips to `&mut` - plus context. -- **Driver-owned state, handles as channel ends.** The driver alone owns - moq-quic. A send stream is the writer of a per-stream lock-free log (the - handle appends; the driver packetizes and retransmits by offset; acks - evict), credit is an atomic the driver publishes, received data arrives - through a per-stream queue, and the waits from - [waits](/quest/m1/quic-api/waits.md) live in that per-stream cell. Keeps - the `&self` API and fits both runtimes; msquic's one owner per connection is - prior art. Costs: an `Arc` per stream and credit mirrored on both sides. - `kio::log` evicts only from its writer, so acked eviction needs a path - back from the driver. -- **Today's lock** as the baseline. - -Rejected: branded cells (`qcell`, GhostCell). Access still needs a `&mut` -owner token, so they reduce to context passing with shared pointers, and a -lifetime brand fights `'static` tasks. - -Measure each on [QUIC lock contention](/quest/m2/perf/quic-lock-contention.md)'s -sweep (connections × streams per connection × workers), on tokio and -moq-uring, and report the API each implies for moq-quic, the drivers, and -moq-net's `transport::poll` traits. Only context passing would revisit their -`&self` shape, and the report says so if it wins. - -Public API: none; prototypes stay on a branch. The adoption is its own quest. - -### Mechanism prototypes (2026-11-01) - -[moq-dev/moq#7531](https://github.com/moq-dev/moq/pull/7531) has the full -report. The prototypes are on the `proto/quic-borrow-free` branch -([c368f72](https://github.com/moq-dev/moq/commit/c368f72c0b)), not to land: -`rs/moq-quic/benches/share.rs` runs a relay that writes new uni streams to N -subscribers over real moq-quic connections (plaintext, virtual time). Every -shape makes the same moq-quic calls through the [waits](/quest/m1/quic-api/waits.md) -`poll_*` forms. The four shapes are context passing, `RefCell`, `Mutex`, and a -driver-owned connection. In the driver-owned shape, each send stream is a -`kio::log` the handle appends to against published credit, and each received -lease goes through a cell to the reader and comes back for release. - -Instructions per stream (`instructions:u`, pinned core, runs differ by under -0.6%): - -- **`RefCell`** costs 0.2–1.0% more than context passing, and **`Mutex`** - costs 1.1–3.5% more (about 290 instructions per stream, plus about 100 per - extra write). -- **Channel** costs 5–29% more: about 4.2–4.8k instructions per stream plus - about 320 per extra write. The cost is highest for one write per stream, - which is fanout's shape. About 1.3k of the extra comes from `kio::log`'s - epoch reclamation, about 1.1k from allocations (the cells, log segments, - tasks, and lease boxes), and the rest from credit, the cell stacks, and the - extra wake hop. -- **Threaded** (noisy host): with the relay's writes spread over 1–7 threads, - `Mutex` had the lower median wall time in 13 of 16 cases. Channel was ahead - only when two or four writers shared a single connection. - -What remains: the driver-level prototypes in moq-tokio and moq-uring, the -fanout-bench sweep, and the recommendation. Whether they are still worth -building is open on the PR. - -## Related - -- [QUIC lock contention](/quest/m2/perf/quic-lock-contention.md) - measures the lock this would remove; its staging queue is a partial form of the channel approach -- [moq-quic owns its waits](/quest/m1/quic-api/waits.md) - the wait shape every prototype keeps diff --git a/quest/m2/perf/quic-lock-contention.md b/quest/m2/perf/quic-lock-contention.md index 0b1a3aeaa3c..789d2e7a005 100644 --- a/quest/m2/perf/quic-lock-contention.md +++ b/quest/m2/perf/quic-lock-contention.md @@ -48,8 +48,13 @@ Levers, only where the profile shows a cost: and only on evidence. The maintainer wants the connection state free of runtime borrow checks -altogether (2026-10-25); [borrow-free state](/quest/m2/perf/quic-borrow-free.md) -designs that, and this profile is its baseline. +altogether (2026-10-25). The borrow-free mechanism report +([moq-dev/moq#7531](https://github.com/moq-dev/moq/pull/7531)) measured the +designs on one core: the `Mutex` costs 1.1 to 3.5% over context passing, and a +driver-owned channel (this quest's staging queue in full) costs about 4.5k +instructions per stream plus about 320 per extra write. Adopting a design, or +keeping the lock, is decided separately; this profile is the contended baseline +that decision weighs. A measured no-win is a valid outcome. Report public API and wire impact; the staging queue and batch dispatch should need none. @@ -61,6 +66,5 @@ staging queue and batch dispatch should need none. ## Related -- [Borrow-free state](/quest/m2/perf/quic-borrow-free.md) - the lock-free designs this profile is the baseline for - [Relay egress profile](/quest/m2/quic-egress-profile.md) - the transmit side of the same path - [Relay allocations](/quest/m2/perf/relay-allocs.md) - the allocation profile of the same workload