diff --git a/quest/m2/perf/kio-channel-contention/README.md b/quest/m2/perf/kio-channel-contention/README.md index 745b5a5ed75..fad845acb5e 100644 --- a/quest/m2/perf/kio-channel-contention/README.md +++ b/quest/m2/perf/kio-channel-contention/README.md @@ -139,8 +139,7 @@ Decided (maintainer, 2026-10-14, per-thread follow-up): the thread-per-core workers (`--runtime-workers`, the default once [Shard by default](/quest/m2/perf/shard-default.md) lands) and link workers over an in-memory transport (`moq_net::transport::memory`), - never real QUIC. The - [replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md) + never real QUIC. The replica spike (results below) replicates the data plane and shares announcements; the [cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) shares nothing. A measured win reopens sharding through `/quest-plan`; a @@ -207,10 +206,12 @@ median): deque 30, runs 30, log 46, ring 42. Draining it: 14, 15, 12, 19. growing stops every reader or needs epochs. Its per-slot locks cost 40% more for a burst on one thread and lose to the log across cores. It wins one-frame groups only because its reader holds no channel handle. -- Not run: `just bench-profile lock` on a multi-frame shape +- Not run on this host: `just bench-profile lock` on a multi-frame shape (`bench/workloads/video-heavy.toml` is one), which needs - `kernel.perf_event_paranoid=-1` and this host has 4. So a relay's share of - `GroupState` wait with multi-frame groups is still unmeasured. + `kernel.perf_event_paranoid=-1` and this host has 4. The replica spike + below captured video-heavy's total lock wait but doesn't split it by + channel, so a relay's share of `GroupState` wait with multi-frame groups + is still unmeasured. Decided (maintainer, 2026-10-24): Front eviction lands on the lock-free append-only log (`kio::log`) itself, a tail run from @@ -284,8 +285,90 @@ at the same throughput. Main at 16 workers was refused twice (perf dropped events); the PR alone measured 4,747 ms (61%) at 16 workers at load 17 to 19, led by `Cursor::sync` (32%) and `probe` (17%), with no `poll_finished` wait. +Replica spike (2026-11-01, +[moq-dev/moq#7533](https://github.com/moq-dev/moq/pull/7533)): the code is +`Cluster::spike_replica` in `rs/moq-relay/src/cluster.rs` at +[`0636821`](https://github.com/moq-dev/moq/blob/0636821358e42363ba9b2a4f05e07f0f85ccb1a8/rs/moq-relay/src/cluster.rs), +switched on by `SPIKE_REPLICA=1`, and did not merge. Each tokio worker +builds a second origin and links the shared one into it with one moq-lite-07 +session over `moq_net::transport::memory` (default 1 MiB stream buffer), +both halves on that worker. Publishers still publish into the shared origin; +the worker's subscribers read its replica. It departs from the plan twice: + +- Announcements cross the hop too, so each replica keeps its own announce + table. These shapes announce 1 to 256 broadcasts, so it doesn't show. +- Every subscription takes the hop, even on the publisher's own worker. So + the 1-worker rows are the hop's whole cost, and the 4-worker rows overstate + it by up to a quarter. + +Method: `just bench-profile lock`, tokio workers over QUIC, with `perf stat` +counting the relay's instructions over the unprofiled run's 5 s steady +window. The relay was pinned to the 16 E-cores and the generator to the 8 +P-cores. The harness pins everything to the P-cores by default, so these +numbers don't compare with the baseline above. Load average 5 to 21. Each +cell is the median of 3 runs (chat at 16 workers: 9 base, 5 replica; +video-heavy: 2). Mutex wait is the median of the runs whose lock capture +the harness accepted, which is fewer where noted below. Instructions are +user and kernel per delivered frame, in thousands; CPU and RSS come from the host sampler; latency is the +generator's p50 in whole milliseconds. Each cell reads base → replica: + +| shape | workers | CPU cores | user instr | kernel instr | mutex wait (share) | RSS MiB | p50 ms | +| --- | --- | --- | --- | --- | --- | --- | --- | +| fanout | 1 | 0.06 → 0.05 | 167 → 170 | 20 → 21 | 0 → 0 | 36 → 36 | 3 → 4 | +| fanout | 4 | 0.15 → 0.06 | 170 → 177 | 52 → 30 | 39 ms (12%) → 9 ms (3.1%) | 37 → 38 | 3 → 3 | +| fanout | 16 | 0.45 → 0.16 | 182 → 211 | 344 → 123 | 807 ms (52%) → 256 ms (40%) | 43 → 47 | 3 → 2 | +| chat | 1 | 1.00 → 1.00 | 185 → 184 | 18 → 18 | 0 → 0 | 576 → 577 | 966 → 1,273 | +| chat | 4 | 1.91 → 1.80 | 168 → 171 | 40 → 20 | 50 ms (0.5%) → 18 ms (0.2%) | 499 → 507 | 31 → 68 | +| chat | 16 | 9.5 → 2.2 | 180 → 170 | 427 → 28 | not captured → 793 ms (7.9%) | 500 → 518 | 34 → 18 | +| video-heavy | 1 | 0.97 → 1.00 | 112 → 125 | 25 → 16 | 0 → 0 | 320 → 357 | 2.5 → 227 | +| video-heavy | 4 | 1.22 → 1.87 | 116 → 207 | 32 → 31 | 27 ms (0.4%) → 16 ms (0.2%) | 324 → 381 | 1 → 1 | +| video-heavy | 16 | 1.84 → 2.59 | 123 → 213 | 51 → 50 | 137 ms (1.5%) → 208 ms (1.6%) | 330 → 424 | 0 → 0 | + +`bench/workloads/video-heavy.toml` is the low-fan-out shape: 256 +connections each publish one 60 fps track of 60-frame groups and subscribe +to two others. Chat at 1 worker saturates the worker either way (about +15,000 of 20,000 messages/s delivered), so its latency is queueing. +Video-heavy at 1 worker saturates only with the replica: 28,400 instead of +31,400 frames/s. The harness refused all nine lock captures of base chat at +16 workers, at least seven of them for a lost perf record (2, 8, and 32 MiB +buffers), so its wait is missing. Its 400,000 +kernel instructions per frame are the contention: futex waits and wakes +around one track with 2,000 readers. Base video-heavy at 16 workers is one +capture; the other was refused for nonalternating switch records. + +- High fan-out wins at 16 workers. Chat drops from 9.5 to 2.2 cores, kernel + instructions per frame from 427k to 28k, and p50 from 34 to 18 ms. + Fanout drops from 0.45 to 0.16 cores while user instructions rise 16%, + the hop's own session work. At 4 workers the gain is small. +- What stays contended with replicas is mostly the shared track, whose + readers are now the hops, one per worker. In chat at 16 workers (four + captures), `TrackState` is 86% of the wait: `Subscriber::poll_drop` 28%, + `Subscriber::poll_finished` via `resume::Reader::learn_end` 26%, + `GroupExpiry::is_expired` 10%, and the rest 21%. Another 13% is the + per-track cache mutex in `cache::Track::watch_wall`, also under + `GroupExpiry::is_expired`. The version counter targets the `TrackState` + reads, and only the ones that find nothing changed: a read of new data + still locks, and the cache mutex isn't a channel, so it stays. +- Low fan-out loses. Video-heavy pays the hop on nearly every subscription. + At 4 and 16 workers that is 73 to 78% more user instructions per frame, + 41 to 53% more CPU, and 18 to 28% more memory, with no lock-wait gain at + 16 workers. At 1 worker the hop saturates the worker instead: 12% more + user instructions and 12% more memory, CPU pinned at one core, fewer + frames delivered, and p50 from 2.5 to 227 ms. +- Memory: fanout and chat grow 0 to 11%. Replicated one-frame groups are + small, and multi-frame groups cost the most. + +Recommendation: a conditional win. Replicas pay off only on high-fan-out +tracks. Applied to every track, they cost low fan-out about 1.8 times the +user instructions per frame at 4 and 16 workers. If sharding reopens, +replicate per track on demand: never on the publisher's own worker, and only above a per-worker fan-out +threshold, serving the rest from the shared origin. Rerun this comparison +with the version counter (landed above), since it attacks the same 16-worker chat `TrackState` waits without +the hop. Weigh it against the +[Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) +before `/quest-plan`. + ## Required -- [Replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md) - measures per-worker data-plane replicas linked over the in-memory transport, announcements shared - [Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) - measures a relay per thread clustered over the in-memory transport, nothing shared - [Log adoption](/quest/m2/perf/kio-channel-contention/log-adoption.md) - the rest of moq-net's log-shaped state moves onto the log where it measures as a win diff --git a/quest/m2/perf/kio-channel-contention/log-adoption.md b/quest/m2/perf/kio-channel-contention/log-adoption.md index afe533d180c..0f2e656724f 100644 --- a/quest/m2/perf/kio-channel-contention/log-adoption.md +++ b/quest/m2/perf/kio-channel-contention/log-adoption.md @@ -18,7 +18,11 @@ stays on its mutex. - Waits on the spikes (decided 2026-10-14 for the log, kept here): with per-thread sessions every consumer of a replica is on one thread, so its mutex is uncontended and a lock-free multi-reader log only pays off if the - shared model survives. + shared model survives. The replica spike (2026-11-01, in the + [epic](/quest/m2/perf/kio-channel-contention/README.md)) does not settle + it: replicas lose on low fan-out, so shared tracks would stay, and the + shared track it keeps is still the most contended channel, with one reader + per worker. - Audit moq-net (and moq-relay's `critical.rs`) for other state that is appended at the back, read by cursor, and evicted from the front, beyond `GroupState` frames, which the log quest already moves. List the @@ -29,5 +33,4 @@ stays on its mutex. ## Required -- [Replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md) - if per-worker replicas win, each channel's cross-core readers collapse to one flow-controlled stream per worker and adoption is likely abandoned -- [Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) - the same question with nothing shared +- [Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) - if a relay per thread wins, each channel's cross-core readers collapse to one flow-controlled stream per worker and adoption is likely abandoned diff --git a/quest/m2/perf/kio-channel-contention/spike-cluster.md b/quest/m2/perf/kio-channel-contention/spike-cluster.md index 6235cc431dc..384c500c372 100644 --- a/quest/m2/perf/kio-channel-contention/spike-cluster.md +++ b/quest/m2/perf/kio-channel-contention/spike-cluster.md @@ -5,9 +5,9 @@ Measure a relay per worker thread, each with its own `Cluster` and origin and nothing shared, clustered over the in-memory transport (`moq_net::transport::memory`) instead -of QUIC. Compare it with the -[replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md), which -shares the control plane. The spike is a draft PR that never merges; its +of QUIC. Compare it with the replica spike, which shares the control plane; +its numbers are in the +[epic](/quest/m2/perf/kio-channel-contention/README.md). The spike is a draft PR that never merges; its numbers and a recommendation go in the [epic](/quest/m2/perf/kio-channel-contention/README.md). diff --git a/quest/m2/perf/kio-channel-contention/spike-replica.md b/quest/m2/perf/kio-channel-contention/spike-replica.md deleted file mode 100644 index 2bd5933c621..00000000000 --- a/quest/m2/perf/kio-channel-contention/spike-replica.md +++ /dev/null @@ -1,39 +0,0 @@ -# [M] Spike: per-worker replicas - -## Goal - -Measure whether per-worker data-plane replicas cut lock wait on the -thread-per-core relay (`--runtime-workers`). Each worker serves tracks and -groups from its own origin, fed once per worker over the -in-memory transport (`moq_net::transport::memory`), while -announcements and routes stay in one shared origin. The spike is a draft PR -that never merges; its numbers and a recommendation go in the -[epic](/quest/m2/perf/kio-channel-contention/README.md). - -## Plan - -- Today every worker's sessions share one `origin::Producer`, built in - `Cluster::new` (`rs/moq-relay/src/cluster.rs`) and cloned per worker - (`rs/moq-relay/src/relay.rs`), so a subscriber on worker B reads state - written on worker A. -- With replicas, a track subscribed on worker B is subscribed once over a - memory session to the worker that holds it, then fanned out locally. Every - consumer of a replica is on one thread, so its kio mutex is uncontended, and - cross-core readers per frame drop from the subscriber count to the worker - count. -- Control plane shared, data plane replicated: the shared origin still - carries announcements, so announce churn is not paid once per worker. -- Flow control throttles only the hop (decided 2026-10-14): a slow worker's - stream lags and its stale groups reset; the publisher and other workers - never wait. -- Measure against the [epic](/quest/m2/perf/kio-channel-contention/README.md)'s - baseline: `just bench-profile lock` fanout and chat at 1, 4, and 16 workers, plus - relay CPU, resident memory (replicated group state), and glass-to-glass - latency added by the hop. Lower lock wait alone is not a win; weigh it - against the CPU, memory, and latency. Chat (`bench/workloads/chat.toml`) - is one broadcast to 2,000 viewers with one-frame groups, so it stresses - per-group track writes rather than frame fan-out. Replicas give little to - low-fanout load; if that matters, name a separate low-fanout workload - instead of reading it from chat. -- A win goes to `/quest-plan` for a real implementation quest; a no-win - restores the perf epic's no-sharding rule.