Skip to content
95 changes: 89 additions & 6 deletions quest/m2/perf/kio-channel-contention/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,7 @@ Decided (maintainer, 2026-10-14, per-thread follow-up):
the thread-per-core workers (`--runtime-workers`, the default once
[Shard by default](/quest/m2/perf/shard-default.md) lands) and link workers over an
in-memory transport (`moq_net::transport::memory`),
never real QUIC. The
[replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md)
never real QUIC. The replica spike (results below)
replicates the data plane and shares announcements; the
[cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md)
shares nothing. A measured win reopens sharding through `/quest-plan`; a
Expand Down Expand Up @@ -207,10 +206,12 @@ median): deque 30, runs 30, log 46, ring 42. Draining it: 14, 15, 12, 19.
growing stops every reader or needs epochs. Its per-slot locks cost 40%
more for a burst on one thread and lose to the log across cores. It wins
one-frame groups only because its reader holds no channel handle.
- Not run: `just bench-profile lock` on a multi-frame shape
- Not run on this host: `just bench-profile lock` on a multi-frame shape
(`bench/workloads/video-heavy.toml` is one), which needs
`kernel.perf_event_paranoid=-1` and this host has 4. So a relay's share of
`GroupState` wait with multi-frame groups is still unmeasured.
`kernel.perf_event_paranoid=-1` and this host has 4. The replica spike
below captured video-heavy's total lock wait but doesn't split it by
channel, so a relay's share of `GroupState` wait with multi-frame groups
is still unmeasured.

Decided (maintainer, 2026-10-24): Front eviction lands on the lock-free
append-only log (`kio::log`) itself, a tail run from
Expand Down Expand Up @@ -284,8 +285,90 @@ at the same throughput. Main at 16 workers was refused twice (perf dropped
events); the PR alone measured 4,747 ms (61%) at 16 workers at load 17 to 19,
led by `Cursor::sync` (32%) and `probe` (17%), with no `poll_finished` wait.

Replica spike (2026-11-01,
[moq-dev/moq#7533](https://github.com/moq-dev/moq/pull/7533)): the code is
`Cluster::spike_replica` in `rs/moq-relay/src/cluster.rs` at
[`0636821`](https://github.com/moq-dev/moq/blob/0636821358e42363ba9b2a4f05e07f0f85ccb1a8/rs/moq-relay/src/cluster.rs),
switched on by `SPIKE_REPLICA=1`, and did not merge. Each tokio worker
builds a second origin and links the shared one into it with one moq-lite-07
session over `moq_net::transport::memory` (default 1 MiB stream buffer),
both halves on that worker. Publishers still publish into the shared origin;
the worker's subscribers read its replica. It departs from the plan twice:

- Announcements cross the hop too, so each replica keeps its own announce
table. These shapes announce 1 to 256 broadcasts, so it doesn't show.
- Every subscription takes the hop, even on the publisher's own worker. So
the 1-worker rows are the hop's whole cost, and the 4-worker rows overstate
it by up to a quarter.

Method: `just bench-profile lock`, tokio workers over QUIC, with `perf stat`
counting the relay's instructions over the unprofiled run's 5 s steady
window. The relay was pinned to the 16 E-cores and the generator to the 8
P-cores. The harness pins everything to the P-cores by default, so these
numbers don't compare with the baseline above. Load average 5 to 21. Each
cell is the median of 3 runs (chat at 16 workers: 9 base, 5 replica;
video-heavy: 2). Mutex wait is the median of the runs whose lock capture
the harness accepted, which is fewer where noted below. Instructions are
user and kernel per delivered frame, in thousands; CPU and RSS come from the host sampler; latency is the
generator's p50 in whole milliseconds. Each cell reads base → replica:

| shape | workers | CPU cores | user instr | kernel instr | mutex wait (share) | RSS MiB | p50 ms |
| --- | --- | --- | --- | --- | --- | --- | --- |
| fanout | 1 | 0.06 → 0.05 | 167 → 170 | 20 → 21 | 0 → 0 | 36 → 36 | 3 → 4 |
| fanout | 4 | 0.15 → 0.06 | 170 → 177 | 52 → 30 | 39 ms (12%) → 9 ms (3.1%) | 37 → 38 | 3 → 3 |
| fanout | 16 | 0.45 → 0.16 | 182 → 211 | 344 → 123 | 807 ms (52%) → 256 ms (40%) | 43 → 47 | 3 → 2 |
| chat | 1 | 1.00 → 1.00 | 185 → 184 | 18 → 18 | 0 → 0 | 576 → 577 | 966 → 1,273 |
| chat | 4 | 1.91 → 1.80 | 168 → 171 | 40 → 20 | 50 ms (0.5%) → 18 ms (0.2%) | 499 → 507 | 31 → 68 |
| chat | 16 | 9.5 → 2.2 | 180 → 170 | 427 → 28 | not captured → 793 ms (7.9%) | 500 → 518 | 34 → 18 |
| video-heavy | 1 | 0.97 → 1.00 | 112 → 125 | 25 → 16 | 0 → 0 | 320 → 357 | 2.5 → 227 |
| video-heavy | 4 | 1.22 → 1.87 | 116 → 207 | 32 → 31 | 27 ms (0.4%) → 16 ms (0.2%) | 324 → 381 | 1 → 1 |
| video-heavy | 16 | 1.84 → 2.59 | 123 → 213 | 51 → 50 | 137 ms (1.5%) → 208 ms (1.6%) | 330 → 424 | 0 → 0 |

`bench/workloads/video-heavy.toml` is the low-fan-out shape: 256
connections each publish one 60 fps track of 60-frame groups and subscribe
to two others. Chat at 1 worker saturates the worker either way (about
15,000 of 20,000 messages/s delivered), so its latency is queueing.
Video-heavy at 1 worker saturates only with the replica: 28,400 instead of
31,400 frames/s. The harness refused all nine lock captures of base chat at
16 workers, at least seven of them for a lost perf record (2, 8, and 32 MiB
buffers), so its wait is missing. Its 400,000
kernel instructions per frame are the contention: futex waits and wakes
around one track with 2,000 readers. Base video-heavy at 16 workers is one
capture; the other was refused for nonalternating switch records.

- High fan-out wins at 16 workers. Chat drops from 9.5 to 2.2 cores, kernel
instructions per frame from 427k to 28k, and p50 from 34 to 18 ms.
Fanout drops from 0.45 to 0.16 cores while user instructions rise 16%,
the hop's own session work. At 4 workers the gain is small.
- What stays contended with replicas is mostly the shared track, whose
readers are now the hops, one per worker. In chat at 16 workers (four
captures), `TrackState` is 86% of the wait: `Subscriber::poll_drop` 28%,
`Subscriber::poll_finished` via `resume::Reader::learn_end` 26%,
`GroupExpiry::is_expired` 10%, and the rest 21%. Another 13% is the
per-track cache mutex in `cache::Track::watch_wall`, also under
`GroupExpiry::is_expired`. The version counter targets the `TrackState`
reads, and only the ones that find nothing changed: a read of new data
still locks, and the cache mutex isn't a channel, so it stays.
- Low fan-out loses. Video-heavy pays the hop on nearly every subscription.
At 4 and 16 workers that is 73 to 78% more user instructions per frame,
41 to 53% more CPU, and 18 to 28% more memory, with no lock-wait gain at
16 workers. At 1 worker the hop saturates the worker instead: 12% more
user instructions and 12% more memory, CPU pinned at one core, fewer
frames delivered, and p50 from 2.5 to 227 ms.
- Memory: fanout and chat grow 0 to 11%. Replicated one-frame groups are
small, and multi-frame groups cost the most.

Recommendation: a conditional win. Replicas pay off only on high-fan-out
tracks. Applied to every track, they cost low fan-out about 1.8 times the
user instructions per frame at 4 and 16 workers. If sharding reopens,
replicate per track on demand: never on the publisher's own worker, and only above a per-worker fan-out
threshold, serving the rest from the shared origin. Rerun this comparison
with the version counter (landed above), since it attacks the same 16-worker chat `TrackState` waits without
the hop. Weigh it against the
[Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md)
before `/quest-plan`.

## Required

- [Replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md) - measures per-worker data-plane replicas linked over the in-memory transport, announcements shared
- [Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) - measures a relay per thread clustered over the in-memory transport, nothing shared
- [Log adoption](/quest/m2/perf/kio-channel-contention/log-adoption.md) - the rest of moq-net's log-shaped state moves onto the log where it measures as a win
9 changes: 6 additions & 3 deletions quest/m2/perf/kio-channel-contention/log-adoption.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,11 @@ stays on its mutex.
- Waits on the spikes (decided 2026-10-14 for the log, kept here): with
per-thread sessions every consumer of a replica is on one thread, so its
mutex is uncontended and a lock-free multi-reader log only pays off if the
shared model survives.
shared model survives. The replica spike (2026-11-01, in the
[epic](/quest/m2/perf/kio-channel-contention/README.md)) does not settle
it: replicas lose on low fan-out, so shared tracks would stay, and the
shared track it keeps is still the most contended channel, with one reader
per worker.
- Audit moq-net (and moq-relay's `critical.rs`) for other state that is
appended at the back, read by cursor, and evicted from the front, beyond
`GroupState` frames, which the log quest already moves. List the
Expand All @@ -29,5 +33,4 @@ stays on its mutex.

## Required

- [Replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md) - if per-worker replicas win, each channel's cross-core readers collapse to one flow-controlled stream per worker and adoption is likely abandoned
- [Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) - the same question with nothing shared
- [Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) - if a relay per thread wins, each channel's cross-core readers collapse to one flow-controlled stream per worker and adoption is likely abandoned
6 changes: 3 additions & 3 deletions quest/m2/perf/kio-channel-contention/spike-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
Measure a relay per worker thread, each with its own `Cluster` and origin and
nothing shared, clustered over the
in-memory transport (`moq_net::transport::memory`) instead
of QUIC. Compare it with the
[replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md), which
shares the control plane. The spike is a draft PR that never merges; its
of QUIC. Compare it with the replica spike, which shares the control plane;
its numbers are in the
[epic](/quest/m2/perf/kio-channel-contention/README.md). The spike is a draft PR that never merges; its
numbers and a recommendation go in the
[epic](/quest/m2/perf/kio-channel-contention/README.md).

Expand Down
39 changes: 0 additions & 39 deletions quest/m2/perf/kio-channel-contention/spike-replica.md

This file was deleted.