Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,12 @@ other count is that many unpinned Tokio workers. The table reports CPU per
connection, so a steering, dispatch, or per-shard cost that grows with either
axis reads as a slope. Linux only, since nothing else steers a reuseport
group. Run the sweep locally when changing relay sharding; its measurements
are not a nightly series.
are not a nightly series. `sock-drops` and `gen-drops` are the datagrams the
kernel dropped over the measured window at the relay's and the generator's
own UDP sockets because a receive queue was full, from the sampler's
`udp_drops` (see `rs/moq-bench/README.md`). Every relay summary carries them
with the granted receive buffer and the peak queued bytes. A nonzero
`sock-drops` is local starvation; loss with zero drops happened elsewhere.

Profile the relay under a workload, in one or more capture modes: `lock`
(worker std mutex waits by stack), `cpu` (CPU stacks of every relay thread),
Expand Down
15 changes: 9 additions & 6 deletions bench/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -816,18 +816,21 @@ run_shard_sweep() {
done

printf '\nShard sweep (0 shards is the shared runtime on %d threads)\n' "$cpus"
printf '%-8s %11s %7s %10s %10s %8s %7s %8s %9s %9s\n' \
workload connections shards recv-Mbps recv-fps p99-ms loss-% CPU CPU/conn RSS-MiB
# sock-drops: datagrams the kernel dropped at the relay's own UDP sockets
# (full receive queues), and gen-drops at the load generator's; see the README.
printf '%-8s %11s %7s %10s %10s %8s %7s %8s %9s %9s %10s %9s\n' \
workload connections shards recv-Mbps recv-fps p99-ms loss-% CPU CPU/conn RSS-MiB sock-drops gen-drops
for workload in video fanout; do
for connections in $SHARD_CONNECTIONS; do
for shards in $SHARD_COUNTS; do
jq -r --arg connections "$connections" --arg shards "$shards" '[
.workload, $connections, $shards, .recv_mbps, .recv_fps, .latency_p99_ms, .loss_pct,
.cpu_cores, (.cpu_cores / ($connections | tonumber)), (.rss_bytes / 1048576)
.cpu_cores, (.cpu_cores / ($connections | tonumber)), (.rss_bytes / 1048576),
(.udp_drops // "n/a"), (.generator.udp_drops // "n/a")
] | @tsv' "$RUN/relay/shard-$shards-$connections/$workload/summary.json" |
while IFS=$'\t' read -r name conns count recv fps p99 loss cpu per rss; do
printf '%-8s %11s %7s %10.3f %10.1f %8.1f %7.3f %8.3f %9.5f %9.1f\n' \
"$name" "$conns" "$count" "$recv" "$fps" "$p99" "$loss" "$cpu" "$per" "$rss"
while IFS=$'\t' read -r name conns count recv fps p99 loss cpu per rss drops gen_drops; do
printf '%-8s %11s %7s %10.3f %10.1f %8.1f %7.3f %8.3f %9.5f %9.1f %10s %9s\n' \
"$name" "$conns" "$count" "$recv" "$fps" "$p99" "$loss" "$cpu" "$per" "$rss" "$drops" "$gen_drops"
done
done
done
Expand Down
32 changes: 32 additions & 0 deletions bench/summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
from pathlib import Path

COUNTERS = ("cpu_user", "cpu_system", "ctx_voluntary", "ctx_involuntary")
# Integer counters from the sampler's UDP accounting. The namespace-wide ones
# include every socket in the network namespace, not only the process's.
UDP_COUNTERS = ("udp_drops", "udp_rcvbuf_errors", "udp_sndbuf_errors", "udp_in_errors")


def records(path):
Expand Down Expand Up @@ -89,9 +92,38 @@ def cpu(rows, start, end):
threads=max(row["threads"] for row in window),
thread_cpu=workers,
thread_scope="threads present at both interpolated window boundaries",
**udp(rows, start, end),
)


def udp(rows, start, end):
"""Kernel UDP accounting over the window, when the sampler recorded it.

Drops are whole events, so the counters diff the samples that bracket the
window instead of interpolating: a drop near a boundary counts rather than
splitting. The granted buffer and queue peak come from samples inside it.
"""
if not all("udp_drops" in row for row in rows):
return {}
first = [row for row in rows if row["timestamp_ms"] <= start][-1]
last = next(row for row in rows if row["timestamp_ms"] >= end)
result = {}
for key in UDP_COUNTERS:
value = last[key] - first[key]
if value < 0:
raise ValueError(f"host counter moved backwards: {key}")
result[key] = value
window = [row for row in rows if first["timestamp_ms"] <= row["timestamp_ms"] <= last["timestamp_ms"]]
granted = [row["udp_rcvbuf_min"] for row in window if row["udp_sockets"]]
result.update(
udp_sockets=max(row["udp_sockets"] for row in window),
udp_rcvbuf_min=min(granted) if granted else None,
udp_rx_queue_peak_bytes=max(row["udp_rx_queue_bytes"] for row in window),
udp_scope="samples bracketing the load window; rcvbuf/sndbuf/in errors are namespace-wide",
)
return result


def main():
parser = argparse.ArgumentParser(description=__doc__)
sub = parser.add_subparsers(dest="command", required=True)
Expand Down
35 changes: 35 additions & 0 deletions bench/summary.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,41 @@ def test_host_uses_load_window(self):
self.assertEqual(result["cpu_cores_per_gbps"], 2)
self.assertEqual(result["thread_cpu"], [dict(tid=8, name="moq-worker-0", cpu_cores=1)])

def test_udp_counts_whole_drops_across_the_bracketing_samples(self):
rows = [
dict(
timestamp_ms=t,
pid=1,
cpu_user=t / 1000,
cpu_system=0,
ctx_voluntary=0,
ctx_involuntary=0,
rss_bytes=1,
threads=1,
udp_sockets=2,
udp_rcvbuf_min=rcvbuf,
udp_sndbuf_min=1,
udp_rx_queue_bytes=queued,
udp_drops=drops,
udp_rcvbuf_errors=drops + 5,
udp_sndbuf_errors=0,
udp_in_errors=drops + 5,
)
for t, drops, queued, rcvbuf in ((0, 1, 0, 9), (1000, 4, 70, 8), (2000, 10, 30, 8), (3000, 11, 0, 8))
]
result = summary_module.cpu(rows, 500, 1500)
# The samples at 0 and 2000 bracket the window: 10 - 1, not an interpolated 4.5.
self.assertEqual(result["udp_drops"], 9)
self.assertEqual(result["udp_rcvbuf_errors"], 9)
self.assertEqual(result["udp_rx_queue_peak_bytes"], 70)
self.assertEqual(result["udp_rcvbuf_min"], 8)
# A sampler without UDP accounting adds nothing.
plain = [{key: value for key, value in row.items() if not key.startswith("udp_")} for row in rows]
self.assertNotIn("udp_drops", summary_module.cpu(plain, 500, 1500))
rows[2]["udp_drops"] = 0
with self.assertRaises(ValueError):
summary_module.cpu(rows, 500, 1500)

def test_refuses_unbracketed_window_and_backward_counters(self):
rows = [
dict(timestamp_ms=t, pid=1, cpu_user=t, cpu_system=0, ctx_voluntary=0, ctx_involuntary=0)
Expand Down
1 change: 0 additions & 1 deletion quest/m2/perf/shard-default.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,4 +63,3 @@ Update `doc/bin/relay/` for the default change.

- [kio channel contention](/quest/m2/perf/kio-channel-contention/README.md) - its per-thread spikes run on the workers
- [Performance survey](/quest/m2/performance-survey/README.md) - paired benchmark evidence and memory/settings controls
- [Socket queues](/quest/m2/performance-survey/socket-queues.md) - queue sizing and local drop attribution
10 changes: 9 additions & 1 deletion quest/m2/performance-survey/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,15 @@ performance suite, GPU/platform codecs, or every generated wrapper. Those
remain with their existing owners. The recurring audit is not completed by
this source survey.

UDP socket queues closed as a measured no-win: the 8 MiB request stays,
with no new setting. `benchmark_socket_queues` (moq-uring, nightly) showed
loss follows the kernel queue against worker stalls, while pool depth, flows
and the receive path did not change it. A bigger queue costs delay only
while a worker is overloaded and the queue sits full. Relay shard sweeps at
up to 256 connections saw no socket drops, with the sampled peak queue under
1% of the granted 16 MiB. The relay sampler now records per-socket drops,
granted buffers and queued bytes (`udp_*`).

An operation-count probe imported the actual JS `bandwidth.allocate`, counted
`Want.priority` getter reads, and called it for every reservation, as
`Allocator.#publish` does. With one tier, 16/64/256 reservations caused
Expand Down Expand Up @@ -200,7 +209,6 @@ The `js/replay` nightly series and the browser smoke's `replay.json` protect it.
- [HLS request fan-out](/quest/m2/performance-survey/hls-serving.md) - repeated fetch, remux and playlist construction per viewer
- [Buffered browser groups](/quest/m2/performance-survey/hang-buffered.md) - per-frame scans and bulk skip work
- [Idle browser stats](/quest/m2/performance-survey/browser-idle-stats.md) - local transport polling without active publishing demand
- [UDP socket queues](/quest/m2/performance-survey/socket-queues.md) - drop attribution and queue-size tradeoffs across workers and load
- [Announcement snapshots](/quest/m2/performance-survey/announced-snapshots.md) - HTTP polling cost versus result size and unrelated routes
- [Selective session operations](/quest/m2/performance-survey/session-selection.md) - exact-ID lookups and registry lock hold time
- [Browser media copies](/quest/m2/performance-survey/browser-media-copies.md) - CMAF sample and LOC framing copies versus retention
Expand Down
36 changes: 0 additions & 36 deletions quest/m2/performance-survey/socket-queues.md

This file was deleted.

2 changes: 2 additions & 0 deletions rs/moq-bench/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ usage-rs = { workspace = true, features = ["completions", "config"] }

# The host sampler reads /proc, so it is Linux-only (production relays are Linux).
[target.'cfg(target_os = "linux")'.dependencies]
# The sampler's sock_diag dump.
libc = { workspace = true }
moq-uring = { workspace = true, optional = true }
procfs = "0.18"

Expand Down
17 changes: 17 additions & 0 deletions rs/moq-bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,23 @@ process's share of the machine is computable. `--threads` adds a per-thread
breakdown including the core each thread last ran on, which is how to verify
pinning once the relay grows a thread-per-core mode.

The `udp_*` fields come from the kernel's own socket accounting (a `sock_diag`
dump, what `ss -uam` reads) for the process's UDP sockets: how many it holds
(`udp_sockets`), the smallest receive and send buffer the kernel granted
(`udp_rcvbuf_min`, `udp_sndbuf_min`, in kernel units, which on Linux are twice
the honored `SO_RCVBUF` request), the bytes queued for receive right now
(`udp_rx_queue_bytes`), and `udp_drops`, the datagrams dropped at those sockets
because a receive queue was full. `udp_drops` is cumulative across every socket
the sampler saw, closed ones included. One drop is one queued skb, so a
coalesced GRO train counts once however many datagrams it carried. A drop here
is local: the process was too slow to read. Loss on the path never reaches the
socket and needs the transport's own counters. `udp_rcvbuf_errors`,
`udp_sndbuf_errors` and `udp_in_errors` are the namespace's `/proc/net/snmp`
counters, IPv4 and IPv6 summed, and count every socket in the namespace. The
dump sees the sampler's network namespace, so run it in the target's. The
fields are absent when the host refuses the dump, or for another user's process,
whose `/proc/<pid>/fd` the sampler may not list; the other fields still sample.

## Measuring CPU cost

Run the load from one machine and the sampler on the relay's host, then join
Expand Down
Loading
Loading