Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ test:
python3 bench/compare.test.py
python3 bench/summary.test.py
python3 bench/profile.test.py
python3 bench/ffi.test.py
python3 sh/gh/gst-formula.test.py
python3 sh/changed.test.py
python3 sh/large-files.test.py
Expand Down
15 changes: 2 additions & 13 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,19 +108,6 @@ jobs:
# Unset keys default to the ARM runner, the plain dev shell, the Rust
# cache, no perf sampling, and nextest's default profile.
include:
- name: Binding benchmark
recipe: bench-ffi
artifact: bench-ffi
artifact-path: target/bench/ffi
benchmark: true
# Keeps the BoltFFI and UniFFI slice building and running for the
# browser generator comparison, which reuses it. It skips the size
# step, whose moq-ffi and moq-wasm builds double the time and are
# already built elsewhere.
- name: Generator comparison
recipe: bench-ffi-gen build bench
artifact: bench-ffi-gen
artifact-path: target/ffi-gen/results
- name: Audio grouping relay benchmark
recipe: bench-audio
- name: Relay shard sweep
Expand All @@ -135,6 +122,8 @@ jobs:
runner: ubuntu-24.04
- name: Rust benchmark tests
recipe: gh nightly rust-test
- name: Binding benchmark runners
recipe: gh nightly bindings
- name: Rust bench targets
recipe: gh nightly rust-bench
artifact: bench-rust
Expand Down
23 changes: 10 additions & 13 deletions .github/workflows/platform.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,13 @@ on:
workflow_call:
inputs:
benchmarks:
description: Measure Swift against Rust baselines on this runner
description: Smoke-run the Swift binding benchmark runner
type: boolean
default: false
workflow_dispatch:
inputs:
benchmarks:
description: Measure Swift against Rust baselines on this runner
description: Smoke-run the Swift binding benchmark runner
type: boolean
default: false

Expand Down Expand Up @@ -165,18 +165,15 @@ jobs:
- name: Swift
if: needs.scope.outputs.swift == 'true'
run: just swift check
- name: Swift binding benchmark
id: swift-bench
# At a token budget, so the runner keeps working at this commit; rust2's
# `just bench-ffi` measures it at its pinned moq revision.
- name: Swift binding benchmark runner
if: inputs.benchmarks
run: just bench-ffi swift
# Only after the benchmark ran, so an earlier failure stays the job's only error.
- name: Preserve Swift benchmark
if: ${{ !cancelled() && steps.swift-bench.outcome != 'skipped' }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: bench-ffi-swift
path: target/bench/ffi/
if-no-files-found: error
env:
MOQ_BENCH_SECONDS: "0.05"
run: |
just swift bench
test -s target/bench/ffi/swift.json
- name: OBS
if: needs.scope.outputs.obs == 'true'
run: just obs package --target aarch64-apple-darwin --output "$RUNNER_TEMP/obs"
Expand Down
65 changes: 22 additions & 43 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,43 +238,26 @@ hide a relay improvement behind generator saturation. CPU efficiency remains
withheld until offered load and delivery/loss are comparable. Zero-baseline
metrics retain absolute deltas when their percentage ratio is undefined.

Measure what each binding adds over Rust:

```bash
nix develop --command just bench-ffi
```

Every runner (Rust, Go, Python, Kotlin, C++, Dart, Swift) runs the same in-process rounds with no
network: each publisher writes one group from the driving thread while every
subscriber reads it on its own task, and a round waits for every subscriber. It
sweeps frame sizes from 100 B to 1 MiB at one publisher and subscriber, then
1/4 publishers by 1/4 subscribers at 1 KiB, and times two sync calls
(`sequence`, `info`). Each case reports delivered MB/s and process CPU (user +
system, every thread) per MB. `MOQ_BENCH_SECONDS` sets each case's time budget
(default 1); every received frame is checked byte for byte.

Each runner writes `target/bench/ffi/<runner>.json`; `bench/ffi.py` combines
them into a table and `report.json`. Ratios are against `rust-ffi`, moq-ffi's
API called from Rust without crossing the FFI, so a wrapper's ratio is what its
boundary and wrapper cost. `rust-net` runs the same rounds on moq-net, so its
ratio is what moq-ffi's handles and runtime cost. Ratios hold only within one
machine. The nightly runs it, prints the table in the job summary, and keeps the
results as the `bench-ffi` artifact. Each language also has its own recipe
(`just go bench`, `just py bench`, `just kt bench`, `just cpp bench`,
`just dart bench`, `just swift bench`). C++ uses C++20 reader coroutines;
Swift uses structured tasks; Dart uses futures on its event loop.
The payload checks and scheduling costs are included in each measurement.

Linux runs every wrapper except Swift. The nightly macOS platform job, or a
manual Platform run with `benchmarks` set, runs `just bench-ffi swift`,
measuring fresh Rust baselines on that same machine, and uploads
`bench-ffi-swift` separately. Never compare its absolute values
against the Linux job's baselines. A local run can select runners explicitly,
for example `just bench-ffi cpp dart`. Each invocation clears its output
directory and refuses a runner that produces no report. Dart's native asset
hook builds its usual minimal-feature release library; C++ and Swift use
their default-feature release libraries. No codec work is timed here.
C++ and Dart process CPU measurement supports Linux and macOS.
Each wrapper carries a runner of the binding benchmark, which measures what the
binding adds over Rust: `just go bench`, `just py bench`, `just kt bench`,
`just cpp bench`, `just dart bench`, and `just swift bench` (macOS), beside the
Rust baselines in `rs/moq-ffi/src/bench.rs`. Every runner runs the same
in-process rounds with no network: each publisher writes one group from the
driving thread while every subscriber reads it on its own task, and a round
waits for every subscriber. It sweeps frame sizes from 100 B to 1 MiB at one
publisher and subscriber, then 1/4 publishers by 1/4 subscribers at 1 KiB, and
times two sync calls (`sequence`, `info`). Each case reports delivered MB/s and
process CPU (user + system, every thread) per MB into
`$MOQ_BENCH_DIR/<runner>.json`. `MOQ_BENCH_SECONDS` sets each case's time
budget (default 1); every received frame is checked byte for byte.

The binding benchmark itself lives in
[rust2](https://github.com/kixelated/rust2/blob/main/bench/ffi/README.md), with
the UniFFI fork whose changes it measures: its `just bench-ffi` runs these
runners nightly at a pinned moq revision, against the fork, and reports each
wrapper's ratio to Rust. So that they keep working here, moq's Nightly runs every
runner at a token budget without measuring it: `just gh nightly bindings` on
Linux, and `just swift bench` in the macOS Platform job.

## Browser

Expand Down Expand Up @@ -403,9 +386,6 @@ that artifact first or wait for the next Nightly.

The table includes these measurements:

- Binding delivery in MB/s, process CPU in ms/MB, sync calls in ns, and each
wrapper's ratio to `rust-ffi`. Ratios above 1 mean more cost than Rust on the
same host, including for throughput, where the ratio is Rust/wrapper.
- Rust Criterion mean ns per iteration; the history also keeps the confidence interval.
The `net-cpu` source and the moq-quic targets time thread CPU; the others
time elapsed time. The Rust bench target jobs measure them with a short warmup
Expand All @@ -416,7 +396,7 @@ The table includes these measurements:
dimension in each series name.

Hosted runners vary across nights, so a trend flag prompts a look at the paired
comparison rather than proving a regression. Binding ratios compare languages within one run.
comparison rather than proving a regression.
Malformed, empty, nonfinite, or duplicate measurements fail validation. The
nightly artifacts retain the reports, raw JS output, and each Criterion run's
metadata and estimates for 45 days. Relay load, audio quality, and the Rust
Expand All @@ -434,12 +414,11 @@ Then open `.scratch/pages/bench/index.html`. Without `--commit`, the newest
recorded commit counts as tonight's. Reproduce the measurements locally:

```bash
nix develop --command just bench-ffi
nix develop --command just gh nightly rust-bench
nix develop --command just gh nightly js
```

The latter two write `target/bench/nightly/{rust-bench,js}/report.json`. For a focused
These write `target/bench/nightly/{rust-bench,js}/report.json`. For a focused
measured Rust run using the same evidence lifecycle:

```bash
Expand Down
Loading
Loading