Skip to content

feat(bench): move moq's binding benchmark here, with a browser runner - #19

Merged
kixelated merged 21 commits into
mainfrom
quest/m1/ffi-bench
Oct 25, 2026
Merged

kixelated merged 21 commits into
mainfrom
quest/m1/ffi-bench

Conversation

@kixelated

@kixelated kixelated commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The fork's perf quests (payload bytes, Kotlin calls, foreign poll) must state their effect on moq's bindings, but the binding benchmark lived in moq and measured moq's pinned generator forks, so a change here could not be measured until moq switched to it. Closes the binding benchmark quest.

Approach

just bench-ffi (bench/ffi/), next to bench/build and following its conventions:

  • Clones moq at the revision in bench/ffi/pins.toml into /var/tmp/rust2-bench-<user>/ffi/, with a private CARGO_HOME and none of the caller's cargo or rustc variables.
  • Points every uniffi dependency in moq's manifests at this tree's uniffi/ (a path rewrite: the fork's crates are 0.1.0, so [patch] cannot match 0.32) and fails if any UniFFI crate still resolves elsewhere. The fork's Go, Dart, and C++ generators go first on PATH. No moq change or release is needed to measure a fork change.
  • Enters the pinned moq's own dev shell and drives moq's runners as moq's bench/ffi.sh did: the Rust baselines (rust-ffi, rust-net), Go, Python, Kotlin, C++, Dart, and Swift on macOS.
  • Adds the browser runner, which moq never had: moq-ffi built for wasm32, TypeScript generated by ubrn's wasm2 player, the same rounds in headless Chromium (browser-page.ts, browser.ts), CPU read from the renderer over CDP. No ubrn release reads UniFFI 0.32, so it pins the head of jhugman/uniffi-bindgen-react-native#468 and builds the CLI and @ubjs/* runtimes from it.
  • Variants fork, control (an A/A rerun of fork's builds), and with --base REV the fork at another rust2 revision. An untimed warm-up builds everything; timed rounds interleave the variants and rotate their order. The report shows trimmed means (as the build benchmark now does), each wrapper's cost against rust-ffi within the same round, the control's change (noise floor), and fork against base. It records the host, load average before and after, and every toolchain version.
  • CI: bench-ffi.yml runs nightly and on PRs touching uniffi/**, the root manifests, bench/ffi/**, or the toolchains; a PR run passes --base with the PR's base, so fork changes report their effect. A Linux job runs every runner; a macOS job runs Swift. bench.yml now triggers only on bench/build/**.
  • The BoltFFI comparison is archived as tables in bench/ffi/README.md; its harness is deleted in moq.

Numbers from this machine (AMD Ryzen 9 5900XT, 32 threads), 3 rounds of 1 s per case, shared with other agents (load average 13 to 19): see the comment below for the full report. Selected trimmed means (with 3 rounds, the median), cost against rust-ffi in parentheses:

case rust-ffi go python kotlin cpp dart browser
100 B, MB/s 83.6 13.1 (6.4x) 1.98 (42x) 1.86 (45x) 31.8 (2.6x) 10.1 (8.3x) 14.5 (5.8x)
1 KiB, MB/s 688 104 (6.8x) 19.2 (37x) 20.1 (32x) 85.4 (8.1x) 95 (7.0x) 46.5 (14x)
1 MiB, MB/s 1794 327 (5.3x) 377 (5.4x) 340 (4.9x) 89.7 (20x) 226 (9.1x) 69 (29x)
100 B, CPU ms/MB 13.9 119 (8.5x) 504 (36x) 2502 (179x) 45.1 (3.3x) 121 (8.7x) 82.5 (6.2x)
sequence(), ns 0.48 141 2415 12543 12 106 154
info(), ns 1.44 473 7228 22033 123 281 447

The A/A control moved most cells by 1 to 20% and a few Kotlin CPU cells by up to 90% under that load; the second A/A (--base at an identical fork) agreed. Compare a change only beyond the control's spread.

Moq's paired PR: moq-dev/moq#6629 deletes just bench-ffi, bench/ffi-gen, and their nightly and Platform jobs (draft; moq merges need the maintainer).

Impact

  • No public API change. New recipe just bench-ffi; no Rust code changes.
  • Moq's nightly trend table loses the binding series; this report replaces them.

Alternatives

  • Keep the harness in moq and call it from here: it measures moq's pinned generators, so a fork change waits on moq's switch.
  • Defer the browser runner until ubrn releases 0.32 support: the payload-bytes quest needs browser numbers now, and the PR head already reads the fork's metadata. A waiting quest tracks moving to a release.
  • [patch.crates-io] instead of rewriting manifests: cargo ignores a patch whose version (0.1.0) does not match the requirement (0.32).
  • A shared target directory for all runners: they build moq-ffi with different features and environments and rebuilt each other every round (3 to 8 minutes per runner per round); every runner except Dart, whose script hardcodes the default directory, now gets its own.
  • Running the Rust baselines through just rs test: its clippy gate failed on macOS on a moq lint (an unfulfilled clippy::too_many_arguments expectation in moq-video), which says nothing about speed. The run calls nextest directly.

Follow-ups

  • Waiting: ubrn reads UniFFI 0.32: pin a ubrn release instead of the PR head.
  • Symlinked paths: on macOS the fork's uniffi_bindgen failed to build through /var (a symlink), with askama reading askama.toml across it. The benchmark now builds from physical paths; the quest fixes the fork.
  • moq: just rs test fails clippy on macOS at the pinned revision (an unfulfilled clippy::too_many_arguments expectation in moq-video).
  • Moq's Dart script runs cargo under env -i, so it reads ~/.cargo rather than the run's CARGO_HOME; harmless today (the only user setting seen is build.incremental = false), but it leaks a user's cargo config into the Dart library. Worth fixing in moq if it bites.

Checks

  • just check and just test pass.
  • Local: a 1-round smoke of every runner, and a 3-round run with --base origin/main (report in the first comment).
  • CI (run 40152584784): Linux ran every runner with --base in 2.5 h, rounds about 2 minutes per variant once built; macOS ran Swift in 1.1 h. Reports in the second comment.
  • The Bench (build benchmark) runs on this PR are cancelled on purpose: they trigger because bench.yml's path filter changed, and take hours measuring nothing this PR touches.

🤖 Generated with Claude Code

https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn

(written by Opus 5.5)

kixelated and others added 3 commits October 23, 2026 23:43
@kixelated kixelated changed the title Move moq's binding benchmark to rust2 feat(bench): move moq's binding benchmark here, with a browser runner Oct 24, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Local report

3 rounds, --base origin/main (an identical fork, so a second A/A).

2026-10-24T15:34:31Z on AMD Ryzen 9 5900XT 16-Core Processor (32 CPUs, 63 GiB), Linux 6.18.56 x86_64, runner local; load average 17.32 15.39 13.00 before, 16.73 15.66 19.24 after.
rust2 353cde96e67f, base f450cf555a6f; moq 0f986d9fdb7c, ubrn 06f0a30d2575; 3 rounds of 1 s per case.
rustc 1.98.1 (48a229cea 2026-09-01); Python 3.14.8; go version go1.26.9 linux/amd64; openjdk version "17.0.20.1" 2026-08-18; Dart SDK version: 3.13.0 (stable) (Sun Oct 11 04:47:24 2026 +0000) on "linux_x64"; cmake version 4.4.4; g++ (GCC) 16.2.0; Playwright 1.64.1.

Each cell is the median over the rounds, and the median of its cost against rust-ffi in the same round: above 1x is slower or costlier.
rust-ffi calls moq-ffi's API from Rust without crossing the FFI; rust-net runs the same rounds on moq-net, below moq-ffi's handles and runtime.

fork

case unit rust-net rust-ffi go python kotlin cpp dart browser
throughput/size=100/pub=1/sub=1 MB/s 315 (0.268x) 83.6 13.1 (6.4x) 1.98 (42.3x) 1.86 (44.8x) 31.8 (2.62x) 10.1 (8.34x) 14.5 (5.81x)
cpu/size=100/pub=1/sub=1 ms/MB 3.15 (0.235x) 13.9 119 (8.51x) 504 (35.9x) 2502 (179x) 45.1 (3.33x) 121 (8.68x) 82.5 (6.15x)
throughput/size=1024/pub=1/sub=1 MB/s 1817 (0.405x) 688 104 (6.82x) 19.2 (37.2x) 20.1 (32.4x) 85.4 (8.06x) 95 (6.98x) 46.5 (14x)
cpu/size=1024/pub=1/sub=1 ms/MB 0.544 (0.355x) 1.65 18.2 (11.2x) 51.9 (33x) 142 (81.8x) 21.3 (12.9x) 11.3 (6.5x) 22.4 (13.2x)
throughput/size=65536/pub=1/sub=1 MB/s 6320 (0.79x) 4995 412 (13x) 297 (16.1x) 234 (20.1x) 104 (48.7x) 341 (15.7x) 67.9 (78x)
cpu/size=65536/pub=1/sub=1 ms/MB 0.156 (0.779x) 0.201 5.64 (29.6x) 3.35 (16x) 21.9 (100x) 18.6 (93x) 2.93 (15.6x) 14.9 (78.4x)
throughput/size=1048576/pub=1/sub=1 MB/s 5098 (0.352x) 1794 327 (5.3x) 377 (5.44x) 340 (4.93x) 89.7 (20x) 226 (9.06x) 69 (28.6x)
cpu/size=1048576/pub=1/sub=1 ms/MB 0.194 (0.349x) 0.557 5.96 (10.2x) 2.64 (5.43x) 4.48 (5.5x) 19.4 (34.8x) 6.94 (14.3x) 14.7 (29.2x)
throughput/size=1024/pub=1/sub=4 MB/s 5767 (0.178x) 968 306 (3.28x) 35.4 (29x) 42 (20.7x) 89.3 (10.8x) 132 (7.17x) 71.1 (13.2x)
cpu/size=1024/pub=1/sub=4 ms/MB 0.172 (0.151x) 1.2 13.1 (11.3x) 28.3 (24.8x) 67.5 (50.5x) 14.2 (11.8x) 7.87 (6.46x) 14.7 (11.6x)
throughput/size=1024/pub=4/sub=1 MB/s 1730 (0.391x) 677 170 (3.98x) 23.2 (29.5x) 22.9 (31x) 86.4 (8.13x) 108 (6.52x) 50.5 (14.1x)
cpu/size=1024/pub=4/sub=1 ms/MB 0.572 (0.348x) 1.64 18.1 (10.7x) 43.1 (26.2x) 125 (77.6x) 21.2 (13.3x) 9.53 (6x) 20.8 (13.1x)
throughput/size=1024/pub=4/sub=4 MB/s 5786 (0.166x) 932 352 (2.65x) 39.9 (24.1x) 31.2 (30.8x) 85.6 (11.1x) 130 (7.18x) 68.3 (13.7x)
cpu/size=1024/pub=4/sub=4 ms/MB 0.171 (0.139x) 1.25 18.4 (14.8x) 25.1 (20.4x) 118 (96x) 14.6 (11.9x) 8.12 (6.51x) 15.6 (12.5x)
call/sequence ns 0.475 141 (303x) 2415 (5175x) 12543 (25952x) 12 (25.6x) 106 (228x) 154 (334x)
call/info ns 1.44 473 (305x) 7228 (5149x) 22033 (15153x) 123 (85.6x) 281 (198x) 447 (310x)

Noise: control against fork

control reruns fork's builds in the same rounds, so its change is this run's noise floor.
Change in median cost (time per call, CPU per MB, or the inverse of throughput): negative is cheaper.

case unit rust-net rust-ffi go python kotlin cpp dart browser
throughput/size=100/pub=1/sub=1 MB/s -10.7% +10.9% +4.5% -2.7% +5.7% +15.5% -5.1% +2.8%
cpu/size=100/pub=1/sub=1 ms/MB -10.9% +9.6% +5.6% -2.4% -9.3% +18.7% -7.5% +1.1%
throughput/size=1024/pub=1/sub=1 MB/s -11.8% +31.3% +4.3% -1.5% -9.2% +21.3% +12.7% -2.6%
cpu/size=1024/pub=1/sub=1 ms/MB -11.4% +29.8% +4.3% -1.5% -4.9% +18.3% +12.6% -0.7%
throughput/size=65536/pub=1/sub=1 MB/s +16.8% +149.5% -4.1% -3.5% +0.6% +6.7% +10.6% +3%
cpu/size=65536/pub=1/sub=1 ms/MB +16.9% +149.2% -5.9% -3.4% -11.7% +4.7% +10.5% +2.2%
throughput/size=1048576/pub=1/sub=1 MB/s +28% +27.3% +4.3% -0.6% +0% +6.2% +18.3% +0.4%
cpu/size=1048576/pub=1/sub=1 ms/MB +27.8% +26.9% +5.6% -0.6% +10.4% +4.3% +19.9% +0%
throughput/size=1024/pub=1/sub=4 MB/s +7.8% +14% -0.5% -0.9% -1.2% +7.6% +3.3% +1.7%
cpu/size=1024/pub=1/sub=4 ms/MB +7.8% +18.2% -2.4% -0.9% -25.1% +10.8% +5.7% +0.9%
throughput/size=1024/pub=4/sub=1 MB/s +15.3% +22.4% +1.9% -0.9% +8.9% +13.2% +5.1% +1.6%
cpu/size=1024/pub=4/sub=1 ms/MB +15.5% +22.5% +1% -0.9% +93.5% +10.6% +6.4% +2.5%
throughput/size=1024/pub=4/sub=4 MB/s +0.8% +25.3% +15.3% +0.7% +36.2% +7.1% +16.7% +8.9%
cpu/size=1024/pub=4/sub=4 ms/MB +0.9% +25.4% +11.8% +0.6% +22% +6.8% +17% +9%
call/sequence ns +14.1% +9.8% +6% +8.2% +5.6% +0.8% +2%
call/info ns +3.4% -4.1% +3.9% +35.8% +2.6% +6.8% -0.4%

Change: fork against base

The change in cost from base to fork. Compare it with the noise floor above.

case unit rust-net rust-ffi go python kotlin cpp dart browser
throughput/size=100/pub=1/sub=1 MB/s +21.1% +9.9% -1.4% -12.7% +4% -3.5% +2.3% -17.2%
cpu/size=100/pub=1/sub=1 ms/MB +21.1% +13.3% -1.1% -12.9% +19.2% -3.2% -0.6% -19.2%
throughput/size=1024/pub=1/sub=1 MB/s +14.9% +13.9% -4% -1.4% +10.3% -2% +4.2% -0.2%
cpu/size=1024/pub=1/sub=1 ms/MB +14.5% +18.4% -4.4% -1.5% -6.1% -1.7% +5% -3.1%
throughput/size=65536/pub=1/sub=1 MB/s -4.2% +6.3% -2.5% -2.6% -6.3% -2.6% +2% -3.2%
cpu/size=65536/pub=1/sub=1 ms/MB -4.2% +6.2% +3.4% -2.6% -9.1% -2.8% +2% -3.1%
throughput/size=1048576/pub=1/sub=1 MB/s +0.6% +16.8% -10.2% -8.4% +3.7% -1.9% +3.6% -4.2%
cpu/size=1048576/pub=1/sub=1 ms/MB +0.5% +17.1% -9.6% -8.4% +17% -2.1% +9.3% -5.1%
throughput/size=1024/pub=1/sub=4 MB/s +8.1% +5.4% -10.1% -6.4% +9.1% +1.8% +2.1% -4.7%
cpu/size=1024/pub=1/sub=4 ms/MB +8% +7.5% -6.9% -6.1% +50% +1.3% +1% -4.8%
throughput/size=1024/pub=4/sub=1 MB/s +4% +11.1% +1.8% -5.2% -1.3% -1.1% -0.6% -4.9%
cpu/size=1024/pub=4/sub=1 ms/MB +4% +8.7% +0.9% -5.3% -44.7% -1.2% -2.2% -3.7%
throughput/size=1024/pub=4/sub=4 MB/s -4.3% +7.3% -1.3% -4.2% -1.2% +3% +0.1% -4.9%
cpu/size=1024/pub=4/sub=4 ms/MB -4.2% +6.2% -1.4% -4.1% -24.9% +4.4% +0.1% -6.1%
call/sequence ns +3.6% -5.2% -4.3% +8.6% +0.3% +3.9% +3.9%
call/info ns +2.8% -2.7% +0.1% -0.8% -0.7% +0.3% +5%

(written by Opus 5.5)

# Conflicts:
#	quest/README.md
#	quest/m1/README.md
#	quest/m1/ffi-bytes.md
#	quest/m1/ffi-poll-protocol.md
#	quest/m1/kt-ffi-calls.md
#	quest/m2/ffi-code-size.md
@kixelated

Copy link
Copy Markdown
Collaborator Author

CI reports

From run 40152584784, with --base at the PR base (an identical fork, so the base change is a second A/A).

Linux

2026-10-25T04:29:14Z on AMD EPYC 7763 64-Core Processor (4 CPUs, 16 GiB), Linux 6.17.0-1022-azure x86_64, runner GitHub Actions 1000009963; load average 2.84 2.85 1.64 before, 2.19 1.69 1.92 after.
rust2 23f3f64953b5, base 8b30a58ff437; moq 0f986d9fdb7c, ubrn 06f0a30d2575; 3 rounds of 1 s per case.
rustc 1.98.1 (48a229cea 2026-09-01); Python 3.14.8; go version go1.26.9 linux/amd64; openjdk version "17.0.20.1" 2026-08-18; Dart SDK version: 3.13.0 (stable) (Sun Oct 11 04:47:24 2026 +0000) on "linux_x64"; cmake version 4.4.4; g++ (GCC) 16.2.0; Swift version 6.4 (swift-6.4-RELEASE); Playwright 1.64.1.

Each cell is the trimmed mean over the rounds (without the fastest and slowest), and the trimmed mean of its cost against rust-ffi in the same round: above 1x is slower or costlier.
rust-ffi calls moq-ffi's API from Rust without crossing the FFI; rust-net runs the same rounds on moq-net, below moq-ffi's handles and runtime.

fork

case unit rust-net rust-ffi go python kotlin cpp dart browser
throughput/size=100/pub=1/sub=1 MB/s 307 (0.206x) 63.6 8.79 (7.18x) 0.896 (70.9x) 0.78 (84x) 20.4 (3.17x) 7.58 (8.31x) 8.51 (7.4x)
cpu/size=100/pub=1/sub=1 ms/MB 3.25 (0.165x) 19.7 160 (8.1x) 1125 (57.1x) 4039 (203x) 74.5 (3.79x) 163 (8.16x) 149 (7.49x)
throughput/size=1024/pub=1/sub=1 MB/s 1990 (0.303x) 590 64.3 (9.1x) 8.75 (67x) 15.4 (39.1x) 58.5 (10.2x) 64.5 (9.09x) 33.4 (17.5x)
cpu/size=1024/pub=1/sub=1 ms/MB 0.502 (0.243x) 2.1 25.3 (12.2x) 115 (54.9x) 149 (70.9x) 31.4 (15.1x) 17.4 (8.3x) 31.9 (15.2x)
throughput/size=65536/pub=1/sub=1 MB/s 8615 (0.678x) 5810 291 (20.1x) 179 (32.5x) 167 (29.2x) 72.5 (79.8x) 251 (23x) 46.8 (124x)
cpu/size=65536/pub=1/sub=1 ms/MB 0.116 (0.649x) 0.18 7.02 (39.1x) 5.6 (31.2x) 16.6 (59.1x) 26.7 (148x) 4.03 (22.1x) 21.7 (120x)
throughput/size=1048576/pub=1/sub=1 MB/s 8578 (0.321x) 2850 241 (11.8x) 250 (11.4x) 255 (11.2x) 65.7 (43.4x) 187 (15.6x) 47.7 (59.7x)
cpu/size=1048576/pub=1/sub=1 ms/MB 0.117 (0.317x) 0.356 7.91 (22.4x) 4.01 (11.3x) 4.58 (12.5x) 26.8 (75.3x) 7.67 (22.5x) 21.5 (59.8x)
throughput/size=1024/pub=1/sub=4 MB/s 5414 (0.145x) 782 173 (4.52x) 15.7 (49.7x) 30.6 (25.7x) 61 (12.8x) 93.9 (8.37x) 49.5 (15.9x)
cpu/size=1024/pub=1/sub=4 ms/MB 0.185 (0.113x) 1.66 17.5 (10.4x) 64.2 (38.7x) 62.6 (37.2x) 20.9 (12.6x) 11.2 (6.83x) 21.3 (13x)
throughput/size=1024/pub=4/sub=1 MB/s 1772 (0.324x) 564 106 (5.32x) 10.5 (53.6x) 15.8 (36.4x) 60.1 (9.54x) 76.8 (7.34x) 35.6 (15.9x)
cpu/size=1024/pub=4/sub=1 ms/MB 0.564 (0.266x) 2.16 25 (11.6x) 95.6 (44.3x) 153 (69.4x) 30.9 (14.5x) 13.6 (6.27x) 29.2 (13.5x)
throughput/size=1024/pub=4/sub=4 MB/s 5096 (0.147x) 746 191 (3.86x) 17.7 (42.5x) 20.5 (40.8x) 61.7 (12.1x) 95.6 (7.8x) 49.5 (15.1x)
cpu/size=1024/pub=4/sub=4 ms/MB 0.196 (0.116x) 1.71 17.3 (9.99x) 56.9 (33.6x) 162 (104x) 20.3 (11.9x) 10.9 (6.45x) 21.5 (12.5x)
call/sequence ns 0.652 202 (311x) 5667 (8695x) 13410 (20588x) 17.5 (26.9x) 149 (229x) 218 (335x)
call/info ns 1.9 613 (322x) 15511 (8163x) 43698 (22969x) 181 (95.2x) 414 (217x) 631 (332x)

Noise: control against fork

control reruns fork's builds in the same rounds, so its change is this run's noise floor.
Change in trimmed-mean cost (time per call, CPU per MB, or the inverse of throughput): negative is cheaper.

case unit rust-net rust-ffi go python kotlin cpp dart browser
throughput/size=100/pub=1/sub=1 MB/s +0.2% -0% +1.3% +0.2% +22.6% +1.5% +0.3% -1.9%
cpu/size=100/pub=1/sub=1 ms/MB +0.2% +0.3% +2.4% +0.2% +19.9% +1.1% +1.7% -2.6%
throughput/size=1024/pub=1/sub=1 MB/s +0% +0.7% -4.4% +0.4% -0.3% +1.9% -0.7% +0.2%
cpu/size=1024/pub=1/sub=1 ms/MB +0% +0.6% -1.6% +0.4% -8.8% +2.1% -1.7% +0.5%
throughput/size=65536/pub=1/sub=1 MB/s +1.9% +36.7% -0% -0.9% -19.5% -0.5% +0.2% +4.4%
cpu/size=65536/pub=1/sub=1 ms/MB +1.9% +34.8% +0.5% -0.9% -57% -0.5% +0% +5.1%
throughput/size=1048576/pub=1/sub=1 MB/s +7.3% -7.8% -0.3% -0.2% +0.3% +0.2% -0.7% +7.9%
cpu/size=1048576/pub=1/sub=1 ms/MB +7.3% -7.7% -1.7% -0.2% -1% +0.2% +2.1% +8.3%
throughput/size=1024/pub=1/sub=4 MB/s -0.3% -3.2% +3.3% +0.5% +17.2% +0.2% +0.1% -0.3%
cpu/size=1024/pub=1/sub=4 ms/MB -0.3% -2.6% +0.3% +0.4% +42.6% +0.1% +0.1% 0%
throughput/size=1024/pub=4/sub=1 MB/s +1.8% -5.1% -1.1% +0.7% +50.9% -0.4% +0.8% +0.3%
cpu/size=1024/pub=4/sub=1 ms/MB +1.8% -4.7% -1.8% +0.7% +113.5% +0% +1.4% -1%
throughput/size=1024/pub=4/sub=4 MB/s +3% -5.1% +0.2% +1.2% -7.1% +0.1% +1.4% +2.4%
cpu/size=1024/pub=4/sub=4 ms/MB +3% -4.2% +0.4% +1% -16% +1% +0.9% +2.1%
call/sequence ns +0.1% +0.1% +0.8% +15% -0.3% +0.6% -1.6%
call/info ns -0.1% -1.1% +0.6% -20.7% +0.1% -3.1% +1.5%

Change: fork against base

The change in cost from base to fork. Compare it with the noise floor above.

case unit rust-net rust-ffi go python kotlin cpp dart browser
throughput/size=100/pub=1/sub=1 MB/s +1.8% +8.7% -0.9% -0.7% -8.4% +0.6% +0.4% +1.1%
cpu/size=100/pub=1/sub=1 ms/MB +1.8% +7% -2.4% -0.7% -6% +1.8% +0.9% +0.3%
throughput/size=1024/pub=1/sub=1 MB/s -1.6% +2.7% +0.4% -1.3% +1.3% -1.2% +1.2% +0.3%
cpu/size=1024/pub=1/sub=1 ms/MB -1.6% +1.9% -1.8% -1.4% +9.2% -1.9% +2% +1.2%
throughput/size=65536/pub=1/sub=1 MB/s -0.3% +6.2% -0% +0.7% +24.6% +0.9% -0.6% +0.1%
cpu/size=65536/pub=1/sub=1 ms/MB -0.3% +6% +0.2% +0.8% +97% +0.5% -0.8% -0.9%
throughput/size=1048576/pub=1/sub=1 MB/s -5.3% -10.1% -1.8% +0.6% +0.2% +0% +2.5% +0.9%
cpu/size=1048576/pub=1/sub=1 ms/MB -5.2% -10.1% -0.7% +0.7% +4.4% -0.1% -0.6% +0.9%
throughput/size=1024/pub=1/sub=4 MB/s +2.3% +7.2% -5.6% +0.2% -3.6% +0.3% -0.7% +0.3%
cpu/size=1024/pub=1/sub=4 ms/MB +2.3% +6.3% -4.9% +0.2% -4.4% +0.4% -0.7% 0%
throughput/size=1024/pub=4/sub=1 MB/s -0.1% +8.2% -1.8% +0.1% -8.1% +1.2% +0.4% -0.2%
cpu/size=1024/pub=4/sub=1 ms/MB -0.1% +6.8% -0.5% +0.1% -29.1% +0.6% +0% +1%
throughput/size=1024/pub=4/sub=4 MB/s -0% +4.5% -0.5% -1.7% +14% +0% +0.5% +0.3%
cpu/size=1024/pub=4/sub=4 ms/MB -0% +3.5% -1.1% -1.6% +34.2% +0.2% -0% +0%
call/sequence ns +0% -4.4% -0.1% -18.3% +0.2% -1% +2.7%
call/info ns +0.2% -1.1% -0.4% +49.9% +1.1% +3.4% +0%

macOS (Swift)

2026-10-25T03:03:44Z on Apple M1 (Virtual) (3 CPUs, 7 GiB), Darwin 25.6.0 arm64, runner GitHub Actions 1000009968; load average 4.63 4.93 6.54 before, 1.75 2.13 3.10 after.
rust2 23f3f64953b5, base 8b30a58ff437; moq 0f986d9fdb7c; 3 rounds of 1 s per case.
rustc 1.98.1 (48a229cea 2026-09-01); Python 3.14.8; go version go1.26.9 darwin/arm64; openjdk version "17.0.19" 2026-04-21 LTS; Dart SDK version: 3.13.0 (stable) (Wed Aug 5 00:28:05 2026 -0700) on "macos_arm64"; cmake version 4.4.4; clang version 21.1.8; Apple Swift version 6.3.3 (swiftlang-6.3.3.1.3 clang-2100.1.1.101); Playwright 1.64.1.

Each cell is the trimmed mean over the rounds (without the fastest and slowest), and the trimmed mean of its cost against rust-ffi in the same round: above 1x is slower or costlier.
rust-ffi calls moq-ffi's API from Rust without crossing the FFI; rust-net runs the same rounds on moq-net, below moq-ffi's handles and runtime.

fork

case unit rust-net rust-ffi swift
throughput/size=100/pub=1/sub=1 MB/s 162 (0.578x) 92.3 20.4 (4.49x)
cpu/size=100/pub=1/sub=1 ms/MB 2.98 (0.336x) 8.95 56.2 (6.22x)
throughput/size=1024/pub=1/sub=1 MB/s 1451 (0.59x) 856 176 (4.79x)
cpu/size=1024/pub=1/sub=1 ms/MB 0.381 (0.377x) 1.01 7.25 (7.35x)
throughput/size=65536/pub=1/sub=1 MB/s 16882 (0.678x) 10898 1174 (9.24x)
cpu/size=65536/pub=1/sub=1 ms/MB 0.054 (0.632x) 0.086 1.6 (18.5x)
throughput/size=1048576/pub=1/sub=1 MB/s 18477 (0.62x) 11304 1243 (9.09x)
cpu/size=1048576/pub=1/sub=1 ms/MB 0.051 (0.578x) 0.089 1.31 (14.7x)
throughput/size=1024/pub=1/sub=4 MB/s 4446 (0.29x) 1284 336 (3.82x)
cpu/size=1024/pub=1/sub=4 ms/MB 0.145 (0.162x) 0.896 6.59 (7.36x)
throughput/size=1024/pub=4/sub=1 MB/s 2641 (0.371x) 979 316 (3.1x)
cpu/size=1024/pub=4/sub=1 ms/MB 0.3 (0.26x) 1.15 7.89 (6.84x)
throughput/size=1024/pub=4/sub=4 MB/s 7102 (0.203x) 1429 408 (3.53x)
cpu/size=1024/pub=4/sub=4 ms/MB 0.12 (0.139x) 0.865 6.65 (7.97x)
call/sequence ns 0.552 15.4 (26.3x)
call/info ns 1.76 1714 (1004x)

Noise: control against fork

control reruns fork's builds in the same rounds, so its change is this run's noise floor.
Change in trimmed-mean cost (time per call, CPU per MB, or the inverse of throughput): negative is cheaper.

case unit rust-net rust-ffi swift
throughput/size=100/pub=1/sub=1 MB/s -6.3% +3.6% +1.5%
cpu/size=100/pub=1/sub=1 ms/MB -9.2% -0.7% +0.8%
throughput/size=1024/pub=1/sub=1 MB/s -2% +9% -0.1%
cpu/size=1024/pub=1/sub=1 ms/MB -2.2% -0.8% -0.2%
throughput/size=65536/pub=1/sub=1 MB/s -1.7% -3.5% -2.9%
cpu/size=65536/pub=1/sub=1 ms/MB -1.8% -0.6% -2.3%
throughput/size=1048576/pub=1/sub=1 MB/s +1% +10.9% +0.3%
cpu/size=1048576/pub=1/sub=1 ms/MB +1.1% +7.2% +0.5%
throughput/size=1024/pub=1/sub=4 MB/s +0.6% -0.4% +3.6%
cpu/size=1024/pub=1/sub=4 ms/MB +0.5% -6.9% +0.7%
throughput/size=1024/pub=4/sub=1 MB/s +6.5% -10.3% +15.6%
cpu/size=1024/pub=4/sub=1 ms/MB +8.3% -11.4% +4%
throughput/size=1024/pub=4/sub=4 MB/s +0.7% -3.5% -1.5%
cpu/size=1024/pub=4/sub=4 ms/MB +1.4% -2.9% +2%
call/sequence ns +2.2% -4.9%
call/info ns -2.4% +9.4%

Change: fork against base

The change in cost from base to fork. Compare it with the noise floor above.

case unit rust-net rust-ffi swift
throughput/size=100/pub=1/sub=1 MB/s +0.5% +4.4% -1.2%
cpu/size=100/pub=1/sub=1 ms/MB +0.3% +5.1% -0.6%
throughput/size=1024/pub=1/sub=1 MB/s -0.4% -5.5% -7.1%
cpu/size=1024/pub=1/sub=1 ms/MB -1.4% -0.8% -7.7%
throughput/size=65536/pub=1/sub=1 MB/s +1.2% +10.8% -5%
cpu/size=65536/pub=1/sub=1 ms/MB +1.2% +7.8% -5.3%
throughput/size=1048576/pub=1/sub=1 MB/s -6.1% +9.5% -1.4%
cpu/size=1048576/pub=1/sub=1 ms/MB -6% +9.5% -1.6%
throughput/size=1024/pub=1/sub=4 MB/s +0.5% +4.7% -7.6%
cpu/size=1024/pub=1/sub=4 ms/MB +0.7% +5.8% -8.6%
throughput/size=1024/pub=4/sub=1 MB/s -7% +4.1% -13.9%
cpu/size=1024/pub=4/sub=1 ms/MB -8.6% +4.7% -13.4%
throughput/size=1024/pub=4/sub=4 MB/s -1.7% +3.4% -3.2%
cpu/size=1024/pub=4/sub=4 ms/MB -2.6% +3% -4.6%
call/sequence ns +3% -0.5%
call/info ns +3.8% -1.3%

(written by Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 25, 2026 05:04
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T14:49:02.313011Z e975390 New commits
🔒 Security Review ✅ Completed 2026-10-25T14:54:58.017020Z e975390 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 305bf39

[P2] Serialize access to the shared benchmark work directory. In bench/ffi/run.sh:327–330, each invocation forcibly removes the same per-user rust2-base worktree and installs an EXIT trap that removes it again. Separate invocations, including from different rust2 worktrees with different --out directories, also share the moq checkouts and generator binaries. Starting a second benchmark can replace the first benchmark's inputs during measurement, and either run can delete the other's base checkout. Acquire an exclusive lock before preparation and hold it through measurement and cleanup, or allocate independent mutable directories per invocation.

Overall direction is good: keeping the harness beside the fork makes changes measurable without a downstream pin update; pinned inputs, A/A control and rotating variant order are appropriate. One work-directory lock is simpler than duplicating build caches.

Verification: static review of the harness, browser runner, reporting, workflow and surrounding rust2 configuration only. No tests or benchmarks run; external pinned moq/ubrn sources were not inspected.

@kixelated

kixelated commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Fixed the P2 in 84c6575 (merged as 9d7bbfb): every run now takes an exclusive flock on $work/lock before preparing and holds it through measurement and cleanup, so a second run waits instead of replacing the first one's checkouts, generators, or base worktree. Waiting rather than failing also keeps two runs from timing each other. flock joins the default dev shell, so it is there on macOS too.

(written by Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 9d7bbfb

The shared-work-directory P2 from the previous review is fixed: bench/ffi/run.sh:205–208 now holds an exclusive lock through preparation, measurement and cleanup, and flake.nix supplies flock. Previous review: #19 (review)

No new actionable bugs found. The single lock is an appropriately simple fix that preserves shared build caches. Base changes were separated from the benchmark delta.

Verification: static review of the changed lock, dependency and surrounding lifecycle only; no tests or benchmarks run.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging. Summary:

  • bench/ffi/ measures moq-ffi's wrappers against the fork: Go, Python, Kotlin, C++, Dart, Swift, and the browser over ubrn. It includes an A/A control, an optional --base, and records the host. It runs nightly and on PRs touching the fork. moq deletes its copy in chore(bench): move the binding benchmark to rust2 moq-dev/moq#6629.
  • Accepted by the maintainer: keep the ubrn PR-head pin, and keep the --base job on PRs touching uniffi/**.
  • Merged main after feat(ffi): exporter spike, a crate's pub API to C #41. The finished quest/m1/ffi-bench.md is gone, and the exporter quest now links /bench/ffi/README.md instead. The follow-ups stay in m2 next to the fork's other maintenance work: ubrn-release (waiting) and uniffi-symlink-path.
  • On e975390: just check, just test, quest check and Check all pass. I cancelled this push's build-bench and Bench FFI runs. The merge brought no change to bench/ffi or uniffi/, and the last full Bench FFI run passed on 9d7bbfb. Nightly reruns it.

(written by Opus 5.5)

@kixelated
kixelated merged commit 14fd23f into main Oct 25, 2026
1 of 7 checks passed
@kixelated
kixelated deleted the quest/m1/ffi-bench branch October 27, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant