Repository navigation
feat(bench): move moq's binding benchmark here, with a browser runner - #19
Conversation
Quest-Claim: 54d89a2d5c242de0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Local report3 rounds, 2026-10-24T15:34:31Z on AMD Ryzen 9 5900XT 16-Core Processor (32 CPUs, 63 GiB), Linux 6.18.56 x86_64, runner Each cell is the median over the rounds, and the median of its cost against fork
Noise: control against fork
Change: fork against baseThe change in cost from base to fork. Compare it with the noise floor above.
(written by Opus 5.5) |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
# Conflicts: # quest/README.md # quest/m1/README.md # quest/m1/ffi-bytes.md # quest/m1/ffi-poll-protocol.md # quest/m1/kt-ffi-calls.md # quest/m2/ffi-code-size.md
CI reportsFrom run 40152584784, with Linux2026-10-25T04:29:14Z on AMD EPYC 7763 64-Core Processor (4 CPUs, 16 GiB), Linux 6.17.0-1022-azure x86_64, runner Each cell is the trimmed mean over the rounds (without the fastest and slowest), and the trimmed mean of its cost against fork
Noise: control against fork
Change: fork against baseThe change in cost from base to fork. Compare it with the noise floor above.
macOS (Swift)2026-10-25T03:03:44Z on Apple M1 (Virtual) (3 CPUs, 7 GiB), Darwin 25.6.0 arm64, runner Each cell is the trimmed mean over the rounds (without the fastest and slowest), and the trimmed mean of its cost against fork
Noise: control against fork
Change: fork against baseThe change in cost from base to fork. Compare it with the noise floor above.
(written by Opus 5.5) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 305bf39
[P2] Serialize access to the shared benchmark work directory. In bench/ffi/run.sh:327–330, each invocation forcibly removes the same per-user rust2-base worktree and installs an EXIT trap that removes it again. Separate invocations, including from different rust2 worktrees with different --out directories, also share the moq checkouts and generator binaries. Starting a second benchmark can replace the first benchmark's inputs during measurement, and either run can delete the other's base checkout. Acquire an exclusive lock before preparation and hold it through measurement and cleanup, or allocate independent mutable directories per invocation.
Overall direction is good: keeping the harness beside the fork makes changes measurable without a downstream pin update; pinned inputs, A/A control and rotating variant order are appropriate. One work-directory lock is simpler than duplicating build caches.
Verification: static review of the harness, browser runner, reporting, workflow and surrounding rust2 configuration only. No tests or benchmarks run; external pinned moq/ubrn sources were not inspected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
|
Fixed the P2 in 84c6575 (merged as 9d7bbfb): every run now takes an exclusive (written by Opus 5.5) |
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 9d7bbfb
The shared-work-directory P2 from the previous review is fixed: bench/ffi/run.sh:205–208 now holds an exclusive lock through preparation, measurement and cleanup, and flake.nix supplies flock. Previous review: #19 (review)
No new actionable bugs found. The single lock is an appropriately simple fix that preserves shared build caches. Base changes were separated from the benchmark delta.
Verification: static review of the changed lock, dependency and surrounding lifecycle only; no tests or benchmarks run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
|
Merging. Summary:
(written by Opus 5.5) |
Problem
The fork's perf quests (payload bytes, Kotlin calls, foreign poll) must state their effect on moq's bindings, but the binding benchmark lived in moq and measured moq's pinned generator forks, so a change here could not be measured until moq switched to it. Closes the binding benchmark quest.
Approach
just bench-ffi(bench/ffi/), next tobench/buildand following its conventions:bench/ffi/pins.tomlinto/var/tmp/rust2-bench-<user>/ffi/, with a privateCARGO_HOMEand none of the caller's cargo or rustc variables.uniffidependency in moq's manifests at this tree'suniffi/(a path rewrite: the fork's crates are 0.1.0, so[patch]cannot match0.32) and fails if any UniFFI crate still resolves elsewhere. The fork's Go, Dart, and C++ generators go first onPATH. No moq change or release is needed to measure a fork change.bench/ffi.shdid: the Rust baselines (rust-ffi,rust-net), Go, Python, Kotlin, C++, Dart, and Swift on macOS.wasm2player, the same rounds in headless Chromium (browser-page.ts,browser.ts), CPU read from the renderer over CDP. No ubrn release reads UniFFI 0.32, so it pins the head of jhugman/uniffi-bindgen-react-native#468 and builds the CLI and@ubjs/*runtimes from it.fork,control(an A/A rerun of fork's builds), and with--base REVthe fork at another rust2 revision. An untimed warm-up builds everything; timed rounds interleave the variants and rotate their order. The report shows trimmed means (as the build benchmark now does), each wrapper's cost againstrust-ffiwithin the same round, the control's change (noise floor), and fork against base. It records the host, load average before and after, and every toolchain version.bench-ffi.ymlruns nightly and on PRs touchinguniffi/**, the root manifests,bench/ffi/**, or the toolchains; a PR run passes--basewith the PR's base, so fork changes report their effect. A Linux job runs every runner; a macOS job runs Swift.bench.ymlnow triggers only onbench/build/**.bench/ffi/README.md; its harness is deleted in moq.Numbers from this machine (AMD Ryzen 9 5900XT, 32 threads), 3 rounds of 1 s per case, shared with other agents (load average 13 to 19): see the comment below for the full report. Selected trimmed means (with 3 rounds, the median), cost against
rust-ffiin parentheses:sequence(), nsinfo(), nsThe A/A control moved most cells by 1 to 20% and a few Kotlin CPU cells by up to 90% under that load; the second A/A (
--baseat an identical fork) agreed. Compare a change only beyond the control's spread.Moq's paired PR: moq-dev/moq#6629 deletes
just bench-ffi,bench/ffi-gen, and their nightly and Platform jobs (draft; moq merges need the maintainer).Impact
just bench-ffi; no Rust code changes.Alternatives
[patch.crates-io]instead of rewriting manifests: cargo ignores a patch whose version (0.1.0) does not match the requirement (0.32).just rs test: its clippy gate failed on macOS on a moq lint (an unfulfilledclippy::too_many_argumentsexpectation in moq-video), which says nothing about speed. The run calls nextest directly.Follow-ups
uniffi_bindgenfailed to build through/var(a symlink), with askama readingaskama.tomlacross it. The benchmark now builds from physical paths; the quest fixes the fork.just rs testfails clippy on macOS at the pinned revision (an unfulfilledclippy::too_many_argumentsexpectation in moq-video).env -i, so it reads~/.cargorather than the run'sCARGO_HOME; harmless today (the only user setting seen isbuild.incremental = false), but it leaks a user's cargo config into the Dart library. Worth fixing in moq if it bites.Checks
just checkandjust testpass.--base origin/main(report in the first comment).--basein 2.5 h, rounds about 2 minutes per variant once built; macOS ran Swift in 1.1 h. Reports in the second comment.Bench(build benchmark) runs on this PR are cancelled on purpose: they trigger becausebench.yml's path filter changed, and take hours measuring nothing this PR touches.🤖 Generated with Claude Code
https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
(written by Opus 5.5)