diff --git a/.github/workflows/bench-ffi.yml b/.github/workflows/bench-ffi.yml new file mode 100644 index 000000000..de132c733 --- /dev/null +++ b/.github/workflows/bench-ffi.yml @@ -0,0 +1,68 @@ +name: Bench FFI + +permissions: + contents: read + +on: + schedule: + - cron: "0 8 * * *" + workflow_dispatch: + # A change to the fork measures its effect against the PR's base, and a change to + # the benchmark or its toolchains reports its numbers, before either merges. + pull_request: + paths: + - uniffi/** + - Cargo.toml + - Cargo.lock + - bench/ffi/** + - .github/workflows/bench-ffi.yml + - flake.nix + - flake.lock + - rust-toolchain.toml + +concurrency: + group: bench-ffi-${{ github.ref }} + cancel-in-progress: true + +jobs: + bench: + name: Bench FFI (${{ matrix.name }}) + runs-on: ${{ matrix.os }} + timeout-minutes: 330 + strategy: + fail-fast: false + # Each job measures its own Rust baselines, so ratios never cross machines. + matrix: + include: + - name: Linux + os: ubuntu-24.04 + - name: Swift + os: macos-latest + args: --runner swift + + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + # --base needs the PR's base commit. + fetch-depth: 0 + + - uses: ./.github/actions/nix + + - name: Bench + run: nix develop --command just bench-ffi ${{ matrix.args }} ${{ github.event_name == 'pull_request' && format('--base {0}', github.event.pull_request.base.sha) || '' }} --out bench-results + env: + # The pinned moq's flake names its binary cache; without it the dev shell builds from source. + NIX_CONFIG: accept-flake-config = true + + - name: Summary + run: cat bench-results/report.md >>"$GITHUB_STEP_SUMMARY" + + - name: Upload + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: bench-ffi-${{ strategy.job-index }} + path: bench-results + if-no-files-found: ignore diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index fe0c6b30e..19ccd1ca7 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -10,7 +10,7 @@ on: # A change to the benchmark or its toolchains reports its numbers before it merges. pull_request: paths: - - bench/** + - bench/build/** - .github/workflows/bench.yml - flake.nix - flake.lock diff --git a/bench/ffi/README.md b/bench/ffi/README.md new file mode 100644 index 000000000..e96dce2d6 --- /dev/null +++ b/bench/ffi/README.md @@ -0,0 +1,120 @@ +# Binding benchmark + +`just bench-ffi` measures what each of moq's binding wrappers adds over Rust, with moq-ffi at the revision pinned in [pins.toml](pins.toml) built against [the UniFFI fork](../../uniffi/README.md) in this tree. +It runs nightly in CI ([bench-ffi.yml](../../.github/workflows/bench-ffi.yml)), and every fork perf change states its effect with its numbers. + +```bash +just bench-ffi # every wrapper this host runs, three rounds +just bench-ffi --runner py --rounds 1 # a quick smoke test +just bench-ffi --base origin/main # a fork change against its base +``` + +Results land in `target/bench/ffi/`, newest at `target/bench/ffi/latest/report.md`. +Checkouts and builds live in `/var/tmp/rust2-bench-/ffi/`, with their own `CARGO_HOME` and none of the caller's cargo or rustc variables, as in the [build-speed benchmark](../build/README.md). +Runs on one host share that directory, so a second run waits for the first to finish. + +## What runs + +The native runners, their recipes, and the Rust baselines stay in moq beside the wrappers they measure (see moq's [bench/README.md](https://github.com/moq-dev/moq/blob/main/bench/README.md)); this harness drives them. +The browser runner lives here, since moq has no browser wrapper over moq-ffi yet. +Every runner runs the same in-process rounds with no network: each publisher writes one group while every subscriber reads it on its own task. +They sweep frame sizes from 100 B to 1 MiB at one publisher and subscriber, then 1 and 4 publishers by 1 and 4 subscribers at 1 KiB, and time two sync calls (`sequence`, `info`). +Each case reports delivered MB/s and process CPU per MB, and checks every received frame byte for byte. + +- `rust-ffi` calls moq-ffi's API from Rust without crossing the FFI, so a wrapper's ratio to it is what its boundary and wrapper cost. +- `rust-net` runs the same rounds on moq-net, so its ratio is what moq-ffi's handles and runtime cost. +- `go`, `py`, `kt`, `cpp`, `dart`, and `browser` run on Linux and macOS; `swift` runs on macOS only. +- `browser` builds moq-ffi for wasm32, generates its TypeScript with ubrn's `wasm2` player, and runs [browser-page.ts](browser-page.ts) in headless Chromium through [browser.ts](browser.ts). Its CPU is every thread of the renderer, read over CDP. No ubrn release reads UniFFI 0.32 yet, so [pins.toml](pins.toml) pins the head of [jhugman/uniffi-bindgen-react-native#468](https://github.com/jhugman/uniffi-bindgen-react-native/pull/468), and the run builds its CLI and packs its `@ubjs/core` and `@ubjs/wasm` runtimes from that source. + +The run enters the pinned moq's own dev shell, so every wrapper builds with the toolchains moq ships with. +moq's Go, Dart, and C++ scripts find their generators on `PATH`, where the run puts the fork's. +Its Kotlin, Python, and Swift generator and moq-ffi's runtime are the `uniffi` crate, which the run points at the fork's path in every manifest; the run fails if any UniFFI crate still resolves elsewhere. +For the browser, the run also links ubrn's `uniffi-runtime-wasm` helper crate into moq-ffi on wasm32, as ubrn's player requires. +Bump a pin in its own change, since numbers on either side of it do not compare. + +## Variants + +- `fork` builds against this tree. +- `control` reruns fork's builds, so its change against fork is the run's noise floor: an A/A control. +- `base`, with `--base REV`, builds against the fork at another rust2 revision, so fork's change against it is the effect of the change. + +An untimed warm-up round builds every runner once per checkout. +The timed rounds interleave the variants and rotate which goes first each round, so drift over a long job spreads across all of them. +The report shows each case's trimmed mean over the rounds (without the fastest and slowest) with its cost against `rust-ffi`, taken within each round so drift between rounds cancels, then each comparison as a change in that cost, so one slow round on a busy host does not move it. +`results.jsonl` keeps every round's value, mean, and standard deviation. +The host, its load average before and after, and every toolchain's version head the report. +No threshold is set: the numbers inform a decision, they do not gate one. + +## BoltFFI comparison + +Before forking UniFFI, moq compared it with [BoltFFI](https://github.com/boltffi/boltffi) 0.31.1 on a slice of moq-ffi bound by each generator, measured from Python and Kotlin with a Rust floor. +Its harness is deleted; [moq-dev/moq@0f986d9](https://github.com/moq-dev/moq/tree/0f986d9fdb7c074b0bd4b34739f7199c3bf2d9e5/bench/ffi-gen) holds it if a decision needs fresh numbers. +BoltFFI's numbers are the targets for the fork's backends: its speed comes from calling conventions a UniFFI fork can adopt one backend at a time (a CPython extension instead of `ctypes`, JNI instead of JNA, and bulk byte copies), not from its core. + +Measured 2026-10-16 on an Intel Core Ultra 7 270K Plus (24 threads), Linux, at moq `b4eab0db1`, on a shared host (load average 8 to 29), so cells are medians of 3 rounds. +Rust 1.98.1, CPython 3.14.7, OpenJDK 17 with Kotlin 2.4.20 and kotlinx-coroutines 1.11.0, JNA 5.19.1. +Gain is how many times better BoltFFI is than UniFFI. + +### Calls (ns per call) + +| call | Rust | Python UniFFI | Python BoltFFI | gain | Kotlin UniFFI | Kotlin BoltFFI | gain | +|---|---|---|---|---|---|---|---| +| `noop()` | 0.2 | 834 | 21.6 | 39x | 3,200 | 5.6 | 569x | +| `group.sequence()` | 0.4 | 1,920 | 63 | 30x | 7,440 | 5.8 | 1,290x | +| `track.info()`, a record return | 12 | 4,890 | 1,120 | 4.4x | 10,400 | 61 | 171x | +| `write_frame`, a 100 B record argument | 78 | 12,000 | 667 | 18x | 12,800 | 134 | 95x | +| `await noop_async()` | 0.3 | 7,190 | 5,330 | 1.3x | 8,320 | 342 | 24x | +| `await yield_once()`, one wakeup | 44 | 12,600 | 10,000 | 1.3x | 6,510 | 587 | 11x | + +In Python, both generators sit at 5 to 13 us per await, which points at asyncio rather than the boundary. + +### Throughput (MB/s, write plus async read) + +| frame | Rust | Python UniFFI | Python BoltFFI | gain | Kotlin UniFFI | Kotlin BoltFFI | gain | +|---|---|---|---|---|---|---|---| +| 100 B | 473 | 3.8 | 12.1 | 3.2x | 2.9 | 105 | 36x | +| 1 KiB | 3,340 | 36 | 117 | 3.3x | 36 | 954 | 27x | +| 16 KiB | 10,400 | 323 | 1,410 | 4.4x | 286 | 2,730 | 9.6x | +| 64 KiB | 8,930 | 531 | 2,840 | 5.4x | 448 | 2,760 | 6.2x | +| 1 MiB | 6,120 | 605 | 3,220 | 5.3x | 562 | 1,560 | 2.8x | + +### CPU (ms per MB) + +| frame | Rust | Python UniFFI | Python BoltFFI | gain | Kotlin UniFFI | Kotlin BoltFFI | gain | +|---|---|---|---|---|---|---|---| +| 100 B | 2.1 | 261 | 83 | 3.2x | 1,290 | 15 | 85x | +| 1 KiB | 0.30 | 28 | 8.6 | 3.3x | 76 | 1.2 | 61x | +| 16 KiB | 0.10 | 3.1 | 0.71 | 4.4x | 10 | 0.39 | 27x | +| 64 KiB | 0.11 | 1.9 | 0.35 | 5.4x | 3.8 | 0.37 | 10x | +| 1 MiB | 0.16 | 1.7 | 0.31 | 5.3x | 2.3 | 0.66 | 3.4x | + +At 100 B, Kotlin through UniFFI spends about 130 us of CPU per frame written and read, so 1,000 audio frames a second cost 13% of a core; BoltFFI spends 1.5 us. + +### Size + +Native libraries are `release` (fat LTO, `strip = "debuginfo"`, as moq-ffi ships); "stripped" is `strip --strip-all` on a copy. + +| native library | as built | stripped | +|---|---|---| +| slice, UniFFI | 17,571 KiB | 14,553 KiB | +| slice, BoltFFI | 17,472 KiB | 14,482 KiB | +| BoltFFI's Python extension (`_native`), extra | 240 KiB | 64 KiB | +| BoltFFI's Kotlin JNI glue, extra | 52 KiB | 38 KiB | +| moq-ffi, default features | 34,100 KiB | 29,396 KiB | +| moq-ffi, no default features | 30,625 KiB | 26,416 KiB | + +| generated bindings | files | raw | gzip | +|---|---|---|---| +| slice, Python: UniFFI / BoltFFI | 1 / 2 | 85 / 36 KiB | 12 / 7 KiB | +| slice, Kotlin: UniFFI / BoltFFI | 1 / 1 | 152 / 57 KiB | 17 / 9 KiB | +| slice, Swift: UniFFI / BoltFFI | 3 / 2 | 101 / 51 KiB | 14 / 8 KiB | +| moq-ffi, UniFFI: Python / Kotlin / Swift | 1 / 1 / 3 | 653 / 914 / 684 KiB | 66 / 73 / 72 KiB | + +Browser builds use opt-level z after wasm-bindgen and no wasm-opt, as moq-wasm ships; the stripped columns drop the `name` section. + +| browser module | .wasm as built | .wasm stripped | stripped, gzip | JS glue | JS glue, gzip | +|---|---|---|---|---|---| +| slice, UniFFI | 1,245 KiB | 515 KiB | 174 KiB | not generated | not generated | +| slice, BoltFFI | 1,148 KiB | 456 KiB | 155 KiB | 126 KiB | 25 KiB | +| moq-ffi, UniFFI | 6,218 KiB | 4,403 KiB | 1,575 KiB | not generated | not generated | +| moq-wasm, wasm-bindgen | 2,116 KiB | 2,115 KiB | 691 KiB | 38 KiB | 8 KiB | diff --git a/bench/ffi/browser-page.ts b/bench/ffi/browser-page.ts new file mode 100644 index 000000000..a0dd5acfa --- /dev/null +++ b/bench/ffi/browser-page.ts @@ -0,0 +1,161 @@ +/** + * The browser runner of the binding benchmark: moq-ffi bound by UniFFI over ubrn's + * `wasm2` player, running the same in-process rounds as moq's other runners. + * + * `browser.ts` bundles this page, serves it, and reads the renderer's CPU for it. + */ +import { + MoqBroadcastProducer, + MoqOriginConfig, + MoqOriginProducer, + MoqTrackInfo, + type MoqTrackConsumerLike, + uniffiInitAsync, +} from "./dist/index.ts"; + +interface Result { + name: string; + unit: string; + value: number; +} + +declare global { + interface Window { + /** Renderer CPU seconds, read by the driver over CDP. */ + rendererCpu(): Promise; + runBench(seconds: number): Promise; + } +} + +const WARMUP = 3; +// Each case's origin cache budget: far above one round's groups, so eviction only +// reclaims groups every subscriber has read, and memory stays flat over a long run. +const CACHE_BYTES = 64n << 20n; +// Far longer than any honest round, so only a lost wakeup trips it. +const ROUND_TIMEOUT_MS = 30_000; + +/** One throughput case: `publishers` tracks, each read by `subscribers` consumers. */ +class Case { + constructor( + readonly size: number, + readonly publishers: number, + readonly subscribers: number, + ) {} + + /** Frames per group, holding a group under moq-net's 32 MB group cache. */ + get frames(): number { + return Math.min(Math.max(Math.floor((8 << 20) / this.size), 1), 64); + } + + get delivered(): number { + return this.publishers * this.subscribers * this.frames * this.size; + } + + get name(): string { + return `size=${this.size}/pub=${this.publishers}/sub=${this.subscribers}`; + } +} + +// Frame sizes at one publisher and subscriber, then fan-out at 1 KiB. +const CASES = [ + ...[100, 1024, 64 * 1024, 1024 * 1024].map((size) => new Case(size, 1, 1)), + ...[ + [1, 4], + [4, 1], + [4, 4], + ].map(([p, s]) => new Case(1024, p, s)), +]; + +/** A non-uniform payload, so a zero-filled or truncated copy fails the check. */ +function payload(size: number): ArrayBuffer { + const bytes = new Uint8Array(size); + for (let i = 0; i < size; i++) bytes[i] = i % 251; + return bytes.buffer; +} + +function same(a: ArrayBuffer, b: ArrayBuffer): boolean { + if (a.byteLength !== b.byteLength) return false; + const x = new Uint8Array(a); + const y = new Uint8Array(b); + for (let i = 0; i < x.length; i++) if (x[i] !== y[i]) return false; + return true; +} + +async function read(c: Case, consumer: MoqTrackConsumerLike, want: ArrayBuffer, n: number): Promise { + const group = await consumer.recvGroup(); + if (!group) throw new Error("track ended"); + for (let f = 0; f < c.frames; f++) { + const frame = await group.readFrame(); + if (!frame) throw new Error("group ended"); + if (!same(frame.payload, want)) throw new Error(`corrupt frame ${f} of round ${n}`); + if (frame.timestampUs !== BigInt(n * c.frames + f)) throw new Error(`wrong timestamp on frame ${f} of round ${n}`); + } +} + +async function within(promise: Promise, ms: number, what: string): Promise { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + promise, + new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`${what} did not finish in ${ms} ms`)), ms); + }), + ]); + } finally { + clearTimeout(timer); + } +} + +async function throughput(c: Case, budget: number, results: Result[]): Promise { + const origin = new MoqOriginProducer(MoqOriginConfig.create({ cacheCapacityBytes: CACHE_BYTES })); + const broadcast = origin.createBroadcast("bench"); + const info = MoqTrackInfo.create({ timescale: 1_000_000n }); + const tracks = Array.from({ length: c.publishers }, (_, p) => broadcast.publishTrack(String(p), info)); + const consumers = tracks.flatMap((track) => Array.from({ length: c.subscribers }, () => track.consume(undefined))); + const bytes = payload(c.size); + + // Each consumer reads on its own task while this one writes each group in turn. + const step = async (n: number) => { + const readers = consumers.map((consumer) => read(c, consumer, bytes, n)); + for (const track of tracks) { + const group = track.appendGroup(); + for (let f = 0; f < c.frames; f++) group.writeFrame({ payload: bytes, timestampUs: BigInt(n * c.frames + f) }); + group.finish(); + } + await within(Promise.all(readers), ROUND_TIMEOUT_MS, `round ${n} of ${c.name}`); + }; + + for (let n = 0; n < WARMUP; n++) await step(n); + const cpuStart = await window.rendererCpu(); + const start = performance.now(); + let n = WARMUP; + while (performance.now() - start < budget * 1000) await step(n++); + const wall = (performance.now() - start) / 1000; + const cpu = (await window.rendererCpu()) - cpuStart; + const mb = ((n - WARMUP) * c.delivered) / 1e6; + results.push({ name: `throughput/${c.name}`, unit: "MB/s", value: mb / wall }); + results.push({ name: `cpu/${c.name}`, unit: "ms/MB", value: (cpu * 1e3) / mb }); +} + +function call(name: string, budget: number, fn: () => unknown, results: Result[]): void { + const start = performance.now(); + let calls = 0; + while (performance.now() - start < budget * 1000) { + for (let i = 0; i < 1000; i++) fn(); + calls += 1000; + } + results.push({ name: `call/${name}`, unit: "ns", value: ((performance.now() - start) * 1e6) / calls }); +} + +window.runBench = async (budget: number): Promise => { + await uniffiInitAsync(new URL("/moq_ffi.wasm", location.href)); + const results: Result[] = []; + for (const c of CASES) await throughput(c, budget, results); + + const track = new MoqBroadcastProducer().publishTrack("calls", undefined); + const group = track.appendGroup().consume(); + call("sequence", budget, () => group.sequence(), results); + const consumer = track.consume(undefined); + call("info", budget, () => consumer.info(), results); + return results; +}; diff --git a/bench/ffi/browser.sh b/bench/ffi/browser.sh new file mode 100755 index 000000000..24a4b8600 --- /dev/null +++ b/bench/ffi/browser.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# The browser runner: builds moq-ffi for wasm32 in a moq checkout, generates its +# TypeScript over ubrn's wasm2 player, and runs browser-page.ts in headless Chromium. +# Runs inside moq's dev shell, which carries cargo, wasm-bindgen, bun, and Chromium. +# Usage: browser.sh CHECKOUT DIR UBRN, with MOQ_BENCH_DIR and MOQ_BENCH_SECONDS set. +set -euo pipefail + +bench=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +checkout=$1 dir=$2 ubrn=$3 +mkdir -p "$dir" + +(cd "$checkout" && cargo build --locked -q -p moq-ffi --profile wasm-release --target wasm32-unknown-unknown) + +# ubrn wants a package.json and its config beside it. Playwright matches the shell's Chromium. +jq -n --arg pkgs "$ubrn/pkgs" --arg playwright "$PLAYWRIGHT_VERSION" '{ + name: "moq-ffi-bench", + private: true, + repository: {type: "git", url: "https://github.com/kixelated/rust2"}, + dependencies: { + "@ubjs/core": "file:\($pkgs)/ubjs-core.tgz", + "@ubjs/wasm": "file:\($pkgs)/ubjs-wasm.tgz", + playwright: $playwright + } +}' >"$dir/package.json" +cat >"$dir/ubrn.config.yaml" </dev/null +bun build page.ts --outfile bundle/page.js --target browser >/dev/null +cp dist/moq_ffi.wasm bundle/ +bun driver.ts bundle diff --git a/bench/ffi/browser.ts b/bench/ffi/browser.ts new file mode 100644 index 000000000..b0a3b51cc --- /dev/null +++ b/bench/ffi/browser.ts @@ -0,0 +1,56 @@ +/** + * Runs the browser runner (`browser-page.ts`) in headless Chromium and writes + * `$MOQ_BENCH_DIR/browser.json`, as moq's other runners write theirs. + * + * Usage: bun browser.ts DIST, where DIST holds the bundled `page.js` and `moq_ffi.wasm`. + */ +import { join } from "node:path"; +import { chromium } from "playwright"; + +const dist = process.argv[2]; +const out = process.env.MOQ_BENCH_DIR; +const seconds = Number(process.env.MOQ_BENCH_SECONDS ?? "1"); +if (!dist || !out) throw new Error("usage: MOQ_BENCH_DIR=... bun browser.ts DIST"); +if (!Number.isFinite(seconds) || seconds <= 0) throw new Error("MOQ_BENCH_SECONDS must be positive and finite"); + +const server = Bun.serve({ + hostname: "127.0.0.1", + port: 0, + async fetch(req) { + const path = new URL(req.url).pathname; + if (path === "/") + return new Response(``, { + headers: { "content-type": "text/html" }, + }); + const file = Bun.file(join(dist, path)); + return (await file.exists()) ? new Response(file) : new Response("not found", { status: 404 }); + }, +}); + +const browser = await chromium.launch({ channel: "chromium", headless: true }); +try { + const page = await browser.newPage(); + const errors: string[] = []; + page.on("pageerror", (error) => errors.push(error.message)); + page.on("console", (message) => { + if (message.text().includes("panicked at")) errors.push(`panic: ${message.text()}`); + }); + // Process CPU like the other runners' getrusage: every thread of the renderers, + // where the module, its runtime, and the page's garbage collection run. + const system = await browser.newBrowserCDPSession(); + await page.exposeFunction("rendererCpu", async () => { + const { processInfo } = (await system.send("SystemInfo.getProcessInfo")) as { + processInfo: { type: string; cpuTime: number }[]; + }; + return processInfo.filter((p) => p.type === "renderer").reduce((sum, p) => sum + p.cpuTime, 0); + }); + await page.goto(`http://127.0.0.1:${server.port}/`); + await page.waitForFunction(() => typeof window.runBench === "function"); + const results = await page.evaluate((budget) => window.runBench(budget), seconds); + if (errors.length) throw new Error(errors.join("; ")); + await Bun.write(join(out, "browser.json"), JSON.stringify(results, null, 2)); + console.log(`browser bench: wrote ${join(out, "browser.json")} (Chromium ${browser.version()})`); +} finally { + await browser.close(); + server.stop(true); +} diff --git a/bench/ffi/pins.toml b/bench/ffi/pins.toml new file mode 100644 index 000000000..985f12c27 --- /dev/null +++ b/bench/ffi/pins.toml @@ -0,0 +1,12 @@ +# What `just bench-ffi` builds. Changing a pin is a pin bump: land it on its own +# so numbers on either side of it are not compared. + +[moq] +git = "https://github.com/moq-dev/moq" +rev = "0f986d9fdb7c074b0bd4b34739f7199c3bf2d9e5" + +# The browser runner's generator and runtime. No release reads UniFFI 0.32 yet, +# so this is the head of jhugman/uniffi-bindgen-react-native#468, which does. +[ubrn] +git = "https://github.com/jhugman/uniffi-bindgen-react-native" +rev = "06f0a30d257573ffcc852e84e814670d53085979" diff --git a/bench/ffi/report.jq b/bench/ffi/report.jq new file mode 100644 index 000000000..513d8adca --- /dev/null +++ b/bench/ffi/report.jq @@ -0,0 +1,73 @@ +# Renders a run as Markdown. Inputs: $host (host.json) and $results +# (results.jsonl). Run with `jq -n -r`. + +def fixed(p): (. * pow(10; p) | round) / pow(10; p) | tostring; +def num: if . >= 100 then round | tostring elif . >= 10 then fixed(1) elif . >= 1 then fixed(2) else fixed(3) end; +def pct: (. * 100 | fixed(1)) as $p | if . > 0 then "+\($p)%" else "\($p)%" end; +def row: "| " + join(" | ") + " |"; +# How much more a result costs than another: time and CPU as they are, throughput inverted. +def cost($other): if .unit == "MB/s" then $other.trimmed / .trimmed else .trimmed / $other.trimmed end; +# Baselines first, then the wrappers in a stable order. +def runners: ["rust-net", "rust-ffi", "go", "python", "kotlin", "cpp", "dart", "swift", "browser"] as $known + | (map(.runner) | unique) as $seen + | ($known | map(select(. as $r | $seen | index($r)))) + ($seen - $known); +# In the order rust-ffi reports them; a runner may order its cases differently. +def cases: group_by(.name) + | map({name: .[0].name, unit: .[0].unit, order: ((map(select(.runner == "rust-ffi") | .order) | min) // 1e9)}) + | sort_by(.order); + +# One table of `$variant`, each cell drawn by `cell` from {result, by}. +def matrix($variant; cell): + map(select(.variant == $variant)) as $rows + | ($rows | map({key: "\(.runner)/\(.name)", value: .}) | from_entries) as $by + | ($rows | runners) as $columns + | (["case", "unit"] + $columns | row), + ("| --- | --- |" + ($columns | map(" ---: |") | join(""))), + ( + $rows | cases[] | . as $case + | [$case.name, $case.unit] + + ($columns | map($by["\(.)/\($case.name)"] as $r | if $r == null then "" else {result: $r, by: $by} | cell end)) + | row + ); + +# The change of each of `$variant`'s results against its base's. +def change($variant): + . as $all + | ($all | map(select(.variant == $variant)) | first | .base) as $base + | ($all | map(select(.variant == $base)) | map({key: "\(.runner)/\(.name)", value: .}) | from_entries) as $baseline + | $all | matrix($variant; .result as $r | $baseline["\($r.runner)/\($r.name)"] as $b | if $b == null then "" else ($r | cost($b)) - 1 | pct end); + +$host[0] as $h +| $results as $results +| [ + "# Binding benchmark", + "", + "\($h.date) on \($h.cpu) (\($h.cpus) CPUs, \($h.memory)), \($h.os), runner `\($h.runner)`; load average \($h.load_start) before, \($h.load_end) after.", + "rust2 `\($h.rust2[:12])`\(if $h.base == "" then "" else ", base `\($h.base[:12])`" end); moq `\($h.moq[:12])`\(if ($h.ubrn // "") == "" then "" else ", ubrn `\($h.ubrn[:12])`" end); \($h.rounds) round\(if $h.rounds == 1 then "" else "s" end) of \($h.seconds) s per case.", + "\([$h.tools[] | select(. != "missing")] | join("; ")).", + "", + "Each cell is the trimmed mean over the rounds (without the fastest and slowest), and the trimmed mean of its cost against `rust-ffi` in the same round: above 1x is slower or costlier.", + "`rust-ffi` calls moq-ffi's API from Rust without crossing the FFI; `rust-net` runs the same rounds on moq-net, below moq-ffi's handles and runtime.", + "", + "## fork", + "", + ($results | matrix("fork"; .result as $r | "\($r.trimmed | num)" + (if $r.ratio == null then "" else " (\($r.ratio | num)x)" end))), + "", + "## Noise: control against fork", + "", + "`control` reruns fork's builds in the same rounds, so its change is this run's noise floor.", + "Change in trimmed-mean cost (time per call, CPU per MB, or the inverse of throughput): negative is cheaper.", + "", + ($results | change("control")), + ( + if $h.base == "" then empty else + "", + "## Change: fork against base", + "", + "The change in cost from base to fork. Compare it with the noise floor above.", + "", + ($results | change("fork")) + end + ) +] +| .[] diff --git a/bench/ffi/run.sh b/bench/ffi/run.sh new file mode 100755 index 000000000..25bf6f8d3 --- /dev/null +++ b/bench/ffi/run.sh @@ -0,0 +1,382 @@ +#!/usr/bin/env bash +# Measures what each of moq's binding wrappers adds over Rust, with moq-ffi at +# the revision in pins.toml built against this tree's UniFFI fork. +# Usage and output are described in README.md. +set -euo pipefail + +bench=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P) +root=$(cd "$bench/../.." && pwd -P) + +die() { + echo "bench-ffi: $*" >&2 + exit 1 +} + +usage() { + cat <<'EOF' +Usage: just bench-ffi [OPTIONS] + +Options: + --runner NAME Measure only this wrapper (repeatable): go, py, kt, cpp, dart, + browser, or swift (macOS). Default: every one this host runs. + The Rust baselines always run. + --rounds N Timed rounds per variant (default: 3) + --seconds S Time budget per case and round (default: 1) + --base REV Also measure the fork at this rust2 revision, as `base` + --out DIR Write results to this new or empty directory + (default: target/bench/ffi/) +EOF +} + +# The result files each runner writes into MOQ_BENCH_DIR. +outputs() { + case $1 in + rust) echo rust-ffi rust-net ;; + py) echo python ;; + kt) echo kotlin ;; + *) echo "$1" ;; + esac +} + +# Summarizes a results directory's raw samples and renders its report.md. One +# sample per runner, case, variant, and round, then one record per runner, case, +# and variant. A wrapper's ratio to rust-ffi is taken within each round and +# variant, so drift between rounds cancels. Trimmed means, which drop each +# variant's fastest and slowest round, so one slow round on a busy host does not +# move a comparison. +report() { + local dir=$1 file rel + for file in "$dir"/raw/*/[0-9]*/*.json; do + rel=${file#"$dir/raw/"} + jq -c --arg variant "${rel%%/*}" --argjson round "$(cut -d/ -f2 <<<"$rel")" --arg runner "$(basename "$file" .json)" \ + 'to_entries[] | .value + {$variant, $round, $runner, order: .key} | {variant, round, runner, name, unit, value, order}' "$file" + done >"$dir/samples.jsonl" + jq -s -c --slurpfile host "$dir/host.json" ' + def mean: add / length; + def trimmed: sort | if length > 2 then .[1:-1] else . end | mean; + def worse($b): if .unit == "MB/s" then $b / .value else .value / $b end; + (map(select(.runner == "rust-ffi") | {key: "\(.variant)/\(.round)/\(.name)", value: .value}) | from_entries) as $rust + | map(. + {ratio: (if .runner == "rust-ffi" then null else worse($rust["\(.variant)/\(.round)/\(.name)"]) end)}) + | group_by([.variant, .runner, .name])[] + | map(.value) as $v + | ($v | mean) as $mean + | { + variant: .[0].variant, + base: ({control: "fork", fork: (if $host[0].base == "" then null else "base" end)}[.[0].variant]), + runner: .[0].runner, + name: .[0].name, + unit: .[0].unit, + trimmed: ($v | trimmed), + $mean, + stddev: (if ($v | length) > 1 then $v | map(. - $mean | . * .) | add / (length - 1) | sqrt else null end), + min: ($v | min), + max: ($v | max), + ratio: (if .[0].ratio == null then null else map(.ratio) | trimmed end), + values: $v, + order: (map(.order) | min) + }' "$dir/samples.jsonl" >"$dir/results.jsonl" + jq -n -r -f "$bench/report.jq" --slurpfile host "$dir/host.json" \ + --slurpfile results "$dir/results.jsonl" >"$dir/report.md" +} + +# Inside moq's dev shell: one untimed round per checkout, then the timed rounds, +# interleaving the variants and rotating which goes first each round. +measure() { + local out=$1 plan seconds rounds ubrn count i j variant checkout + local -a runners variants + plan=$out/plan.json + seconds=$(jq -r .seconds "$plan") + rounds=$(jq -r .rounds "$plan") + ubrn=$(jq -r .ubrn "$plan") + mapfile -t runners < <(jq -r '.runners[]' "$plan") + mapfile -t variants < <(jq -r '.variants[].name' "$plan") + count=${#variants[@]} + + jq -n \ + --arg rustc "$(rustc -V)" \ + --arg python "$(python3 --version)" \ + --arg go "$(go version 2>/dev/null || echo missing)" \ + --arg java "$(java -version 2>&1 | head -1 || echo missing)" \ + --arg dart "$(dart --version 2>&1 || echo missing)" \ + --arg cmake "$(cmake --version 2>/dev/null | head -1 || echo missing)" \ + --arg cxx "$(c++ --version 2>/dev/null | head -1 || echo missing)" \ + --arg swift "$(swift --version 2>/dev/null | head -1 || echo missing)" \ + --arg playwright "Playwright ${PLAYWRIGHT_VERSION:-missing}" \ + '$ARGS.named' >"$out/tools.json" + + # One round, timed or not, of every runner on one variant. + run_round() { # variant round + local variant=$1 round=$2 checkout generators browser dest runner log file + checkout=$(jq -r --arg n "$variant" '.variants[] | select(.name == $n) | .checkout' "$plan") + generators=$(jq -r --arg n "$variant" '.variants[] | select(.name == $n) | .generators' "$plan") + browser=$(jq -r --arg n "$variant" '.variants[] | select(.name == $n) | .browser' "$plan") + dest=$out/raw/$variant/$round + mkdir -p "$dest" + for runner in rust "${runners[@]}"; do + log=$out/logs/$variant.$round.$runner.log + echo "bench-ffi: $variant round $round $runner" + ( + cd "$checkout" + export PATH=$generators:$PATH MOQ_BENCH_DIR=$dest MOQ_BENCH_SECONDS=$seconds + # Each runner builds moq-ffi with its own features and environment; in a + # shared target directory they rebuild each other every round. Dart's + # script hardcodes the default one, and its host build scripts land in + # target/release, so the others leave that directory to it. + case $runner in + rust | go | py | kt | cpp) export CARGO_TARGET_DIR=$checkout/target/bench-$runner ;; + esac + case $runner in + # As `just rs test` runs it, minus its clippy gate: lints are not this run's to judge. + rust) bash sh/rs/nextest.sh cargo nextest run --locked --tests -p moq-ffi --cargo-profile release \ + -E 'test(bench::benchmark_ffi)' --run-ignored only ;; + go) just go bench -benchtime="${seconds}s" ;; + browser) "$bench/browser.sh" "$checkout" "$browser" "$ubrn" ;; + *) just "$runner" bench ;; + esac + ) >"$log" 2>&1 || { tail -40 "$log" >&2 && die "$runner failed on $variant; see $log"; } + for file in $(outputs "$runner"); do + [[ -s $dest/$file.json ]] || die "$runner wrote no $file.json on $variant; see $log" + done + done + } + + # The warm-up builds every runner once per checkout, so no build lands in a timed round. + local -A warmed=() + for variant in "${variants[@]}"; do + checkout=$(jq -r --arg n "$variant" '.variants[] | select(.name == $n) | .checkout' "$plan") + [[ -z ${warmed[$checkout]:-} ]] || continue + warmed[$checkout]=1 + run_round "$variant" warmup + done + for ((i = 0; i < rounds; i++)); do + for ((j = 0; j < count; j++)); do + run_round "${variants[(i + j) % count]}" "$i" + done + done +} + +case ${1:-} in + measure) + measure "${2:?measure needs a results directory}" + exit 0 + ;; + report) + report "${2:?report needs a results directory}" + exit 0 + ;; +esac + +all_runners=(go py kt cpp dart browser) +[[ $(uname -s) != Darwin ]] || all_runners+=(swift) +runners=() +rounds=3 +seconds=1 +base= +out= +while (($#)); do + case $1 in + --runner) runners+=("${2:?--runner needs a name}") && shift 2 ;; + --rounds) rounds=${2:?--rounds needs a count} && shift 2 ;; + --seconds) seconds=${2:?--seconds needs a duration} && shift 2 ;; + --base) base=${2:?--base needs a revision} && shift 2 ;; + --out) out=${2:?--out needs a directory} && shift 2 ;; + -h | --help) usage && exit 0 ;; + *) usage >&2 && die "unknown argument: $1" ;; + esac +done +[[ $rounds =~ ^[1-9][0-9]*$ ]] || die "--rounds must be a positive integer, got $rounds" +[[ $seconds =~ ^[0-9]*\.?[0-9]+$ && $seconds != 0 ]] || die "--seconds must be a positive number, got $seconds" +((${#runners[@]})) || runners=("${all_runners[@]}") +for runner in "${runners[@]}"; do + [[ " ${all_runners[*]} " == *" $runner "* ]] || die "unknown runner on this host: $runner" +done + +# Cargo reads every .cargo/config.toml above its working directory, and a user's +# (say, a target-cpu rustflag) would change the code measured. Build outside +# $HOME, with a private CARGO_HOME and none of the caller's cargo or rustc variables. +work=/var/tmp/rust2-bench-$(id -un)/ffi +results=$root/target/bench/ffi +out=${out:-$results/$(date -u +%Y%m%dT%H%M%SZ)} +[[ -z $(find "$out" -mindepth 1 -print -quit 2>/dev/null) ]] || die "$out is not empty; pass a new --out" +mkdir -p "$work" "$out/logs" +# Physical paths: on macOS /var is a symlink, and askama resolves its templates +# by a relative path that breaks across one. +work=$(cd "$work" && pwd -P) +# Every run shares the checkouts and builds under $work, and timings taken side +# by side would measure each other, so runs take turns. +exec 9>"$work/lock" +flock -n 9 || { echo "bench-ffi: waiting for another run to release $work" && flock 9; } +out=$(cd "$out" && pwd) +dir=$work +while :; do + for f in "$dir/.cargo/config" "$dir/.cargo/config.toml"; do + [[ ! -e $f ]] || die "$f would configure the builds; move it" + done + [[ $dir != / ]] || break + dir=$(dirname "$dir") +done +caller_vars=("${!CARGO@}" "${!RUSTC@}" "${!RUSTDOC@}" "${!RUSTUP@}") +[[ -z ${RUSTFLAGS+set} ]] || caller_vars+=(RUSTFLAGS) +if ((${#caller_vars[@]})); then + echo "bench-ffi: ignoring ${caller_vars[*]}" + unset "${caller_vars[@]}" +fi +export CARGO_HOME=$work/cargo-home + +moq_git=$(taplo get -f "$bench/pins.toml" moq.git) +moq_rev=$(taplo get -f "$bench/pins.toml" moq.rev) +ubrn_git=$(taplo get -f "$bench/pins.toml" ubrn.git) +ubrn_rev=$(taplo get -f "$bench/pins.toml" ubrn.rev) +fork_rev=$(git -C "$root" rev-parse HEAD) +[[ -z $(git -C "$root" status --porcelain -- uniffi Cargo.toml Cargo.lock) ]] || fork_rev+="-dirty" +base_rev= +if [[ -n $base ]]; then + base_rev=$(git -C "$root" rev-parse --verify "$base^{commit}") || die "unknown revision: $base" +fi + +# Builds the fork's Go, Dart, and C++ generators from a rust2 tree. moq's scripts +# find them on PATH; its own Kotlin, Python, and Swift generator builds from the +# redirected `uniffi` crate. +generators() { # tree dest + local tree=$1 dest=$2 bin + CARGO_TARGET_DIR=$dest/target cargo build --locked -q --manifest-path "$tree/Cargo.toml" \ + -p uniffi-bindgen-go -p uniffi-bindgen-cpp -p uniffi-dart --features uniffi-dart/binary --bins + mkdir -p "$dest/bin" + for bin in uniffi-bindgen-go uniffi-bindgen-cpp uniffi_bindgen_dart; do + ln -sfn "$dest/target/debug/$bin" "$dest/bin/$bin" + done +} + +# Clones a repository at a pinned revision, undoing anything an earlier run left +# in its tracked files. +clone() { # git rev dir + local url=$1 rev=$2 dir=$3 + if [[ ! -d $dir/.git ]]; then + git init -q "$dir" + git -C "$dir" remote add origin "$url" + fi + if [[ $(git -C "$dir" rev-parse -q --verify HEAD || true) != "$rev" ]]; then + git -C "$dir" fetch -q --depth 1 origin "$rev" + git -C "$dir" checkout -q --detach FETCH_HEAD + fi + git -C "$dir" reset -q --hard "$rev" + git -C "$dir" clean -qfd +} + +# Builds ubrn's CLI and packs its two TypeScript runtimes, @ubjs/core and @ubjs/wasm. +ubrn() { # dir + local dir=$1 + clone "$ubrn_git" "$ubrn_rev" "$dir/src" + [[ $(cat "$dir/stamp" 2>/dev/null) == "$ubrn_rev" ]] && return + rm -rf "${dir:?}/bin" "${dir:?}/pkgs" + cargo install --locked -q --root "$dir" --path "$dir/src/crates/ubrn_cli" + mkdir -p "$dir/pkgs" + (cd "$dir/src/typescript" && npm ci -s --no-audit --no-fund && npm run -s build && + npm pack -s --pack-destination "$dir/pkgs" >/dev/null) + (cd "$dir/src/runtimes/wasm" && npm ci -s --no-audit --no-fund && npm pack -s --pack-destination "$dir/pkgs" >/dev/null) + mv "$dir"/pkgs/ubjs-core-*.tgz "$dir/pkgs/ubjs-core.tgz" + mv "$dir"/pkgs/ubjs-wasm-*.tgz "$dir/pkgs/ubjs-wasm.tgz" + echo "$ubrn_rev" >"$dir/stamp" +} + +# Clones moq at the pin and points every `uniffi` dependency at the fork in `tree`. +checkout() { # tree dir + local tree=$1 dir=$2 strays + clone "$moq_git" "$moq_rev" "$dir" + # A crate keeps its upstream name but not its version, so `[patch]` cannot + # redirect it: rewrite each `uniffi` dependency's source to the fork's path. + git -C "$dir" ls-files -z '*Cargo.toml' | (cd "$dir" && xargs -0 sed -i -E \ + -e "s#^(uniffi = )\"[^\"]*\"\$#\\1{ path = \"$tree/uniffi/uniffi\" }#" \ + -e "s#^(uniffi = \\{ )(version = \"[^\"]*\"|git = \"[^\"]*\", (rev|tag|branch) = \"[^\"]*\")#\\1path = \"$tree/uniffi/uniffi\"#") + # Resolving without --locked updates the lockfile to the fork, so moq's own + # --locked builds accept it. Every uniffi crate must then be the fork's. + strays=$(cargo metadata --format-version 1 --manifest-path "$dir/Cargo.toml" | jq -r --arg fork "$tree/uniffi/" ' + .packages[] + | select(.name | test("^uniffi(_|$)")) + | select(.manifest_path | startswith($fork) | not) + | "\(.name) \(.version) from \(.source // .manifest_path)"') + [[ -z $strays ]] || die "moq still builds upstream UniFFI: $(tr '\n' ' ' <<<"$strays")" + # ubrn's player calls an allocator and panic hook its helper crate exports on wasm32. + if [[ -n $browser ]]; then + sed -i "/^uniffi = { workspace = true, features = \[\"wasm-unstable-single-threaded\"\] }\$/a uniffi-runtime-wasm = { path = \"$work/ubrn/src/runtimes/wasm/helper-crate\" }" \ + "$dir/rs/moq-ffi/Cargo.toml" + sed -i 's/^uniffi::setup_scaffolding!/#[cfg(target_arch = "wasm32")]\nextern crate uniffi_runtime_wasm as _;\n\n&/' "$dir/rs/moq-ffi/src/lib.rs" + if ! grep -q '^uniffi-runtime-wasm = ' "$dir/rs/moq-ffi/Cargo.toml" || + ! grep -q '^extern crate uniffi_runtime_wasm' "$dir/rs/moq-ffi/src/lib.rs"; then + die "could not link ubrn's runtime into moq-ffi; its manifest or crate root moved" + fi + cargo metadata --format-version 1 --manifest-path "$dir/Cargo.toml" >/dev/null + fi +} + +browser= +if [[ " ${runners[*]} " == *" browser "* ]]; then + browser=1 + echo "bench-ffi: preparing ubrn" + ubrn "$work/ubrn" +fi + +variants=() +add_variant() { # name tree + variants+=("$(jq -n -c --arg name "$1" --arg checkout "$work/moq/$1" --arg generators "$work/generators/$1/bin" \ + --arg browser "$work/browser/$1" '$ARGS.named')") + echo "bench-ffi: preparing $1" + generators "$2" "$work/generators/$1" + checkout "$2" "$work/moq/$1" +} +add_variant fork "$root" +# The A/A control reruns fork's builds, so its change is the run's noise floor. +variants+=("$(jq -c '.name = "control"' <<<"${variants[0]}")") +if [[ -n $base_rev ]]; then + tree=$work/rust2-base + git -C "$root" worktree remove --force "$tree" 2>/dev/null || rm -rf "$tree" + git -C "$root" worktree add -q --detach "$tree" "$base_rev" + trap 'git -C "$root" worktree remove --force "$tree"' EXIT + add_variant base "$tree" +fi + +jq -n \ + --argjson rounds "$rounds" \ + --arg seconds "$seconds" \ + --arg ubrn "$work/ubrn" \ + --args '{$rounds, $seconds, $ubrn, runners: $ARGS.positional}' "${runners[@]}" | + jq -c --slurpfile variants <(printf '%s\n' "${variants[@]}") '. + {$variants}' >"$out/plan.json" + +# Linux reads /proc; macOS has none and answers through sysctl. +if [[ -r /proc/cpuinfo ]]; then + cpu=$(sed -n 's/^model name[[:space:]]*: //p' /proc/cpuinfo | head -1) + memory=$(awk '/MemTotal/ { printf "%.0f GiB", $2 / 1048576 }' /proc/meminfo) +else + cpu=$(sysctl -n machdep.cpu.brand_string) + memory="$(($(sysctl -n hw.memsize) / 1073741824)) GiB" +fi +load() { + if [[ -r /proc/loadavg ]]; then cut -d' ' -f1-3 /proc/loadavg; else sysctl -n vm.loadavg | tr -d '{}' | xargs; fi +} +load_start=$(load) + +# The dev shell of the pinned moq carries every wrapper's toolchain. +nix develop "github:moq-dev/moq/$moq_rev" --command "$bench/run.sh" measure "$out" + +jq -n \ + --arg date "$(date -u +%FT%TZ)" \ + --arg os "$(uname -srm)" \ + --arg cpu "$cpu" \ + --arg cpus "$(getconf _NPROCESSORS_ONLN)" \ + --arg memory "$memory" \ + --arg runner "${RUNNER_NAME:-local}" \ + --arg load_start "$load_start" \ + --arg load_end "$(load)" \ + --arg rust2 "$fork_rev" \ + --arg base "$base_rev" \ + --arg moq "$moq_rev" \ + --arg ubrn "${browser:+$ubrn_rev}" \ + --argjson rounds "$rounds" \ + --arg seconds "$seconds" \ + --slurpfile tools "$out/tools.json" \ + '$ARGS.named + {tools: $tools[0]}' >"$out/host.json" + +report "$out" +mkdir -p "$results" +ln -sfn "$out" "$results/latest" +echo "bench-ffi: report at $out/report.md" diff --git a/flake.nix b/flake.nix index 83c4a5527..b158c405a 100644 --- a/flake.nix +++ b/flake.nix @@ -178,6 +178,8 @@ shellcheck shfmt taplo + # bench/ffi/run.sh serializes runs that share a work directory. + flock nixfmt # The TypeScript driver's tests run and type-check its output, and bundle a # caller with Bun to check that unused output shakes out. diff --git a/justfile b/justfile index 9e0f73c82..a7db08e53 100644 --- a/justfile +++ b/justfile @@ -17,8 +17,8 @@ default: check: quest check actionlint - shellcheck .claude/hooks/*.sh .codex/hooks/*.sh bench/build/*.sh - shfmt -d .claude/hooks/*.sh .codex/hooks/*.sh bench/build/*.sh + shellcheck .claude/hooks/*.sh .codex/hooks/*.sh bench/*/*.sh + shfmt -d .claude/hooks/*.sh .codex/hooks/*.sh bench/*/*.sh taplo format --check nixfmt --check flake.nix cargo fmt --all --check @@ -38,9 +38,13 @@ test: bench *args: bench/build/run.sh {{ args }} +# Measure what moq's binding wrappers add over Rust, against the UniFFI fork; see bench/ffi/README.md. +bench-ffi *args: + bench/ffi/run.sh {{ args }} + # Apply formatters without changing program behavior. fix: - shfmt -w .claude/hooks/*.sh .codex/hooks/*.sh bench/build/*.sh + shfmt -w .claude/hooks/*.sh .codex/hooks/*.sh bench/*/*.sh taplo format nixfmt flake.nix cargo fmt --all diff --git a/quest/README.md b/quest/README.md index 124d7d2f4..033c7db38 100644 --- a/quest/README.md +++ b/quest/README.md @@ -31,6 +31,6 @@ Decided with the maintainer on 2026-10-23, when rust2 was planned from moq ## Required - [m0: immediate priorities](/quest/m0/README.md) - urgent fixes -- [m1: consumers waiting](/quest/m1/README.md) - the UniFFI fork moq switches to, its binding benchmark, the exporter spike, and the fork's perf work +- [m1: consumers waiting](/quest/m1/README.md) - the UniFFI fork moq switches to, the exporter spike, and the fork's perf work - [m2: research](/quest/m2/README.md) - the TypeScript translator's spikes and features toward moq-net's model, and the fork's gated work - [m3: deferred](/quest/m3/README.md) - the pure-Rust profile and upstream's build-speed levers, measured by the benchmark diff --git a/quest/m1/README.md b/quest/m1/README.md index 9ef87679d..07676dc0c 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -11,11 +11,10 @@ Decided with the maintainer on 2026-10-25, from the [spike](https://github.com/kixelated/rust2/pull/41): the [exporter](/ffi/README.md), which turns a crate's `pub` API into bindings with no attributes, replaces the UniFFI fork. The fork stays in maintenance and keeps building moq's bindings until then; its feature quests -are folded into the [exporter's](/quest/m1/ffi-export.md). The binding benchmark continues -as the exporter's baseline. +are folded into the [exporter's](/quest/m1/ffi-export.md). The +[binding benchmark](/bench/ffi/README.md) continues as the exporter's baseline. ## Required - [Waiting: publish the UniFFI fork](/quest/m1/uniffi-release.md) - the fork publishes to crates.io under `rust2-*` names, once the maintainer decides it should -- [Binding benchmark](/quest/m1/ffi-bench.md) - moq's `bench-ffi` moves here and measures every wrapper against the fork, nightly - [Exporter](/quest/m1/ffi-export.md) - moq's bindings exported from its crates' `pub` API, replacing the fork diff --git a/quest/m1/ffi-bench.md b/quest/m1/ffi-bench.md deleted file mode 100644 index 6c2e167a2..000000000 --- a/quest/m1/ffi-bench.md +++ /dev/null @@ -1,28 +0,0 @@ -# [M] Move moq's binding benchmark to rust2 - -## Goal - -`just bench-ffi` runs here, nightly in CI: it builds moq-ffi at a pinned moq -revision against [the fork](/uniffi/README.md) and measures every wrapper -(Python, Go, Kotlin, Swift, Dart, C++, and the browser over ubrn) as moq's -`bench/` does today. Each [exporter](/quest/m1/ffi-export.md) generator states its effect against these -numbers. moq deletes its copy (`just bench-ffi`, `bench/ffi-gen`, and their -nightly jobs) in a paired PR. - -## Plan - -Decided with the maintainer on 2026-10-23: the custom FFI tooling and its -benchmarks leave moq, so the fork's perf work lives and measures here. -moq-ffi and its wrappers and packaging stay in moq; the benchmark drives them -at a pinned revision, as the [build-speed benchmark](/bench/build/README.md) pins -moq. A pin bump is its own change, so numbers compare across runs. - -- Build the pinned moq-ffi against the fork through a local patch or flake - override, so measuring a fork change never waits on moq's switch. -- Archive the BoltFFI comparison (`bench/ffi-gen`): keep its measured table in - this benchmark's README as the target, and delete its harness. Git history - holds it if a decision needs fresh numbers. -- Report variance with an A/A control and record the host, as the build-speed - benchmark does. - -Public API: none. diff --git a/quest/m1/ffi-export.md b/quest/m1/ffi-export.md index af834971d..6dcbfe077 100644 --- a/quest/m1/ffi-export.md +++ b/quest/m1/ffi-export.md @@ -28,7 +28,7 @@ census refuses and what moq waits on: - **A contract symbol** per build, checked when bindings load, so a header and a library from different builds refuse to mix. It replaces the fork's namespaced contract and per-method checksums, which the export never had. -- **Generators** on the C ABI, measured on the [binding benchmark](/quest/m1/ffi-bench.md): +- **Generators** on the C ABI, measured on the [binding benchmark](/bench/ffi/README.md): - C++ per standard (17, 20, 23) over the one C ABI, compiled with GCC, Clang, and MSVC, with and without exceptions and RTTI; OBS builds at C17 and C++17. - Kotlin through JNI or Panama rather than JNA: the fork measured 3.2 us per call against @@ -43,7 +43,3 @@ census refuses and what moq waits on: window, and pick a semantic for concurrent pollers. Go's cancel contract (moq's `go/wrapper/doc.go` ends the stream) is decided then. A hot read still makes a future handle per read; measure it before adding a per-handle poll. - -## Related - -- [Binding benchmark](/quest/m1/ffi-bench.md) - the fork's numbers, the baseline each generator is measured against diff --git a/quest/m2/README.md b/quest/m2/README.md index 6b44edb37..577a855ad 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -36,3 +36,5 @@ The fork's feature quests folded into the [exporter](/quest/m1/ffi-export.md); - [Tree-shakable runtime](/quest/m2/ts-runtime-shake.md) - `rt`'s adaptors and formatters become individually exported functions, kept only if no slower - [Translate moq-net's model to TypeScript](/quest/m2/ts-model.md) - the driver grows from moq-net's codec to its model, for moq to compare against WASM - [Nightly MSVC and macOS](/quest/m2/uniffi-ci-msvc-macos.md) - the C and C++ fixtures compile nightly with MSVC and Apple Clang, at C++17 and C++20, and without exceptions or RTTI +- [Symlinked paths](/quest/m2/uniffi-symlink-path.md) - the fork builds when a consumer reaches it through a symlink, as on macOS's `/var` +- [Waiting: ubrn reads UniFFI 0.32](/quest/m2/ubrn-release.md) - the binding benchmark's browser runner pins a ubrn release, not an open PR diff --git a/quest/m2/ubrn-release.md b/quest/m2/ubrn-release.md new file mode 100644 index 000000000..e919d4f46 --- /dev/null +++ b/quest/m2/ubrn-release.md @@ -0,0 +1,18 @@ +# [XS] Waiting: ubrn releases UniFFI 0.32 support + +## Goal + +uniffi-bindgen-react-native (ubrn) publishes a release whose `wasm2` flavour +reads UniFFI 0.32 metadata, so the [binding benchmark](/bench/ffi/README.md)'s +browser runner pins a release instead of an open PR's head. Check: +[jhugman/uniffi-bindgen-react-native#468](https://github.com/jhugman/uniffi-bindgen-react-native/pull/468) +or a successor has merged, and a tag after `0.31.0-6` carries it. + +## Plan + +Then bump `[ubrn]` in `bench/ffi/pins.toml` to the release in its own change, +and take `@ubjs/core`, `@ubjs/wasm`, and `uniffi-runtime-wasm` from npm and +crates.io instead of building them from source. moq waits on the same release +([moq-dev/moq `quest/m2/rs2ts/ubrn-032.md`](https://github.com/moq-dev/moq/blob/main/quest/m2/rs2ts/ubrn-032.md)). +If ubrn stalls, carrying its TypeScript generator in the fork is the +alternative; that is the maintainer's call. diff --git a/quest/m2/uniffi-symlink-path.md b/quest/m2/uniffi-symlink-path.md new file mode 100644 index 000000000..e528441c8 --- /dev/null +++ b/quest/m2/uniffi-symlink-path.md @@ -0,0 +1,16 @@ +# [S] The fork builds through a symlinked path + +## Goal + +A consumer that depends on [the fork](/uniffi/README.md) by a path through a +symlink (macOS's `/var` and `/tmp` are both) builds `uniffi_bindgen`. + +## Plan + +Seen on macOS CI of the [binding benchmark](/bench/ffi/README.md): with the +fork at `/var/tmp/.../uniffi`, askama's derive could not read `askama.toml`, +reaching for a relative path that climbs out of `/var/tmp/...` and back into +`/private/var/tmp/...`. The benchmark now builds from physical paths, so it no +longer hits this. Reproduce with a path dependency through a symlink, find +whether the mix comes from askama (0.16) or from how the fork names its config, +and fix it at that source.