Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions .github/workflows/codspeed.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: CodSpeed

# The binding benchmark under CodSpeed's CPU simulation (bench/ffi/README.md): each push to
# main is its baseline, and a PR that changes what `just ffi bench` reads gets CodSpeed's
# comment once a case changes. Informational: no threshold fails it, and no ruleset
# requires it.

permissions:
contents: read

on:
push:
branches: [main]
paths:
- ffi/**
# rust2-ffi builds over the driver by path.
- driver/**
- bench/ffi/**
- flake.nix
- flake.lock
- .github/workflows/codspeed.yml
- .github/actions/nix/**
# What ffi/ takes from ts/: the moq pin ts/justfile reads, and the nightly that
# ffi/rust-toolchain.toml links to.
- ts/justfile
- ts/fixtures/model/Cargo.toml
- ts/rust-toolchain.toml
- "!**/*.md"
pull_request:
paths:
- ffi/**
Comment thread
kixelated marked this conversation as resolved.
Resolved
- driver/**
- bench/ffi/**
- flake.nix
- flake.lock
- .github/workflows/codspeed.yml
- .github/actions/nix/**
- ts/justfile
- ts/fixtures/model/Cargo.toml
- ts/rust-toolchain.toml
- "!**/*.md"
workflow_dispatch:

# A PR's newer push supersedes its run. Each of main's runs is a baseline, so each takes its own
# group: a shared one would still replace a pending run with a newer one.
concurrency:
group: codspeed-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
codspeed:
name: CodSpeed
# Hosted, which costs a public repository nothing, as does CodSpeed.
runs-on: ubuntu-24.04
timeout-minutes: 180
permissions:
contents: read
# CodSpeed's upload authenticates through GitHub's OIDC token.
id-token: write

steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: ./.github/actions/nix

- name: Build
run: nix develop .#bench --command just ffi bench-build

# The bench shell's environment as a script the run sources, so Nix itself never runs
# under the simulation's Valgrind, which traces every process the run starts.
- name: Environment
run: nix print-dev-env .#bench > "$RUNNER_TEMP/bench.sh"

- name: Run
uses: CodSpeedHQ/action@af75f77f17a054769abefaa42fe8dc2975469b5f # v5.6.3
with:
mode: simulation
run: . "$RUNNER_TEMP/bench.sh" && just ffi codspeed
56 changes: 55 additions & 1 deletion bench/ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ from an emptied target and without a compiler cache, the export (`rust2-ffi`, wh
runs `cargo check` of moq under its driver) beside moq's own `cargo check` of the same
crates, and the generated crate's release build beside moq's own release build, each with
its peak RSS, through [`measure.py`](measure.py). Nightly runs it on arm64 Linux into
the run's summary; nothing fails on a number.
the run's summary; nothing fails on a number. [CodSpeed](#codspeed) counts each
single-threaded case on every push to main and on each PR that can move one.

| case | what it times | BoltFFI's nearest |
|---|---|---|
Expand Down Expand Up @@ -73,6 +74,59 @@ must not be ranked against the managed `loop` row. C, C++, and Rust's existing `
cases already manually poll in one loop; they remain the native floor, without printing
the same measurement twice under another name.

## CodSpeed

[CodSpeed](https://app.codspeed.io/kixel-ai/rust2) keeps main's history of each
single-threaded case, and comments on a PR whose run changes a case, with its change per
case.
[`codspeed.yml`](../../.github/workflows/codspeed.yml) runs on every push to main, its
baseline, and on each PR that changes what `just ffi bench` reads: it builds the programs in
the bench shell, then runs `just ffi codspeed` under CodSpeed's CPU simulation in that
shell's environment, so Nix itself never runs under Valgrind. It is informational: no
threshold fails it and no ruleset requires it.

```bash
nix develop .#bench --command just ffi bench-build
nix develop .#bench --command codspeed run --mode simulation -- just ffi codspeed # CodSpeed's CLI, on Ubuntu or Debian
```

Each program calls CodSpeed's
[instrument-hooks](https://github.com/CodSpeedHQ/instrument-hooks), which the flake pins,
through [`codspeed.h`](../../ffi/moq/bench/codspeed.h) and its own FFI around each case:
one benchmark per language and case, such as `python::write` or `c::throughput[1024]`.
Under CodSpeed a program runs its round once, without the warm-up round, and with a tenth
of the groups (so 100,000 calls a case, Python's 10,000), since the simulation counts
every call exactly and costs 50 to 100 times the program's own time. It skips the
`shared:*`, `own:*`, and Kotlin's `objects` cases: the simulation runs one thread at a
time, so it would count a contention that never happens. It counts instructions and
simulated cache misses, not time, so a system call or a wait costs nothing in it. The
tables below keep wall and CPU time, and a case's absolute count is for its whole loop,
not a call: read CodSpeed for each case's change.

A language joins only if its counts hold still: the same build, run twice, must move no
case far enough for CodSpeed to report a change, which its default 10% regression
threshold decides. `codspeed_langs` in [`ffi/justfile`](../../ffi/justfile) lists those
that pass. Measured 2026-11-14 on **nixos** with stock Valgrind 3.27.1 and the flags
CodSpeed's runner passes, without its fork's cycle estimation, in pairs of runs of one
build, the largest move of any case between two runs:

| language | largest move | |
|---|---|---|
| Rust, C, C++ | none: every count identical | joins |
| moq-c | 0.36% (`stamped`, with all the calls), 0.1% with a tenth | joins: moq's runtime thread runs beside it |
| Python | 0.21% (`throughput[1024]`) | joins: asyncio's poll of its selector depends on time |
| Swift | 52% (`throughput[16384]`), 8.8% in another pair | leaves: the global executor's threads interleave by time |
| Go | 362% (`stamped`), 330% (`record`), 84% (`loop`) | leaves: its collector and scheduler run inside each case, and its preemption signal crashes Callgrind unless `GODEBUG=asyncpreemptoff=1` |
| Kotlin | 172% (`call`), 108% (`getter`) | leaves: without the warm-up, the JIT's and collector's threads compile and collect inside each case, and the JVM's C2 code crashes under Valgrind unless its implicit null checks are off |
| Dart | does not run | leaves: the VM maps its code at an address hint off a page boundary, which Valgrind refuses |

On CI, two runs of one commit (kixel-ai/rust2#805's 11a94ff, a `pull_request` run and a
`workflow_dispatch` one) left all 58 benchmarks untouched in CodSpeed's comparison: the
largest move was Python's `throughput[1024]`, 2.4%, then its `throughput[100]` and
`ready`, under 2%, and every one of Rust's, C's, C++'s, and moq-c's rounded to 0%.

Each leaver keeps its hooks, so it rejoins by `codspeed_langs` once a pair of runs holds.

## Concurrent calls

Measured 2026-11-13 on **nixos**, an AMD Ryzen 9 5900XT (16 cores in two 8-core dies, 32
Expand Down
1 change: 1 addition & 0 deletions ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1045,6 +1045,7 @@ just ffi census # census each into target/census
just ffi shapes # round-trip every sampled type shape in every language but KNOWN's, as the nightly does
nix develop .#android --command just ffi android # the Kotlin test, shrunk by R8, on the Android device adb reaches
nix develop .#bench --command just ffi bench # the binding benchmark: bench/ffi/README.md's cases from Rust, and through the export from C, C++, Kotlin, Python, Swift, Go, and Dart
nix develop .#bench --command just ffi codspeed # after bench-build, each single-threaded case once under CodSpeed's CPU simulation, as the CodSpeed workflow runs it
nix develop .#sanitize --command just ffi sanitize # api's C and C++ scripts under ASan and TSan, and its C ABI under Miri
```

Expand Down
98 changes: 78 additions & 20 deletions ffi/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ moq_rev := "ffcb7cd9748422daa30e935ec464108b5a8ebb71"
# swift-collections 1.7.5, whose `UniqueArray` the generated Swift imports.
swift_collections_rev := "f96b909e1a9e469620bc5037c1e24e00153f376f"

# The binding benchmark's programs, each a language, as bench-program runs them.
bench_langs := "rust c cpp moq-c kotlin python swift go dart"

# Those whose cases CodSpeed counts the same on every run of one commit: bench/ffi/README.md's CodSpeed section says why Swift, Kotlin, Go, and Dart do not.
codspeed_langs := "rust c cpp moq-c python"

default:
@just --list

Expand Down Expand Up @@ -62,36 +68,64 @@ census checkout="": (standin checkout)
bench checkout="": (bench-build checkout) bench-run

# Build the binding benchmark's programs: Rust's floor, C, C++, Kotlin, Python, Swift, Go, and Dart through the export, and C through moq's hand-written rs/moq-c.
bench-build checkout="": (moq-export checkout)
bench-build checkout="": (moq-export checkout) codspeed-hooks
#!/usr/bin/env bash
set -euo pipefail
[ -n "${LIBCLANG_PATH:-}" ] || { echo "LIBCLANG_PATH is unset: run this in the bench shell, nix develop .#bench" >&2; exit 1; }
src="{{ checkout }}"
[ -n "$src" ] || src=target/moq
out=target/export/moq
hooks=$PWD/target/codspeed
# The exporter bench-run times, built as a consumer installs it.
cargo build --locked --quiet --release
# The floor: the stand-in's API called from Rust, resolved as the export is.
rm -rf target/bench-rust
mkdir -p target/bench-rust
printf '[package]\nname = "bench"\nversion = "0.0.0"\nedition = "2024"\npublish = false\n\n[[bin]]\nname = "bench"\npath = "%s"\n\n[dependencies]\nlibc = "0.2"\nmoq = { package = "moq-ffi", path = "../moq-ffi" }\n\n[workspace]\n' "$PWD/moq/bench/export.rs" > target/bench-rust/Cargo.toml
cp target/moq-ffi/Cargo.lock target/bench-rust/Cargo.lock
cargo build --quiet --release --manifest-path target/bench-rust/Cargo.toml
# The hooks' search path goes to the program alone, so moq builds as bench-run's export does.
cargo rustc --quiet --release --manifest-path target/bench-rust/Cargo.toml -- -L "native=$hooks"
# moq-c's header is written to its build script's output directory.
moqc=$(cd "$src" && cargo build --locked --quiet --release --package moq-c --message-format=json \
| jq -r 'select(.reason == "build-script-executed" and (.package_id | contains("moq-c#"))) | .out_dir')
cc -O2 -std=c11 -Wall -Wextra -Werror -I "$out/include" moq/bench/export.c "$out/target/release/libmoq.a" -lpthread -ldl -lm -o target/bench-export
cc -O2 -std=c11 -Wall -Wextra -Werror -I "$moqc/include" moq/bench/moq-c.c "$src/target/release/libmoq.a" -lpthread -ldl -lm -o target/bench-moq-c
c++ -O2 -std=c++17 -Wall -Wextra -Werror -I "$out/include" moq/bench/export.cpp "$out/target/release/libmoq.a" -lpthread -ldl -lm -o target/bench-export-cpp
cc -O2 -std=c11 -Wall -Wextra -Werror -I "$out/include" -I "$hooks" moq/bench/export.c "$out/target/release/libmoq.a" "$hooks/libcodspeed.a" -lpthread -ldl -lm -o target/bench-export
cc -O2 -std=c11 -Wall -Wextra -Werror -I "$moqc/include" -I "$hooks" moq/bench/moq-c.c "$src/target/release/libmoq.a" "$hooks/libcodspeed.a" -lpthread -ldl -lm -o target/bench-moq-c
c++ -O2 -std=c++17 -Wall -Wextra -Werror -I "$out/include" -I "$hooks" moq/bench/export.cpp "$out/target/release/libmoq.a" "$hooks/libcodspeed.a" -lpthread -ldl -lm -o target/bench-export-cpp
{{ quote(just_executable()) }} go-module bench-go moq/bench/export.go
read -ra native < target/moq-native
export CGO_LDFLAGS="$PWD/$out/target/release/libmoq.a ${native[*]}" GOFLAGS=-buildvcs=false
# The include path goes in CGO_CPPFLAGS, so cgo keeps its default -O2 -g in CGO_CFLAGS.
export CGO_CPPFLAGS="-I$hooks" CGO_LDFLAGS="$PWD/$out/target/release/libmoq.a $hooks/libcodspeed.a ${native[*]}" GOFLAGS=-buildvcs=false
(cd target/bench-go && go build -o bench .)
{{ quote(just_executable()) }} dart-program bench-dart moq/bench/export.dart
rm -rf target/bench-kt
kotlinc -Werror -cp "$KOTLINX_COROUTINES" "$out/kotlin/moq.kt" moq/bench/export.kt -d target/bench-kt
# moq's Kotlin outgrows the 512 MiB heap kotlinc's launcher defaults to.
JAVA_OPTS="-Xmx2g -Xms128M" kotlinc -Werror -cp "$KOTLINX_COROUTINES" "$out/kotlin/moq.kt" moq/bench/export.kt -d target/bench-kt
{{ quote(just_executable()) }} swift-module -O
swiftc -O -warnings-as-errors $({{ quote(just_executable()) }} swift-target 15) -I target/swift -I target/swift-collections/build -I "$out/swift" moq/bench/export.swift target/swift/libMoq.a -L target/swift-collections/build -lBasicContainers -lInternalCollectionsUtilities -lSpanPreview "$out/target/release/libmoq.a" -lpthread -ldl -lm -o target/bench-export-swift
swiftc -O -warnings-as-errors $({{ quote(just_executable()) }} swift-target 15) -I target/swift -I target/swift-collections/build -I "$out/swift" -I "$hooks" moq/bench/export.swift target/swift/libMoq.a -L target/swift-collections/build -lBasicContainers -lInternalCollectionsUtilities -lSpanPreview "$out/target/release/libmoq.a" "$hooks/libcodspeed.a" -lpthread -ldl -lm -o target/bench-export-swift

# Build moq/bench/codspeed.c over the bench shell's instrument-hooks into target/codspeed: a static library the compiled programs link, a shared one with the JNI functions that Kotlin, Python, and Dart load, and a module map Swift imports it by.
[private]
codspeed-hooks:
#!/usr/bin/env bash
set -euo pipefail
[ -n "${CODSPEED_INSTRUMENT_HOOKS:-}" ] || { echo "CODSPEED_INSTRUMENT_HOOKS is unset: run this in the bench shell, nix develop .#bench" >&2; exit 1; }
src=$CODSPEED_INSTRUMENT_HOOKS
dir=target/codspeed
rm -rf "$dir"
mkdir -p "$dir"
case "{{ os() }}" in
macos) lib=libcodspeed.dylib jni=darwin ;;
*) lib=libcodspeed.so jni=linux ;;
esac
# instrument-hooks' generated C warns, harmlessly, its guide says, and puts C23's
# attributes where C23 refuses them.
cc -O2 -fPIC -std=gnu11 -w -I "$src/includes" -c "$src/dist/core.c" -o "$dir/core.o"
flags=(-O2 -fPIC -std=c11 -Wall -Wextra -Werror -isystem "$src/includes")
cc "${flags[@]}" -c moq/bench/codspeed.c -o "$dir/codspeed.o"
ar rcs "$dir/libcodspeed.a" "$dir/codspeed.o" "$dir/core.o"
cc "${flags[@]}" -shared -DCODSPEED_JNI -isystem "${JAVA_HOME:?}/include" -isystem "$JAVA_HOME/include/$jni" moq/bench/codspeed.c "$dir/core.o" -o "$dir/$lib"
cp moq/bench/codspeed.h "$dir/"
printf 'module CodSpeed {\n header "codspeed.h"\n export *\n}\n' > "$dir/module.modulemap"

# Run what bench-build built: each program `rounds` times into target/bench/raw.tsv, and the export and its crate's build, beside moq's, each round into target/bench/build.tsv, with the 1-minute load average over each run, and their medians and the export's sizes into target/bench/report.md.
bench-run rounds="3":
Expand All @@ -113,9 +147,8 @@ bench-run rounds="3":
}
run() {
local lang=$1 before after
shift
before=$(load)
"$@" > target/bench/run.tsv
{{ quote(just_executable()) }} bench-program "$lang" > target/bench/run.tsv
after=$(load)
awk -v lang="$lang" -v round="$r" -v loadavg="$(echo "$before $after" | awk '{ print ($1 + $2) / 2 }')" -F '\t' \
'NF == 3 { print lang "\t" round "\t" $1 "\t" $2 "\t" $3 "\t" loadavg; next } { print "bench: " lang ": unexpected line: " $0 > "/dev/stderr"; exit 1 }' \
Expand Down Expand Up @@ -150,19 +183,44 @@ bench-run rounds="3":
}
for r in $(seq {{ rounds }}); do
build
run rust target/bench-rust/target/release/bench
run c target/bench-export
run cpp target/bench-export-cpp
run moq-c target/bench-moq-c
run kotlin kotlin -Djava.library.path="$out/target/release" -cp "target/bench-kt:$KOTLINX_COROUTINES" ExportKt
PYTHONPATH="$out/python" run python python3 -P moq/bench/export.py
run swift target/bench-export-swift
run go target/bench-go/bench
run dart dart target/bench-dart/main.dill
for lang in {{ bench_langs }}; do
run "$lang"
done
done
python3 -I ../bench/ffi/report.py "$raw" "$builds" "$out" > target/bench/report.md
cat target/bench/report.md

# Run what bench-build built once each, under `codspeed run` in the bench shell's environment, so CodSpeed's CPU simulation counts each single-threaded case: the CodSpeed workflow's run.
codspeed:
Comment thread
kixelated marked this conversation as resolved.
Resolved
#!/usr/bin/env bash
set -euo pipefail
for lang in {{ codspeed_langs }}; do
{{ quote(just_executable()) }} bench-program "$lang"
done

# Run what bench-build built for one of bench_langs.
[private]
bench-program lang:
#!/usr/bin/env bash
set -euo pipefail
out=target/export/moq
case "{{ os() }}" in
macos) hooks=$PWD/target/codspeed/libcodspeed.dylib ;;
*) hooks=$PWD/target/codspeed/libcodspeed.so ;;
esac
case "{{ lang }}" in
rust) exec target/bench-rust/target/release/bench ;;
c) exec target/bench-export ;;
cpp) exec target/bench-export-cpp ;;
moq-c) exec target/bench-moq-c ;;
kotlin) exec kotlin -Djava.library.path="$out/target/release:$PWD/target/codspeed" -cp "target/bench-kt:$KOTLINX_COROUTINES" ExportKt ;;
python) PYTHONPATH="$out/python" exec python3 -P moq/bench/export.py "$hooks" ;;
swift) exec target/bench-export-swift ;;
go) exec target/bench-go/bench ;;
dart) exec dart target/bench-dart/main.dill "$hooks" ;;
*) echo "bench-program: no language {{ lang }}" >&2; exit 1 ;;
esac

# Export moq's crates' C ABI, build it, and run moq/moq.c, moq/moq.cpp, moq/moq.kt, moq/moq.py, moq/moq.swift, moq/moq.go, and moq/moq.dart against it, from the pinned moq revision or a checkout.
moq checkout="": (moq-cpp checkout)
#!/usr/bin/env bash
Expand All @@ -173,7 +231,7 @@ moq checkout="": (moq-cpp checkout)
target/moq-test
# The Kotlin, on the JVM, through the JNI functions the kotlin feature builds.
rm -rf target/moq-kt
kotlinc -Werror -cp "$KOTLINX_COROUTINES" "$out/kotlin/moq.kt" moq/moq.kt -d target/moq-kt
JAVA_OPTS="-Xmx2g -Xms128M" kotlinc -Werror -cp "$KOTLINX_COROUTINES" "$out/kotlin/moq.kt" moq/moq.kt -d target/moq-kt
kotlin -Djava.library.path="$out/target/release" -cp "target/moq-kt:$KOTLINX_COROUTINES" MoqKt
# The Python, through the CPython extension the python feature builds, and its types
# through mypy and pyright, as `api_py_types` checks the api fixture's.
Expand Down
Loading