Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ jobs:
ffi-android:
name: Export to Kotlin on Android
runs-on: ubuntu-24.04
timeout-minutes: 60
timeout-minutes: 90

steps:
- name: Checkout
Expand Down Expand Up @@ -423,6 +423,20 @@ jobs:
test "$($ANDROID_HOME/platform-tools/adb shell getconf PAGE_SIZE | tr -d '\r')" = 16384
nix develop .#android --command just ffi android

# The binding benchmark's Kotlin rows on ART beside HotSpot's, into the run's summary:
# no threshold fails the step. Four cores, for its threaded cases. Last, so a failed
# measurement can't skip a correctness gate above.
- name: Bench on API 35
uses: reactivecircus/android-emulator-runner@a421e43855164a8197daf9d8d40fe71c6996bb0d # v2.38.0
with:
api-level: 35
arch: x86_64
cores: 4
script: nix develop .#android --command just ffi bench-android

- name: Summarize the bench
run: cat ffi/target/bench-android/report.md >> "$GITHUB_STEP_SUMMARY"

ffi-bench:
name: Bench the bindings
runs-on: ubuntu-24.04-arm
Expand Down
77 changes: 74 additions & 3 deletions bench/ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ stand-in `just ffi bench` exports, against BoltFFI's numbers below as targets.
```bash
nix develop .#bench --command just ffi bench # build, run 3 rounds, and write ffi/target/bench/report.md
nix develop .#bench --command just ffi bench-run 5 # rerun what bench-build built, 5 rounds
nix develop .#android --command just ffi bench-android # the Kotlin rows on HotSpot and on the Android device adb reaches
```

`ffi/moq/bench/export.*` time the same cases in each language, `export.rs` the floor
Expand All @@ -32,7 +33,7 @@ single-threaded case on every push to main and on each PR that can move one.
| shared getter, own getter | the u64 getter the same way | none |
| record return | `track.largest()`: an `Option` of two `u64`s, copied across | `track.info()`, a record |
| object return | `track.info()`: a new handle, freed | none |
| field | `info.priority`: an `i8` `pub` field read, which no user code reaches, so a leaf in C, Python, and Dart alone | none |
| field | `info.priority`: an `i8` `pub` field read, which no user code reaches, so a leaf in C, Kotlin, Python, and Dart alone | none |
| hop call, hop field | Dart's handle call and field read while Rust holds a closure of the isolate | none |
| write | `group.write_frame(at, frame)` with one `Timestamp` a group and 100 B, copied into Rust | `write_frame` |
| stamped write | the same with a new `Timestamp` per frame, as a caller holding a `u64` must | none |
Expand Down Expand Up @@ -219,8 +220,9 @@ GIL, which dwarfs the handle's own contention.
A leaf is a call the export proves runs no code but std's and the glue's, waits on no
lock, and calls no binding: a primitive `pub` field of a `Send` object reached without a
`Deref`, or a `PartialEq` or `Hash` derived down to primitives ([`Abi::leaf`](../../ffi/src/abi.rs)).
Python keeps the GIL across one, and Dart makes it a leaf call (`isLeaf`) to the C function
itself on the isolate's thread, with no serving thunk, no thread hop, and no `_Arena`.
Python keeps the GIL across one, Dart makes it a leaf call (`isLeaf`) to the C function
itself on the isolate's thread, with no serving thunk, no thread hop, and no `_Arena`, and
Kotlin marks its native method `@FastNative` ([Kotlin on ART](#kotlin-on-art)).
The benchmark's u64 getter, `demand.sequence()`, is a method, which the proof cannot see
into, so the `field` case reads `track::Info::priority` instead; `group.sequence` is a
field too, but reached through `group::Producer`'s `Deref`, which is user code.
Expand Down Expand Up @@ -263,6 +265,75 @@ round trip, as `rt::call` does more around it: it may look for the event loop ru
after the call drains what other threads freed and resumes what the call woke, all of which
a leaf skips. A getter that is no leaf, such as `demand.sequence()`, still pays it.

## Kotlin on ART

`just ffi bench-android` runs `export.kt` on HotSpot on the host and, shrunk by R8 with the
generated rules as an app is, on ART through `dalvikvm` on the Android device adb reaches,
over a library cargo-ndk builds for it, round after round, into one report. Nightly runs it
in the Android job, on its x86_64 API 35 emulator with 4 cores, into the run's summary; the
other languages' rows run on arm64, so read Kotlin's on HotSpot beside ART's from this
report alone. ART has no `java.lang.management`, so its CPU time is the kernel's tick count
in `/proc/self/stat`, at HotSpot's own 10 ms resolution on Linux.

Measured 2026-11-20 on **fedora**, an AMD Ryzen 9 9950X, through `quest gate`, against moq
`1aa86d5e`, with this change, 3 rounds at a load of 6 to 8, on an Android 15 (API 35)
x86_64 emulator with 4 cores under KVM. OpenJDK 21.0.13 with Kotlin 2.4.21, R8 8.6 from
build-tools 35.0.0. Wall ns per call, the median; the CPU cells are coarse at this load's
tick:

| call | HotSpot | ART |
|---|---|---|
| handle call | 32 | 41 |
| u64 getter | 23 | 32 |
| field, a leaf | 23 | 25 |
| record return | 56 | 129 |
| object return | 97 | 230 |
| write, 100 B | 224 | 382 |
| ready read | 630 | 1,190 |
| woken read | 1,761 | 3,751 |
| `frame.payload`, 100 B | 64 | 143 |
| throughput, 16 KiB frames | 9,268 MB/s | 652 MB/s |

1. **A plain JNI call costs ART about 9 ns more than HotSpot.** The handle call and the
getter run the same Kotlin and the same Rust on both.
2. **A leaf is `@FastNative`, which keeps ART's thread runnable across the call instead of
switching its state there and back.** Each leaf's native method takes it, as
`_Native.track_Info_priority_get` does, and `_Native`'s first call registers each through
`RegisterNatives` from the library's one exported `__contract`, since ART binds such a
method by its name only from Android 12. The annotation blocks the collector for the
call, so only a leaf takes it: a call that may wait on a lock, as `is_closed()` waits on
moq's, or call back into Kotlin, could stall every thread's collection behind it.
3. **Throughput falls off from 16 KiB on ART alone**, near 650 MB/s against HotSpot's 6.7 to
9.3 GB/s, while 100 B and 1 KiB frames stay within 3x.
[ART's throughput](/quest/m1/ffi-speed/art-throughput.md) finds why.

The spike, 5 rounds of each side alternating on one emulator at a load of 12 to 29, wall ns,
before (main) and after (`@FastNative` on leaves, registered):

| case | API 35 before | API 35 after | API 30 before | API 30 after |
|---|---|---|---|---|
| field, a leaf | 34.1 | 25.8 | 43.5 | 28.1 |
| handle call, no leaf | 40.3 | 43.6 | 80.3 | 80.8 |
| u64 getter, no leaf | 32.6 | 33.8 | 43.2 | 43.4 |

Marking the handle call and the getter too, by hand, on API 35 (3 rounds), cost 36.6 against
41.1 and 25.7 against 34.2: the same few ns, which a call the proof cannot bound may not
take. moq's export has 200 leaves, the api fixture's 17. Registration covers only those:
registering all of moq's 6,455 native methods would scan the class's methods once each, and
R8 strips the natives nothing calls, which `RegisterNatives` would then refuse, so the
generated rules keep each `@FastNative` method instead. `kotlin/FastNative.kt` declares
the annotation, which Android's SDK makes public only from API 34, for the JVM and older
SDKs. Compiled as a program class, R8 keeps it on the methods only by the rule naming it;
against android.jar's own class, R8 keeps it with no rule, so a build against SDK 34 or
later leaves the file out, as one of two exports in an app must, since D8 refuses a class
defined twice. Only leaves whose `eq` and `hash` walk no value of a caller's size are
marked, as Dart's leaf calls are.

`just ffi android` passes, with these rules, on API 24 (Android 7, which ignores the
annotation), 28, 30, and 35. On API 28, `export.kt` itself never finishes, with or without
the annotation: within 100 s its native memory passes 1.5 GB under a 256 MB heap
([Kotlin on Android 9](/quest/m1/ffi-speed/android-9-memory.md)).

## Swift's inlined calls

Measured 2026-11-16 on **nixos**, as above, through `quest gate`, against moq `ffcb7cd9`.
Expand Down
16 changes: 11 additions & 5 deletions bench/ffi/report.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
from `just ffi bench-run`'s raw.tsv; the median time and peak RSS of the export and its
crate's build, beside moq's own, from its build.tsv; and the sizes of the export's generated bindings and
native libraries, and of the wasm32 stand-in's TypeScript, module, and npm package, and of
one caller's bundle.
one caller's bundle. `just ffi bench-android`'s raw.tsv alone prints only the time tables.

Usage: report.py <raw.tsv> <build.tsv> <export dir> <web export dir> <caller.ts>
Usage: report.py <raw.tsv> [<build.tsv> <export dir> <web export dir> <caller.ts>]
"""

import collections
Expand All @@ -23,6 +23,7 @@
("cpp", "C++"),
("moq-c", "moq-c"),
("kotlin", "Kotlin"),
("kotlin-art", "Kotlin (ART)"),
("python", "Python"),
("swift", "Swift"),
("go", "Go"),
Expand Down Expand Up @@ -219,7 +220,7 @@ def builds(path):
return table(["step", "wall s", "CPU s", "peak RSS", "load average"], rows)


def main(raw_path, build_path, export, web, caller):
def main(raw_path, build_path=None, export=None, web=None, caller=None):
rounds = collections.defaultdict(list)
with open(raw_path) as f:
for line in f:
Expand Down Expand Up @@ -248,7 +249,9 @@ def cell(lang, case, i):
for i, clock, rate in [(1, "CPU", "MB per CPU second"), (0, "wall", "MB/s")]:
print(f"### Calls (ns per call, {clock} time; load average)\n")
header = ["call"] + [name for _, name in langs]
print(table(header, [[label] + [cell(lang, case, i) for lang, _ in langs] for case, label in CALLS]))
rows = [[label] + [cell(lang, case, i) for lang, _ in langs] for case, label in CALLS]
# A case no language here runs, as in bench-android's Kotlin alone.
print(table(header, [row for row in rows if any(row[1:])]))
print(f"\n### Throughput ({rate}, write plus ready read; load average)\n")
header = ["frame"] + [name for _, name in thru]
rows = [[LABELS[n]] + [cell(lang, f"mbps:{n}", i) for lang, _ in thru] for n in SIZES]
Expand All @@ -261,6 +264,9 @@ def cell(lang, case, i):
print(table(header, rows))
print()

if build_path is None:
return

print("### Build (median, from an emptied target)\n")
print(builds(build_path))
print()
Expand Down Expand Up @@ -305,6 +311,6 @@ def cell(lang, case, i):


if __name__ == "__main__":
if len(sys.argv) != 6:
if len(sys.argv) not in (2, 6):
sys.exit(__doc__)
main(*sys.argv[1:])
16 changes: 13 additions & 3 deletions ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,9 @@ its source's URL.
A `[replace]` from the source it replaces, which no `[patch]` can carry, is an error.
Beside the C header, `include/<prefix>.hpp` holds the same API in C++, over a header per crate,
`kotlin/<prefix>.kt` holds it in Kotlin, over JNI functions the crate builds with
`--features kotlin`, and `kotlin/<prefix>.pro` the R8 and ProGuard rules keeping what
those functions look up by name. `python/<prefix>.py` holds it in Python, over a CPython
`--features kotlin`, beside `kotlin/FastNative.kt`, an annotation a build may need with it, and
`kotlin/<prefix>.pro` the R8 and ProGuard rules keeping what those functions look up by
name. `python/<prefix>.py` holds it in Python, over a CPython
extension the crate builds with `--features python`, `swift/<prefix>.swift` holds it in
Swift, over the C header as a clang module, `go/` holds it in Go, a package per crate
over the C ABI through cgo, and `dart/<prefix>.dart` holds it in Dart, over the C ABI through
Expand Down Expand Up @@ -641,7 +642,15 @@ packages place it in.

`<prefix>.kt` is Kotlin for the JVM and Android over the crate's JNI functions, which
`--features kotlin` builds into the same library, loaded by `System.loadLibrary`. It needs
kotlinx-coroutines. Each value crosses JNI as a scalar, a handle as a `Long`, bytes passed
kotlinx-coroutines. A leaf's native method, a primitive field read or write or a derived
`equals` or `hashCode` the export proves waits on nothing and walks no value of a size the
caller sets ([`Abi::leaf`](src/abi.rs)), takes ART's `@FastNative`, so ART keeps the thread runnable across the call;
`_Native`'s first call registers each through `RegisterNatives`, since ART binds such a
method by its name only from Android 12, and the rules keep each. `FastNative.kt`
declares the annotation for a build whose classpath lacks it: the JVM, or an Android SDK
below API 34, where it became public. An Android build against SDK 34 or later leaves the
file out, as android.jar's own class serves, and so does every build after its first of
two exports in one app, whose two copies D8 refuses as one class defined twice. Each value crosses JNI as a scalar, a handle as a `Long`, bytes passed
in as a `ByteArray`, bytes returned as their owner, a block of numbers as its primitive
array, and anything else encoded into a `ByteArray` as its C struct holds it. A record a
call takes or returns directly, or in an `Option`, crosses as its scalars, a nested
Expand Down Expand Up @@ -1644,6 +1653,7 @@ just ffi corpus-update # take corpus/crates.txt afresh from crates.io's top cra
just ffi protocol # run longer protocol scripts from more seeds through the C ABI and every binding, as the nightly does
nix develop .#bench --command just ffi browsers # api's TypeScript in headless Chromium, Firefox, and on Linux WebKit, lowering `using` three ways for WebKit, as the nightly runs it
nix develop .#android --command just ffi android # the Kotlin test, shrunk by R8, under CheckJNI on the Android device adb reaches
nix develop .#android --command just ffi bench-android # the benchmark's Kotlin rows on HotSpot and, shrunk by R8, on ART on the Android device adb reaches
nix develop .#bench --command just ffi bench # the binding benchmark: bench/ffi/README.md's cases from Rust, and through the export from C, C++, Kotlin, Python, Swift, Go, and Dart
nix develop .#bench --command just ffi codspeed # after bench-build, each single-threaded case once under CodSpeed's CPU simulation, as the CodSpeed workflow runs it
nix develop .#sanitize --command just ffi sanitize # api's C and C++ scripts under ASan with UBSan, TSan, and Valgrind, its Swift under TSan, its Go under the race detector and cgocheck2, its Python looped on a debug CPython, and its C ABI under Miri
Expand Down
Loading