Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -296,11 +296,13 @@
'';
};
# `just ts bench` adds Chrome and Firefox as engines, and `just ffi browsers` runs the
# web binding's test in them. Apart for their size.
# web binding's test in them. Apart for their size. Binaryen's wasm2js lowers the
# bench's wasm32 build to JavaScript for its wasm2js route.
devShells.bench = self.devShells.${system}.default.overrideAttrs (old: {
nativeBuildInputs = old.nativeBuildInputs ++ [
pkgs.chromium
pkgs.firefox
pkgs.binaryen
];
# `just ffi bench` builds moq's rs/moq-c, whose VAAPI bindings run bindgen.
LIBCLANG_PATH = "${pkgs.llvmPackages.libclang.lib}/lib";
Expand Down
75 changes: 34 additions & 41 deletions quest/m2/wasm/routes.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,42 @@
# [M] Every route on the browser benchmark
# [M] The binding benchmark's cases through the translation

## Goal

`just ts bench` ([the TS benchmark](/ts/README.md#bench)) times each workload through two
more routes, in Chrome, Firefox, Node, and Bun: the fixture's `pub` API exported by
[rust2-wasm](/ffi/README.md#typescript-web), called through its generated binding, and the
same `.wasm` lowered to JavaScript by Binaryen's `wasm2js`. Each route's output must match
native Rust's first, and the report gives each route's shipped bytes, raw and gzipped,
with its `.wasm` or runtime included, its cold start, from a fresh worker to its first
result with `WebAssembly.compile` or the module's evaluation inside the time, and how long
it takes to build. [The binding benchmark](/bench/ffi/README.md)'s cases, which cross
handles, records, objects, reads, and payloads, run through the translation too, beside
rust2-wasm's TypeScript column there.
[The binding benchmark](/bench/ffi/README.md)'s cases, which cross handles, records,
objects, reads, and payloads, run through rust2-ts's translation of the same moq API too,
beside rust2-wasm's TypeScript column there, in Node, Chromium, and Firefox, each case's
result matching the export's first. With [the TS benchmark](/ts/README.md#bench)'s routes,
this is the measurement [the gate](/quest/m2/wasm/README.md) reads.

## Plan

Decided with the maintainer on 2026-11-09: this is the measurement
[the gate](/quest/m2/wasm/README.md) reads.

- The benchmark's existing wasm32 route stays, as compiled Rust with a hand-written
boundary whose copies are timed apart: against it, the rust2-wasm route shows what the
generated binding costs.
- `wasm2js` is the control no one has measured: it needs no translator and no WebAssembly
at run time, which is what moq's
[`rs2ts-wasm2js`](https://github.com/moq-dev/moq/blob/main/quest/m5/rs2ts-wasm2js.md)
planned to try first. No published speed or size figures for it were found. If it
matches the translation on speed and size, that is a finding about the translator, to
report plainly.
- What is published points both ways, so only this benchmark decides: a Rust and WASM
parser lost 2.2 to 4.6 times to a TypeScript rewrite on boundary costs
([OpenUI, 2026-03](https://www.openui.com/blog/rust-wasm-parser)), while Scala.js on
Wasm won up to 5.8 times on hashing and lost on object-heavy workloads
([VirtusLab, 2024-07](https://virtuslab.com/blog/scala/scala-to-web-assembly/)).
- Decided with the maintainer on 2026-11-12, in the quest audit: ✅ the routes are built
alike. Today `ts/Cargo.toml`'s `wasm-release` profile is release at `opt-level = "z"`,
where overflow wraps and debug assertions are off, while `ts/src/cli.rs` translates
from the dev profile, with both on. Add a wasm route built with overflow checks and
debug assertions, or a translation with release semantics, so each pair compared runs
the same checks; the report says which pair it compares.
- The existing benchmark crosses the boundary once per sample, since every route is one
`run(&[u8]) -> Vec<u8>` (`ts/bench/entry.ts`), which is why the binding benchmark's
cases join it.
- moq's `js/net/bench/frames.ts` workload is [the model epic](/quest/m2/ts-model/README.md)'s
to add once the model translates, on every route here.
- `wasm2js` needs Binaryen, which the flake does not have yet, and it legalizes `i64`
exports, so the generated binding may need its own entry for that route.
Decided with the maintainer on 2026-11-09: this is the measurement the gate reads.

Decided with the maintainer on 2026-11-17, on
[#860](https://github.com/kixel-ai/rust2/pull/860#issuecomment-6947175990):

- ✅ `just ts bench`'s routes land in #860, and this quest keeps only the binding benchmark's
cases through the translation, blocked on the model epic (rejected: holding #860 until the
model translates; dropping these cases).
- ✅ The `wasm2js` route lowers the hand-boundary `wasm checked` build, which isolates the
lowering's cost (rejected: teaching the generated binding to load a lowered rust2-wasm
module, which needs a new `load` source and `i64` legalization on the binding).

- [#860](https://github.com/kixel-ai/rust2/pull/860) gave `just ts bench` its rust2-wasm and
`wasm2js` routes, each route's build time and cold start, and the `wasm-checked` profile,
so the codec's and the handoff's routes run the same checks. What is left needs moq's
model translated: the cases call `TrackProducer`, `GroupProducer`, `GroupConsumer`,
`FrameFrame`, and `Timestamp` (`ffi/moq/bench/export.ts`), which no translation reaches
until [the model epic](/quest/m2/ts-model/README.md) lands.
- Translate with the model fixture's roots (`ts/moq/model.toml`) and port `export.ts`'s
cases onto the translation's API, which [the epic](/quest/m2/wasm/README.md)'s 2026-11-12
decision made the binding's too, so the port renames imports rather than rewriting calls.
- Report the translation as one more column in `bench/ffi/report.py`'s TypeScript table.
The translation keeps the overflow checks the export's release build drops, so the
report says which checks each column runs, as `just ts bench`'s does.

Public API: none.

## Required

- [Translate moq-net's model](/quest/m2/ts-model/README.md) - the translation reaches the handles, groups, and frames the cases call
14 changes: 14 additions & 0 deletions ts/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions ts/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ members = [
"fixtures/bind",
"fixtures/census",
"fixtures/codec",
"fixtures/codec/web",
"fixtures/crates",
# A fixture per module of the language, beside what they share.
"fixtures/lang/*",
Expand All @@ -27,6 +28,7 @@ members = [
"fixtures/share",
"fixtures/stock",
"fixtures/task",
"fixtures/task/web",
"fixtures/tokio",
]
resolver = "3"
Expand Down Expand Up @@ -67,3 +69,10 @@ lto = "fat"
codegen-units = 1
strip = "symbols"
opt-level = "z"

# wasm-release with the checks the translation keeps, overflow checks and debug assertions,
# so the bench's wasm32 routes built with it run what the translated route runs.
[profile.wasm-checked]
inherits = "wasm-release"
overflow-checks = true
debug-assertions = true
31 changes: 22 additions & 9 deletions ts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1436,28 +1436,41 @@ does, whose products pass 2^53, then added by its `checked_add` and ordered by i
whose cross products do too, integers sorted, searched for their greatest and least,
and kept in a `BinaryHeap`, TS packets numbered through `as_chunks_mut` as moq-mux
numbers them, each also read by value, a copy of its bytes, and frames' `Bytes` split off a
group's, cloned, sliced, and dropped, as moq-net's model hands them to its consumers. Each runs four ways: natively, as wasm32 at the
size-optimized profile moq's WASM shipped with, hand-written (moq's `js/net` in `bench/moq.ts` and its `Timestamp`
group's, cloned, sliced, and dropped, as moq-net's model hands them to its consumers. Each runs seven ways: natively; as wasm32 at the
size-optimized profile moq's WASM shipped with, through `fixtures/wasm.rs`'s hand-written
boundary; the same at `wasm-checked`, that profile with the overflow checks and debug
assertions the translation keeps; that build lowered to JavaScript by Binaryen's `wasm2js`;
through [rust2-wasm](../ffi/README.md#typescript-web), the export of a facade whose `pub`
API is `bench::run` alone (`fixtures/codec/web`), built at `wasm-checked` for
`wasm32-wasip1` and called through its generated TypeScript; hand-written (moq's `js/net` in `bench/moq.ts` and its `Timestamp`
in `bench/timestamp.ts`, ES iterator helpers in `bench/iter.ts`, a sorted array in
`bench/sorted.ts`, `bigint` in `bench/rescale.ts`, as moq's `js/hang` converts
timescales, typed arrays and a heap of its own in `bench/order.ts`, offsets into one
`Uint8Array` in `bench/packets.ts`, and `subarray`s in `bench/frames.ts`), and translated; every route's output must match
native Rust's first. The JS routes run minified, the same bundles whose size it reports,
in Node, Bun, Chrome, and Firefox, with the WASM route's copies across the boundary timed
`Uint8Array` in `bench/packets.ts`, and `subarray`s in `bench/frames.ts`); and translated. Every route's output must match
native Rust's first. The four routes the translation is compared with run the same checks,
so rust2-wasm against `wasm checked` is the generated binding's cost, `wasm2js` against it
the lowering's, and `wasm checked` against `wasm` the checks'. The JS routes run minified, the same bundles whose size it reports,
in Node, Bun, Chrome, and Firefox, with the reference WASM route's copies across the boundary timed
apart from its compute, and an A/A copy of the
translation to show the noise. Each route runs alone in a worker of its own, since routes
translation to show the noise. `bench/build.ts` builds every route from the fixtures'
source, timing each build from an emptied target directory without a compiler cache, and
each round's fresh worker times its route's cold start, from loading its files to its
first output, compiling its `.wasm` or evaluating its JavaScript included; each `.wasm` is
compiled with a custom section of its own, since V8 shares a module's code across a
process. Each route runs alone in a worker of its own, since routes
in one realm share type profiles and run slower loaded second, and each round starts every
route's worker afresh: a worker keeps the inlining and deopts it settles into for the whole
run, which put identical copies of a route up to 1.6x apart in Chrome, so a round is one
sample of each workload from fresh workers. Each round warms up and times the routes and
workloads in a new random order, so no route keeps a slot that runs it faster or slower
than identical code in another; the report records the seed. It also records the host and engine versions,
and each route's bundle size raw and gzipped. It sets no threshold; the nightly workflow
and each route's shipped bytes raw and gzipped, rust2-wasm's `.wasm` and binding together. It sets no threshold; the nightly workflow
runs it and posts the report to its summary. Each browser's page reports each step it
starts, and one that runs 30 s fails the run, naming it.

The task fixture's `bench::run` times an async handoff: a value sent to a task parked on
its channel, read through 1, 4, or 16 async functions. Its translation runs polled at
its channel, read through 1, 4, or 16 async functions, through the same wasm32 routes, its
facade in `fixtures/task/web`. Its translation runs polled at
once, as natively, and on `rs.spawn`; its hand-written route is the same handoff with
promises, as moq's TypeScript writes it.

Expand All @@ -1476,4 +1489,4 @@ workloads still settle into one of two speeds per worker, up to 2x apart, so on
their A/A stayed 6-35% wide while the median workload's was 1-3%. The report's
A/A line also gives the noise of a family's geometric mean, the steadier gauge.

The bench shell adds Chromium and Firefox.
The bench shell adds Chromium, Firefox, and Binaryen.
145 changes: 145 additions & 0 deletions ts/bench/build.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
// Builds each fixture's routes from its Rust source, timing each step as a consumer's first
// build: from an emptied target directory, without the dev shell's compiler cache, in wall
// and CPU seconds, its processes' children included.

import { copyFile, rm } from "node:fs/promises";
import { join } from "node:path";

/** A step's cost, in seconds. */
export interface Step {
step: string;
wall: number;
cpu: number;
}

/** What `build` made, as paths, and what each step cost. */
export interface Built {
/** The wasm32 build at the profile moq's WASM shipped with. */
wasm: string;
/** The same at `wasm-checked`, which keeps the translation's checks. */
checked: string;
/** `checked` as wasm2js lowered it: an ES module of its exports. */
lowered: string;
/** rust2-ffi's export of the fixture's facade, whose `web/` the rust2-wasm route imports. */
export: string;
/** The export's module, built at `wasm-checked`. */
exported: string;
/** The translation. */
translated: string;
steps: Step[];
}

/** The translators, and where the builds go. */
export interface Tools {
/** rust2-ts's release binary. */
ts: string;
/** rust2-ffi's release binary. */
ffi: string;
/** ts/, which `fixtures/` is in. */
root: string;
/** Where the routes' files go. */
out: string;
/** The target directory, emptied before each step. */
scratch: string;
}

// The drivers link the toolchain's own libraries, as rustc does.
const sysroot = Bun.spawnSync(["rustc", "--print", "sysroot"], { stderr: "inherit" }).stdout.toString().trim();
const libpath = process.platform === "darwin" ? "DYLD_LIBRARY_PATH" : "LD_LIBRARY_PATH";

// wasm2js validates against the MVP unless told what rustc turns on for wasm32.
const features = [
"--enable-bulk-memory",
"--enable-bulk-memory-opt",
"--enable-nontrapping-float-to-int",
"--enable-sign-ext",
"--enable-mutable-globals",
"--enable-reference-types",
"--enable-multivalue",
];

/**
* Builds the routes of `fixtures/<name>` (`codec` or `task`), the crate `<name>-fixture` and
* its facade in `web/`, into `tools.out`, and its translation into the directory `translated`.
*/
export async function build(name: string, translated: string, tools: Tools): Promise<Built> {
const { root, out, scratch } = tools;
const steps: Step[] = [];
const step = async (step: string, cmd: string[]) => steps.push({ step, ...(await timed(cmd, root, scratch)) });
const built: Built = {
wasm: join(out, `${name}.wasm`),
checked: join(out, `${name}-checked.wasm`),
lowered: join(out, `${name}-wasm2js.mjs`),
export: join(out, `${name}-web`),
exported: join(out, `${name}-web.wasm`),
translated,
steps,
};

const wasm = (profile: string) =>
step(profile === "wasm-release" ? "wasm" : "wasm checked", [
"cargo",
"build",
"--locked",
"--quiet",
"--package",
`${name}-fixture`,
"--features",
"bench",
"--target",
"wasm32-unknown-unknown",
"--profile",
profile,
]);
await wasm("wasm-release");
await copyFile(join(scratch, "wasm32-unknown-unknown/wasm-release", `${name}_fixture.wasm`), built.wasm);
await wasm("wasm-checked");
await copyFile(join(scratch, "wasm32-unknown-unknown/wasm-checked", `${name}_fixture.wasm`), built.checked);
await step("wasm2js", ["wasm2js", "-O", ...features, built.checked, "-o", built.lowered]);

const facade = join(root, "fixtures", name, "web");
await rm(built.export, { recursive: true, force: true });
await step("rust2-ffi", [tools.ffi, "--config", join(facade, "ffi.toml"), "--target", "wasm32-wasip1", "--out", built.export]);
await step("export's build", [
"cargo",
"build",
"--locked",
"--quiet",
"--manifest-path",
join(built.export, "Cargo.toml"),
"--target",
"wasm32-wasip1",
"--profile",
"wasm-checked",
"--features",
"web",
]);
await copyFile(join(scratch, "wasm32-wasip1/wasm-checked", `${name}_web.wasm`), built.exported);

await rm(built.translated, { recursive: true, force: true });
await step("rust2-ts", [tools.ts, "--config", join(root, "fixtures", name, "ts.toml"), "--out", built.translated]);
return built;
}

/** Wall and CPU seconds of `cmd` in `cwd`, from an emptied `scratch` target, uncached. */
async function timed(cmd: string[], cwd: string, scratch: string): Promise<Omit<Step, "step">> {
// Removed, not emptied: cargo cleans only a target directory it made.
await rm(scratch, { recursive: true, force: true });
const env: Record<string, string | undefined> = { ...process.env, CARGO_TARGET_DIR: scratch, [libpath]: join(sysroot, "lib") };
// mbx's Cargo shim keeps a compiler cache that emptying the target does not empty: the
// wrapper is its, and CARGO would name the shim to the drivers.
const shim = process.env.RUST2_CARGO_SHIM;
env.PATH = (process.env.PATH ?? "")
.split(":")
.filter((dir) => dir !== shim)
.join(":");
delete env.RUSTC_WRAPPER;
delete env.CARGO;
const start = performance.now();
const proc = Bun.spawn(cmd, { cwd, env, stdout: "inherit", stderr: "inherit" });
if ((await proc.exited) !== 0) throw new Error(`${cmd.join(" ")} failed`);
const wall = (performance.now() - start) / 1000;
const usage = proc.resourceUsage();
if (!usage) throw new Error(`${cmd[0]} reported no resource usage`);
return { wall, cpu: Number(usage.cpuTime.total) / 1e6 };
}
Loading