Skip to content

fix(ts): shuffle the bench's order each round so no route keeps a slot - #39

Merged
kixelated merged 3 commits into
mainfrom
fix/ts-bench-slot-rotation
Oct 25, 2026
Merged

kixelated merged 3 commits into
mainfrom
fix/ts-bench-slot-rotation

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Problem

just ts bench ran every round in nearly the same order: each round started one body later, but with ~180 bodies and 15 rounds every route always ran right after the same one. Identical code then timed differently by slot, and earlier A/B conclusions through A/A (such as #24's "Bun varint decode 10-30% slower") were mostly that bias.

Reproduced on main with an A/A/A run (--route same=target/bench, so translated, A/A, and same all load the same translation), 6 runs:

engine translated ÷ A/A, median workload per run workloads where translated is slower
Node -0.1% to +2.8% 61%
Bun +3.3% to +11.2% 80%
Chrome +2.5% to +5.2% 74%

In Bun and Chrome the slots order by position: translated > A/A > same, each identical copy faster than the one before it on the same workload.

Approach

Each round runs the bodies in a new Fisher-Yates order from a seeded PRNG. The seed is random per run, recorded in report.md and report.json, and bench/main.ts --seed replays it. After the fix, 5 runs:

engine translated ÷ A/A, median workload per run workloads where translated is slower median |gap| before → after
Node -2.2% to +0.7% 44% 3.5% → 2.9%
Bun -0.9% to -0.1% 46% 10.2% → 3.9%
Chrome -0.8% to +1.6% 49% 4.1% → 4.6%

A/A ÷ same and translated ÷ same look the same (45-52% slower, per-run medians within ±3.3%), so the gap is now the A/A noise floor: no sign, coin-flip direction. Host: 32 threads shared with other agents' benches, load average 12-31 throughout; runs alternated main's harness and this one.

Cause, from a Bun-only harness over the three identical copies: with the copies adjacent on each workload, the earlier copy is slower (26-30 of 34 workloads); running all workloads of one copy before the next removes the bias (11-21 of 34), and so does a full Bun.gc(true) before each sample (10-23 of 34). So a body inherits the allocator and GC state its predecessor leaves, and a fixed order hands every body the same predecessor.

The README now says how to read an A/B run: its gap from translated against A/A's in the same run, with no reversed run needed.

Impact

  • bench/main.ts takes --seed <u32>; the report records the seed. No public API change.

Alternatives

  • Rotating which route goes first per workload: still fixed neighbors for most pairs, and it only covers routes, not workloads.
  • A full GC before each sample: needs a flag in Node and Chrome, and charges each body none of its own GC, which real code pays.
  • Isolating each route in its own process: drops the interleaving that spreads drift, and was not needed once the order was random.

Follow-ups

  • The native route (fixtures/bench.rs) still times its workloads in a fixed order; it has no identical-code slots to compare, so no bias was measured there.

🤖 Generated with Claude Code

https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn

(written by Opus 5.5)

kixelated and others added 2 commits October 24, 2026 15:27
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T01:52:02.034533Z 2a007e7 New commits
🔒 Security Review ✅ Completed 2026-10-25T01:52:33.089944Z 2a007e7 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Ready to merge:

  • CI passes on 2a007e7. Codex reviewed 2a007e7 with no findings.
  • quest check --base origin/main --branch fix/ts-bench-slot-rotation passes; just check and just test pass locally.
  • A/A/A runs, translated ÷ A/A, median workload per run: before (6 runs), Node -0.1% to +2.8%, Bun +3.3% to +11.2%, Chrome +2.5% to +5.2%, with translated slower on 61%, 80%, and 74% of workloads. After (5 runs): Node -2.2% to +0.7%, Bun -0.9% to -0.1%, Chrome -0.8% to +1.6%, slower on 44%, 46%, and 49%, the same as A/A ÷ same.

(written by Opus 5.5)

@kixelated
kixelated merged commit c56c7ea into main Oct 25, 2026
1 check passed
@kixelated
kixelated deleted the fix/ts-bench-slot-rotation branch October 27, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant