Skip to content

fix(ts): run each bench route alone in a worker - #116

Merged
kixelated merged 4 commits into
mainfrom
quest/m2/ts-bench-aa
Oct 27, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m2/ts-bench-aa

Conversation

@kixelated

@kixelated kixelated commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

just ts bench ran every JS route in one realm, so routes shared state and the A/A route did not measure only noise. I reproduced each effect on main and in in-realm variants of the bench:

  • Firefox, handoff. translated and spawned shared one module, so the same drain and channel call sites saw both routes' wakers. A/A had a module of its own and ran 20% faster than translated at depth 1 and 11-15% faster at depth 4, in both low-load runs on main. Giving spawned its own module brought the gap to -6.7%, +1.1%, and +0.4%. feat(ts): smaller output from MIR #95's finding was this same effect: whichever build sat in the translated slot shared a module with spawned.
  • JavaScriptCore. Modules with identical text share type profiles. With spawned in a module of its own but with the same text, Bun's translated ran depth 16 in 587 ns. Once each bundle's text differed (an extra export naming the route), it ran in 454 ns. So spawned's types also slowed down translated and A/A in Bun.
  • V8, positional. With distinct texts in one realm, the copy loaded second ran slower in Chrome's handoff: +2.0% to +8.1% across two runs. Swapping the load order flipped the sign (-3.7% to -12.4%). Randomizing the load order would only turn that bias into run-to-run noise.
  • Correlated states. In one realm, both copies settled into the same JIT state, so A/A agreed with translated even when that state was slow. For example, Chrome's btree/256 ran at 122 ns in both copies on main. Run alone, each copy runs at 88-90 ns.

Approach

  • Each route, and the wasm32 build's three steps, runs alone in a worker of its own. measure drives the workers one sample at a time, so rounds stay interleaved in a random order across all routes. Browsers use module workers, and Node and Bun use worker_threads.
  • Each route loads a file of its own: task-spawned.js is new. No cache keyed by URL can serve one route's code to another.
  • After loading, a worker blocks in Atomics.wait on a shared control word between samples. It never goes back to its event loop, and each await returns only to the microtask queue, which is all an async body needs. A first version that sent each request by postMessage let the engine do idle-time work between samples. That spread Firefox's samples of one body up to 2.5x apart (±10-30% per row), against ±1-2% now.
  • Updated the report text, ts/README.md, and the comments in time.ts and entry.ts. Deleted the quest.

Measurements

All runs went through quest gate on an AMD Ryzen 9 5900XT (32 threads). The 1-minute load average was 1.2-2.4 during the compared runs. "Before" is main (two runs) and "after" is this branch (three runs; the last is the pushed head). Codec cells are |A/A ÷ translated - 1| at the median and maximum workload. Handoff cells are A/A ÷ translated - 1 at depths 1, 4, and 16.

engine codec before codec after handoff before handoff after
Chrome 155 0.5%/8.5%, 0.6%/2.6% 0.9%/16%, 1.1%/30%, 1.3%/24% -4/-1/-6, -4/-1/+1 +2/-2/+3, -3/+2/+1, +1/-2/-7
Firefox 158 2.8%/16%, 2.0%/10% 2.3%/7.8%, 1.5%/12%, 2.1%/14% -20/-11/-5, -20/-15/-5 -2/+1/-1, +2/-2/+1, -3/-1/+1
Bun 1.4.2 1.4%/31%, 1.1%/4.0% 1.7%/7.5%, 1.0%/7.6%, 3.0%/31% -13/-15/-11, -2/-8/+1 +1/+6/0, -13/-7/-10, +4/0/-1
Node 24 0.9%/15%, 1.7%/18% 1.6%/15%, 1.1%/11%, 1.1%/15% +1/+9/+2, -11/-2/+5 -1/+1/0, -2/0/+2, -9/+1/-2
  • Firefox's handoff bias is gone.
  • Bun's translated handoff no longer carries spawned's types: depth 16 dropped from 552-618 ns to 431-469 ns.
  • Chrome's and Bun's codec maxima rose. That spread is real: each copy now settles into its JIT state on its own. Chrome's 1-byte varint decode copies ran up to 1.3x apart. In the third run, Bun's translated copy ran varints up to 1.45x slower than its A/A copy. One realm hid this by putting both copies in the same state, so a --route build with different text was then compared against a correlated pair.
  • The median per-sample spread of translated codec rows is about the same in Chrome (0.7-0.8% before, 0.8-0.9% after) and Firefox (3.6-4.1% before, 3.2-5.5% after). It rose in Node (0.5% to 1.3-1.5%) and Bun (1.2% to 1.4-1.8%).
  • Wall time of bench/main.ts: 292-293 s before, 293-295 s after. CPU cost is unchanged: same samples, plus a few workers' startup.

Impact

  • Public API: none. Bench only.

Alternatives

  • One realm, a module and distinct text per route. This fixes the Firefox and JavaScriptCore effects, but not V8's load-order effect or the correlated states.
  • Randomized load order. It turns the positional bias into noise between runs instead of removing it.
  • A process per route. It loses the interleaved rounds, so drift would land on one route.
  • Workers answering postMessage requests. Firefox's per-sample noise rose 3-10x, as above.
  • Randomized worker spawn order. I did not add it, since I found no positional effect between workers.

Follow-ups

  • Suggested for /quest-plan: per-instance JIT states now dominate the A/A maximum in Chrome and Bun. Timing more than one instance per route, or reporting the gap between instances, would tell a real change from a route that settled into a slow state.
  • #108 edits the same report text in main.ts. Whichever lands second merges the other.

Checks

just check, just test, and quest check --base origin/main pass. Ran 5 full just ts bench runs of this design (2 with the postMessage transport, 3 with Atomics.wait) and 4 of main through quest gate, plus in-realm and Chrome-only diagnostic variants. Every route's output matched native Rust's on every engine.

(written by Opus 5.5)

🤖 Generated with Claude Code

https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn

kixelated and others added 3 commits October 27, 2026 02:55
Quest-Claim: c70fa28a30e9fb78
Routes in one realm shared state: Firefox's handoff `translated` shared its
module with `spawned`, JavaScriptCore merged the type profiles of identical
bundles, and V8 ran a route slower loaded after its identical copy. Each
route now loads its own file in a worker of its own, which blocks in
`Atomics.wait` between samples instead of going back to its event loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
@kixelated kixelated changed the title An A/A control that measures noise fix(ts): run each bench route alone in a worker Oct 27, 2026
@kixelated
kixelated marked this pull request as ready for review October 27, 2026 11:58
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-27T16:12:10.331475Z 0b67964 New commits
🔒 Security Review ✅ Completed 2026-10-27T16:14:48.248789Z 0b67964 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: da50acf

No actionable bugs found. Reviewed worker startup/error handling and cleanup, the single-outstanding-request shared-control handshake, output checks, warmup/sample accounting, and the microtask-only handoff workloads.

Direction looks sound: per-route workers remove shared-realm profiling effects while retaining interleaved rounds; the small Atomics protocol is justified by the reported idle-time distortion rather than expanding into a general worker RPC layer. Independent JIT-instance variation remains a measurement limitation, as the PR notes; this does not establish that a single A/A pair captures all benchmark uncertainty.

Verification: static source review only. I did not run tests or benchmarks, reproduce browser/JIT measurements, or independently validate the reported before/after results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
@kixelated
kixelated merged commit 9fc2c22 into main Oct 27, 2026
1 check passed
@kixelated
kixelated deleted the quest/m2/ts-bench-aa branch October 27, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant