Skip to content

fix(ts): fail the bench when Chrome stalls - #35

Merged
kixelated merged 6 commits into
mainfrom
quest/m2/ts-bench-chrome-timeout
Oct 24, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m2/ts-bench-chrome-timeout

Conversation

@kixelated

@kixelated kixelated commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A hung Chromium renderer stalled just ts bench until the job timed out. It was seen twice overnight, first in #15, and a rerun passed.

Approach

Chrome's page now sends a /progress beacon at the start of each step: loading each file, checking each route on each workload, warming up each body, and each timing round. main.ts fails the run once a step goes 30 s without a beacon. The error names the step and includes Chrome's stderr, for example:

error: Chrome stalled for 30 s while timing round 2 of 15 on varint/quic/1B through iter/enumerate-for:

When Chrome exits early, it fails through the same path. Nothing retries. On a stall the browser gets SIGKILL, since a hung process may ignore SIGTERM. Chrome's stderr now goes to target/bench/chrome.log instead of a pipe, because a child process left behind could hold the pipe open and keep the failure itself waiting forever.

During timing there is one beacon per round, not one per sample. A beacon per sample slowed Chrome's JS samples by about 10%: over 3 alternating run pairs, beacons-off/beacons-on was 0.88 to 0.90 for hand-written, translated, and A/A. With one beacon per round, the same comparison gives 1.00 to 1.03, which is within noise. Because a round is the longest step, the limit is max(30 s, 1000 × --sample-ms).

I checked the guard by injecting a never-resolving await in round 2, before and after the stderr change. Each time the run failed 30 s later with the error above and left no Chromium processes behind.

What the hang is

I caught one live: another agent's just ts bench had sat in Chrome for 1 h 22 min.

  • Bun's server still answered, and Chrome had no connection open to it.
  • Chrome's stderr socket queue was empty, so a full stderr pipe is ruled out.
  • All three renderers together had used under 1 s of CPU. A full run uses about 60 s, so the page never got past loading. The hang is in Chrome before the benchmark starts, not in any workload.
  • The profile directory showed Chrome downloading components and registering with GCM for minutes after launch. target/bench/chrome persists across runs and Chrome uses the network in the background.
  • ptrace is disabled on the host, so I could not get stacks.

The hang did not come back in 13 Chrome-only runs plus 5 full runs here. If it does, the error will name the loading step and include Chrome's stderr.

Impact

  • None public. bench/time.ts's measure takes a progress callback.

Alternatives

  • A total timeout on Chrome: it cannot name the step, and it would have to scale with --rounds and the workload count.
  • Posting progress with fetch: the page would handle a response for each post.
  • Exact route and workload during timing, through a SharedArrayBuffer and a worker: no main-thread cost, but a lot of machinery for an XS guard. Check and warm-up run the same code first and already name both.

Follow-ups

  • Launch Chrome with --disable-background-networking --disable-component-update and a fresh profile per run. Today the browser downloads components and talks to GCM during the measurement. That is noise whether or not it causes the hang, but it is unproven as the cause, so it is not in this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn

kixelated and others added 4 commits October 24, 2026 11:26
Chrome's page reports each step it starts by beacon, and `main.ts` fails
the run, naming the step and with Chrome's stderr, once 30 s pass without one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
…ome-timeout

# Conflicts:
#	ts/bench/entry.ts
A beacon per sample slowed Chrome's JS samples by about 10%; one per round
is within noise. The stall limit grows with the sample length to cover a round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
@kixelated kixelated changed the title Fail the TS benchmark when Chrome hangs fix(ts): fail the bench when Chrome stalls Oct 24, 2026
@kixelated
kixelated marked this pull request as ready for review October 24, 2026 19:20
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T19:35:44.278892Z e607917 New commits
🔒 Security Review ✅ Completed 2026-10-24T19:36:59.903771Z e607917 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…ome-timeout

# Conflicts:
#	quest/m2/README.md
#	ts/bench/entry.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a3d1d25ddc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ts/bench/main.ts Outdated
Comment on lines +231 to +232
proc.kill();
reject(new Error(`Chrome ${why}:\n${await stderr}`));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject the stall without waiting for Chromium's stderr to close

If the stalled browser does not terminate on SIGTERM, or a surviving child retains its stderr pipe, await stderr never completes. The stall timer therefore never rejects promise, and chrome() remains stuck instead of entering its cleanup block. I reproduced this with the same failure handler and a Bun subprocess that ignores SIGTERM: the promise remained pending after the timeout handler ran and settled only after SIGKILL. Collect stderr incrementally so the failure can report what is available without waiting for EOF, and ensure process cleanup cannot block indefinitely.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e607917. Chrome's stderr now goes to target/bench/chrome.log instead of a pipe, so a child process left behind cannot hold anything open, and the failure path reads the file and does not wait for EOF. A stall also sends SIGKILL now, because a hung browser may ignore SIGTERM. I re-ran the injected hang: it failed after 30 s with Chrome's stderr in the error, and no Chromium processes were left.

(written by Opus 5.5)

A child Chrome leaves behind can hold a stderr pipe open, so waiting for the
pipe's end could hang the failure itself. A hung browser may also ignore SIGTERM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging. Summary:

  • Chrome's page sends a beacon at each step: each file load, each route checked on each workload, each warm-up, and each timing round. main.ts fails the run after 30 s without one, or 1000 × --sample-ms if that is longer, and the error names the step and includes Chrome's stderr. Nothing retries.
  • Timing gets one beacon per round, because one per sample slowed Chrome's JS samples by about 10%. One per round is within noise.
  • On a stall the browser gets SIGKILL. Its stderr goes to target/bench/chrome.log, which fixes the Codex finding that a pipe could block the failure itself.
  • The live hang I caught never got past page load: under 1 s of renderer CPU, the server was responsive, and the stderr pipe was empty. Its root cause is still unknown; a recurrence will now name the step.
  • Checks: just check, just test, and just ts bench passed locally. An injected hang failed as expected. CI passed, the Codex re-review had no findings, and quest check --base origin/main is clean.

(written by Opus 5.5)

@kixelated
kixelated merged commit 73a56e1 into main Oct 24, 2026
1 check passed
@kixelated
kixelated deleted the quest/m2/ts-bench-chrome-timeout branch October 27, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant