Skip to content

test(website): mark s2 dispatches and selectorCalls report-only in render benchmark - #33444

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/31963-render-bench-flaky
Nov 8, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/31963-render-bench-flaky

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop the Render Benchmark s2 scenario from failing CI intermittently when no frontend change caused it.

The Render Benchmark (counter budgets) job fails s2.selectorCalls (and sits near the s2.dispatches limit) on diffs that never touch the chat page. In some runs the store sees ~25-45 more dispatches than the usual ~309-312, selector calls follow, and when the median crosses the limit the job goes red (CI has observed dispatches=351).

Why it matters

A benchmark that red-fails on unrelated PRs blocks and distracts every contributor whose diff happens to land in a high-variance window. The jitter is not a performance regression, so the failures are pure noise that erodes trust in the gate and costs reruns.

Not a goal

  • Not changing any budget value or the tolerance (the numbers are not raised to hide the jump; they stay and still print every run).
  • Not changing product code, bench-render.mjs, or any scenario other than s2.
  • Not eliminating the underlying frame-rate-bound variance itself (that is inherent to the browser's per-animation-frame batching under the benchmark's wall-clock frame delivery).

What changed (motivation → approach → change)

Observed symptom: s2.dispatches (and the dependent s2.selectorCalls) spike in some runs with no related diff.

Root cause: the s2 dispatch count decomposes into 9 fixed one-off dispatches (setSlotStatusDetail ×2, sseChatMessage:user, setVoicePlaying, syncSlotRunningFromServer, setSidebarOrder, sseChatMessage:_done, refreshSlot/pending, refreshSlot/fulfilled) plus stream/flush, nominally 300 (one per 100 ms tick) — so a clean run is 309. Every extra dispatch is one extra stream/flush: confirmed by per-type breakdown, when total dispatches rose 309 → 310 the stream/flush count rose 300 → 301 and the 9 one-offs stayed constant. stream/flush is dispatched once per animation frame by the streaming-chunk coalescer (src/hooks/websocket/streamBuffers.ts); the benchmark delivers the prose stream on a 100 ms wall clock while the flush fires on requestAnimationFrame, so a tick's chunks can split across two rAF flushes when frames render fast. Under runner load the drift accumulates (~40). Each open-slot flush notifies immediately, so selectorCalls track dispatches.

8-run reproduction on main (no code change):

r1 dispatches=309 selectorCalls=187305
r2 dispatches=313 selectorCalls=189840
r3 dispatches=309 selectorCalls=187440
r4 dispatches=327 selectorCalls=196956   <-- jump
r5 dispatches=309 selectorCalls=187245
r6 dispatches=309 selectorCalls=187341
r7 dispatches=309 selectorCalls=187421
r8 dispatches=309 selectorCalls=187341

Approach / change: perf-budgets.json's own note defines the per-scenario report list for exactly this case — a counter that varies between runs on the runner by more than the tolerance is printed but never fails the check. The same pattern was already applied to s1x5.selectorCalls. This adds dispatches and selectorCalls to s2's report list. Budget values and tolerance are unchanged.

Reviewer note: with this change, s2.dispatches and s2.selectorCalls become report-only — still printed every run, but a real regression in those two s2 counters is no longer gated by --check. s2.commits and s2.renders remain gated.

Backwards compatibility

Removes nothing: the change only moves two s2 counters from gated to report-only in a CI budget file; no contract, field, key, or user-facing path is rejected, renamed, or removed.

Tests

No test is added: this is a CI-config (budget-list) change with no production code path to assert. Verified locally that perf-budgets.json validates as JSON and node scripts/bench-render.mjs --scenario s2 --runs 3 --check perf-budgets.json exits 0 with both counters printing (report only) while s2.commits/s2.renders stay gated.

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient. The only exercise is the benchmark's own --check, run locally and re-run by the CI bench-render job.

Screenshots / video

N/A — this diff touches no user-visible frontend surface (a CI budget JSON file only).

Related Issues

Closes #31963

Sibling sweep

Swept: git grep -nE '"report"' -- website/perf-budgets.json (the construct this change uses) and reviewed all four scenarios' gated counters for the same streamed-open-slot, frame-rate-bound pattern.
Sibling: website/perf-budgets.json:s1x5.selectorCalls -- already report-only -- the earlier instance of this same class (PR #29352).
Sibling: website/perf-budgets.json:s1x1 -- not affected -- its 1 background slot is not the open/active slot, so its flush count does not ride the open-slot notify path the way s2 does; it has not been observed to jump.

Pattern harvest

Not generalizable: this is a per-scenario benchmark-budget tuning decision (which counters are frame-rate-bound enough to be report-only on the pinned runner), judged case by case against observed run-to-run variance; there is no code defect class to lint for.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…nder benchmark

s2's dispatch count is frame-rate-bound: every extra dispatch is one extra per-animation-frame stream/flush (9 one-offs stay fixed). selectorCalls track it. This moves both counters to s2's report list, matching the existing s1x5.selectorCalls treatment; budget values and tolerance are unchanged.

Closes #31963
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The only change is to a CI perf-budget config file (website/perf-budgets.json). It adds or changes no user-visible control, so the missing blind read leaves no evidence gap.

[UX-REVIEWED] 25fc1ac

Description read

[DESCRIPTION-READ] 7e6d5d59c8d7abc4186264279d1654e3759480997948c9c7175fdd69efd7e841

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: it reuses the existing report mechanism for a counter that tracks frame rate, and leaves budgets, tolerance and the commits/renders gates unchanged.

Notes

  • This hides the noise instead of fixing it. A real dispatch regression in s2 (for example, a new per-chunk dispatch) is now ungated. A tighter fix would gate the non-stream/flush dispatches, the 9 fixed one-offs, so the steady part of the counter stays guarded. Better still, drive the benchmark's chunk clock and requestAnimationFrame from one clock, so stream/flush stops drifting.

[DESIGN-REVIEWED] 25fc1ac

Description read

[DESCRIPTION-READ] e54cde830e5033d8219fc647653c391d405888a333ad2d0f1acba91e38b0ddce

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 25fc1ac

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 25fc1ac

Verdict parsed from the review's SHA-scoped output markers for commit 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 25fc1ac1ad6b44e4ecbf4b1e25b8cbe4651f91c1 false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 marked this pull request as ready for review November 8, 2026 22:29
@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners November 8, 2026 22:29
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 8, 2026 23:58

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. This review checked head 25fc1ac. The diff is one line in website/perf-budgets.json. It adds "report": ["dispatches", "selectorCalls"] to s2 and leaves every budget and the tolerance unchanged. It uses the existing per-scenario report mechanism in bench-render.mjs the same way s1x5 already does. No blocking repo rule covers this file, and the CI render benchmark job is continue-on-error, so no merge outcome changes.

Done well: the body traces the drift to stream/flush firing on each animation frame while the benchmark ticks on a 100 ms wall clock. It backs that with a per-type dispatch breakdown and an 8-run reproduction on main. It does not raise the budget, which is what the issue asked.

Optional:

  • website/docs/testing.md:202 still says the only report-only counter is s1x5 selector calls. A short follow-up could name the s2 counters and the reason.
  • s2.dispatches never went over its limit (max observed 351 against 356). The prepare-pr gate floor runs --fail-on dispatches,selectorCalls, so after this change it checks no s2 counter at all. Keeping only selectorCalls in the list would keep one gated s2 counter there.
  • Closes #31963 will close the issue while the s1x5.renders jump reported in its comments is still open. Part of #31963 or a follow-up issue would keep that tracked.

@KiroCrewCaptain
KiroCrewCaptain merged commit c25ef64 into main Nov 8, 2026
110 of 111 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/31963-render-bench-flaky branch November 8, 2026 23:59
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: Render Benchmark s2 dispatches jump by ~40 in some runs

2 participants