Repository navigation
test(website): mark s2 dispatches and selectorCalls report-only in render benchmark - #33444
Conversation
…nder benchmark s2's dispatch count is frame-rate-bound: every extra dispatch is one extra per-animation-frame stream/flush (9 one-offs stay fixed). selectorCalls track it. This moves both counters to s2's report list, matching the existing s1x5.selectorCalls treatment; budget values and tolerance are unchanged. Closes #31963
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The only change is to a CI perf-budget config file ( [UX-REVIEWED] 25fc1ac Description read[DESCRIPTION-READ] 7e6d5d59c8d7abc4186264279d1654e3759480997948c9c7175fdd69efd7e841 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: it reuses the existing Notes
[DESIGN-REVIEWED] 25fc1ac Description read[DESCRIPTION-READ] e54cde830e5033d8219fc647653c391d405888a333ad2d0f1acba91e38b0ddce That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving. This review checked head 25fc1ac. The diff is one line in website/perf-budgets.json. It adds "report": ["dispatches", "selectorCalls"] to s2 and leaves every budget and the tolerance unchanged. It uses the existing per-scenario report mechanism in bench-render.mjs the same way s1x5 already does. No blocking repo rule covers this file, and the CI render benchmark job is continue-on-error, so no merge outcome changes.
Done well: the body traces the drift to stream/flush firing on each animation frame while the benchmark ticks on a 100 ms wall clock. It backs that with a per-type dispatch breakdown and an 8-run reproduction on main. It does not raise the budget, which is what the issue asked.
Optional:
website/docs/testing.md:202still says the only report-only counter iss1x5selector calls. A short follow-up could name thes2counters and the reason.s2.dispatchesnever went over its limit (max observed 351 against 356). The prepare-pr gate floor runs--fail-on dispatches,selectorCalls, so after this change it checks nos2counter at all. Keeping onlyselectorCallsin the list would keep one gateds2counter there.Closes #31963will close the issue while thes1x5.rendersjump reported in its comments is still open.Part of #31963or a follow-up issue would keep that tracked.
Problem / Motivation
Goal: Stop the Render Benchmark
s2scenario from failing CI intermittently when no frontend change caused it.The
Render Benchmark (counter budgets)job failss2.selectorCalls(and sits near thes2.dispatcheslimit) on diffs that never touch the chat page. In some runs the store sees ~25-45 more dispatches than the usual ~309-312, selector calls follow, and when the median crosses the limit the job goes red (CI has observeddispatches=351).Why it matters
A benchmark that red-fails on unrelated PRs blocks and distracts every contributor whose diff happens to land in a high-variance window. The jitter is not a performance regression, so the failures are pure noise that erodes trust in the gate and costs reruns.
Not a goal
bench-render.mjs, or any scenario other thans2.What changed (motivation → approach → change)
Observed symptom:
s2.dispatches(and the dependents2.selectorCalls) spike in some runs with no related diff.Root cause: the s2 dispatch count decomposes into 9 fixed one-off dispatches (
setSlotStatusDetail×2,sseChatMessage:user,setVoicePlaying,syncSlotRunningFromServer,setSidebarOrder,sseChatMessage:_done,refreshSlot/pending,refreshSlot/fulfilled) plusstream/flush, nominally 300 (one per 100 ms tick) — so a clean run is 309. Every extra dispatch is one extrastream/flush: confirmed by per-type breakdown, when total dispatches rose 309 → 310 thestream/flushcount rose 300 → 301 and the 9 one-offs stayed constant.stream/flushis dispatched once per animation frame by the streaming-chunk coalescer (src/hooks/websocket/streamBuffers.ts); the benchmark delivers the prose stream on a 100 ms wall clock while the flush fires onrequestAnimationFrame, so a tick's chunks can split across two rAF flushes when frames render fast. Under runner load the drift accumulates (~40). Each open-slot flush notifies immediately, soselectorCallstrackdispatches.8-run reproduction on
main(no code change):Approach / change:
perf-budgets.json's own note defines the per-scenarioreportlist for exactly this case — a counter that varies between runs on the runner by more than the tolerance is printed but never fails the check. The same pattern was already applied tos1x5.selectorCalls. This addsdispatchesandselectorCallstos2'sreportlist. Budget values and tolerance are unchanged.Reviewer note: with this change,
s2.dispatchesands2.selectorCallsbecome report-only — still printed every run, but a real regression in those two s2 counters is no longer gated by--check.s2.commitsands2.rendersremain gated.Backwards compatibility
Removes nothing: the change only moves two s2 counters from gated to report-only in a CI budget file; no contract, field, key, or user-facing path is rejected, renamed, or removed.
Tests
No test is added: this is a CI-config (budget-list) change with no production code path to assert. Verified locally that
perf-budgets.jsonvalidates as JSON andnode scripts/bench-render.mjs --scenario s2 --runs 3 --check perf-budgets.jsonexits 0 with both counters printing(report only)whiles2.commits/s2.rendersstay gated.Proof: not provable -- test-only change
Manual verification
N/A — unit coverage sufficient. The only exercise is the benchmark's own
--check, run locally and re-run by the CI bench-render job.Screenshots / video
N/A — this diff touches no user-visible frontend surface (a CI budget JSON file only).
Related Issues
Closes #31963
Sibling sweep
Swept:
git grep -nE '"report"' -- website/perf-budgets.json(the construct this change uses) and reviewed all four scenarios' gated counters for the same streamed-open-slot, frame-rate-bound pattern.Sibling:
website/perf-budgets.json:s1x5.selectorCalls-- already report-only -- the earlier instance of this same class (PR #29352).Sibling:
website/perf-budgets.json:s1x1-- not affected -- its 1 background slot is not the open/active slot, so its flush count does not ride the open-slot notify path the way s2 does; it has not been observed to jump.Pattern harvest
Not generalizable: this is a per-scenario benchmark-budget tuning decision (which counters are frame-rate-bound enough to be report-only on the pinned runner), judged case by case against observed run-to-run variance; there is no code defect class to lint for.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)