Repository navigation
ci: run the render benchmark with counter budgets - #29352
Conversation
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every item traces to #23430 or to a test the profile must satisfy. The one leftover is a second "don't fail this counter" switch with only one caller. Notes
What this change shipsInventory (8 items) — 7 justifiedIntent: CI flags a PR that makes the dashboard render or dispatch noticeably more than main does. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] e3d6b13 Description read[DESCRIPTION-READ] 277919574dafa7e502dce39dfb4a86e7091e5efcf47903a91bb4055c640bb520 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The PR adds no user-visible control or text. It changes only the CI benchmark, its budgets and docs, and a store hook that exists only in the profiling build, so the missing blind read leaves nothing unreviewed. [UX-REVIEWED] e3d6b13 Description read[DESCRIPTION-READ] e648aead3c231bf4f86455d8213851c4c2fd055841c092adb2cb0c8b321ee0bf That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only measures, the CI job is advisory, and every part can be reverted cleanly. Notes
[DESIGN-REVIEWED] e3d6b13 Description read[DESCRIPTION-READ] af0de1a0b4796b17a5a9c88a0abd8913eb2ef089713bb6bdc3b2781bfeb53605 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
7b08597 to
d6f5e18
Compare
d6f5e18 to
103af4b
Compare
|
103af4b to
4780808
Compare
|
4780808 to
54a0efe
Compare
|
Add a bench-render job that runs npm run bench:render on diffs touching website/src or the benchmark's files, pinned to kirocrew-linux-4-core, and uploads the JSON report. The kirocrew-prepare-pr gate floor runs the same script when the diff touches the same inputs. Replace the profiling build's exact-string store patch with a real seam in store/index.ts (window.__kcPerfMw, window.__kcSelWrap), gated on VITE_KC_PERF so other builds drop it. A run now fails when it counts no selector calls, so a dropped seam cannot pass as a low count. Set the budgets from the runner's 3-run medians. s1x5 selector calls vary by more than the tolerance there, so they are report-only. Part of #23430
54a0efe to
e3d6b13
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Approving at e3d6b13. I read all 7 files.
- Store seams:
website/src/store/index.tsreads the profiling hooks only whenimport.meta.env.VITE_KC_PERF === '1', which only the profiling build config sets, so the production bundle behaves as before (Bundle Size Gate is green). This replaces the old string rewrite in the profiling build, and the PR removes that rewrite's pattern. - Run guards: the new checks in
bench-render.mjsmake a broken seam fail the run instead of counting zero. - Workflow change, read line by line: the new
bench-renderjob pins every action by sha, checks out withpersist-credentials: false, uses no secrets, widens no permissions, adds no trigger, and runs in this repository only. It iscontinue-on-error, so it reports without blocking a merge, as described. The newrender_benchpaths-filter bucket only chooses when the job runs.
The benchmark job ran and passed on this head, all AI lanes pass, and all 96 checks are green.
Problem / Motivation
Goal: CI fails a pull request that makes the dashboard render or dispatch noticeably more than main does in the four replayed websocket scenarios.
#23431 added
npm run bench:renderand its budgets, but nothing runs it, so a regression in React commits, renders, store dispatches or selector calls lands unnoticed. The profiling build also hooked the store by rewriting exact strings instore/index.ts, which breaks without warning the first time someone edits those lines.Why it matters
Render count regressions on the streaming paths are what made the dashboard sluggish with many slots open. A count that only a person running the script locally can see does not stop the next one.
Not a goal
What changed (motivation → approach → change)
store/index.tsnow readswindow.__kcPerfMw(a middleware that counts dispatches) andwindow.__kcSelWrap(a wrapper that counts selector calls) only whenimport.meta.env.VITE_KC_PERF === '1'. Onlyvite.profiling.config.mjsdefines that flag, so a production build folds the branch away and contains none of the three hook names. The exact-string transform plugin is gone. Its error on a missed match is replaced by run guards inbench-render.mjs: a streaming scenario with no dispatches already failed, and any scenario with no selector calls now fails too, so a dropped seam cannot pass as a low count.The
bench-renderjob inci.ymlruns when a diff toucheswebsite/src/**(which includesstore/index.ts) or the benchmark's own files. It runs onkirocrew-linux-4-corerather than through the fleet resolver, because commit counts depend on runner speed, and only in this repository. It uploadsbench-render.jsonas an artifact even when it fails. The job carriescontinue-on-error: true, so a budget overrun turns its own check red without failing the CI run that PR Readiness reads; dropping that line is the step that makes it blocking.The budgets are the medians of this branch's first 3 runs on that runner. Main has moved since #23431: idle commits went from 5 to 12, idle renders from 1,100 to 2,450, and idle no longer dispatches. On the runner,
s2commits ranged 3,053 to 3,069 and renders 258,364 to 258,820, so both are gated.s1x5selector calls ranged 143,415 to 192,771, wider than the 15% tolerance, so a scenario may now list counters underreport, which--checkprints but never fails on, and that one counter is listed there.test_prepare_pr_profiles.pyrequires everynpm runscript CI calls to be a kirocrew-prepare-pr gate, soprofiles/kirocrew.jsongains one. It runsnpm --prefix website run bench:render -- --fail-on dispatches,selectorCallswhen the diff since the merge base touches an input of CI'srender_benchfilter, and otherwise prints that it skipped. The browser it needs is already in the profile'ssetup[]. Commits and renders depend on host speed, since a faster host commits more often, so--fail-on(new inbench-render.mjs) keeps them report only on a contributor's machine. Dispatches and selector calls follow the frames sent: local and runner medians agreed within 4% for dispatches, and within 5% for selector calls wherever those are gated.website/docs/testing.mddocuments the seam, the job, the local gate and thereportlist.Backwards compatibility
Compatible: the store's exported names and types are unchanged, and the seam is absent outside the profiling build. A budget file without
reportgates every counter, as before.Removes nothing: the only removed code is the profiling config's string-rewrite plugin, whose sole reader was the profiling build, which now uses the seam.
Tests
Proof: not provable -- docs or config only
src/test/renderProbe.test.tsand thesrc/storetests pass (711 tests), as does the frontend related set (486 tests).test/test_prepare_pr_profiles.py,test/test_ci_nonroot_boundary.py,test/test_ci_fleet_routing_expression_parity.py,test/test_local_gate.pyand the flake-pattern ratchet pass (212 tests), as do the kirocrew-prepare-pr suites that read the profile and gate map (test_prepare_pr_profiles.py,test_gate_map_covers_ci.py,test_prepare_pr_pre_push_gate.py,test_prepare_pr_local_review.py,test_governance_profile_reprojection.py; 432 tests).Manual verification
bench-renderjob on the first head passed in 9 minutes. Its three runs per scenario are the budgets above, and every gated median is within the tolerance.--fail-on dispatches,bogusstops before the browser starts and names the unknown counter.node scripts/bench-render.mjs --runs 3counts dispatches and selector calls through the new seam in every streaming scenario. Locals2commits ranged 2,053 to 3,141 across runs on a shared host, against 3,053 to 3,069 on the runner, which is why the local gate does not fail on commits.vite buildcontains no__kcPerfMw,__kcSelWraporVITE_KC_PERF; the profiling bundle's store chunk does.Related Issues
Fixes #23430
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)