Skip to content

ci: run the render benchmark with counter budgets - #29352

Merged
Pearcekieser merged 1 commit into
mainfrom
pearcekieser/perf-bench-ci-job
Nov 4, 2026
Merged

Pearcekieser merged 1 commit into
mainfrom
pearcekieser/perf-bench-ci-job

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: CI fails a pull request that makes the dashboard render or dispatch noticeably more than main does in the four replayed websocket scenarios.

#23431 added npm run bench:render and its budgets, but nothing runs it, so a regression in React commits, renders, store dispatches or selector calls lands unnoticed. The profiling build also hooked the store by rewriting exact strings in store/index.ts, which breaks without warning the first time someone edits those lines.

Why it matters

Render count regressions on the streaming paths are what made the dashboard sluggish with many slots open. A count that only a person running the script locally can see does not stop the next one.

Not a goal

  • Making the job a required check. It runs advisory until its numbers have held on main for a while.
  • Gating milliseconds. Script time and long tasks are still reported only.
  • Reducing any render count. This PR measures; it does not optimise.

What changed (motivation → approach → change)

store/index.ts now reads window.__kcPerfMw (a middleware that counts dispatches) and window.__kcSelWrap (a wrapper that counts selector calls) only when import.meta.env.VITE_KC_PERF === '1'. Only vite.profiling.config.mjs defines that flag, so a production build folds the branch away and contains none of the three hook names. The exact-string transform plugin is gone. Its error on a missed match is replaced by run guards in bench-render.mjs: a streaming scenario with no dispatches already failed, and any scenario with no selector calls now fails too, so a dropped seam cannot pass as a low count.

The bench-render job in ci.yml runs when a diff touches website/src/** (which includes store/index.ts) or the benchmark's own files. It runs on kirocrew-linux-4-core rather than through the fleet resolver, because commit counts depend on runner speed, and only in this repository. It uploads bench-render.json as an artifact even when it fails. The job carries continue-on-error: true, so a budget overrun turns its own check red without failing the CI run that PR Readiness reads; dropping that line is the step that makes it blocking.

The budgets are the medians of this branch's first 3 runs on that runner. Main has moved since #23431: idle commits went from 5 to 12, idle renders from 1,100 to 2,450, and idle no longer dispatches. On the runner, s2 commits ranged 3,053 to 3,069 and renders 258,364 to 258,820, so both are gated. s1x5 selector calls ranged 143,415 to 192,771, wider than the 15% tolerance, so a scenario may now list counters under report, which --check prints but never fails on, and that one counter is listed there.

test_prepare_pr_profiles.py requires every npm run script CI calls to be a kirocrew-prepare-pr gate, so profiles/kirocrew.json gains one. It runs npm --prefix website run bench:render -- --fail-on dispatches,selectorCalls when the diff since the merge base touches an input of CI's render_bench filter, and otherwise prints that it skipped. The browser it needs is already in the profile's setup[]. Commits and renders depend on host speed, since a faster host commits more often, so --fail-on (new in bench-render.mjs) keeps them report only on a contributor's machine. Dispatches and selector calls follow the frames sent: local and runner medians agreed within 4% for dispatches, and within 5% for selector calls wherever those are gated. website/docs/testing.md documents the seam, the job, the local gate and the report list.

Backwards compatibility

Compatible: the store's exported names and types are unchanged, and the seam is absent outside the profiling build. A budget file without report gates every counter, as before.

Removes nothing: the only removed code is the profiling config's string-rewrite plugin, whose sole reader was the profiling build, which now uses the seam.

Tests

Proof: not provable -- docs or config only

  • src/test/renderProbe.test.ts and the src/store tests pass (711 tests), as does the frontend related set (486 tests).
  • test/test_prepare_pr_profiles.py, test/test_ci_nonroot_boundary.py, test/test_ci_fleet_routing_expression_parity.py, test/test_local_gate.py and the flake-pattern ratchet pass (212 tests), as do the kirocrew-prepare-pr suites that read the profile and gate map (test_prepare_pr_profiles.py, test_gate_map_covers_ci.py, test_prepare_pr_pre_push_gate.py, test_prepare_pr_local_review.py, test_governance_profile_reprojection.py; 432 tests).

Manual verification

  • The bench-render job on the first head passed in 9 minutes. Its three runs per scenario are the budgets above, and every gated median is within the tolerance.
  • The new profile gate, run as written from the worktree root, built the profiling bundle and passed, printing commits and renders as report only. --fail-on dispatches,bogus stops before the browser starts and names the unknown counter.
  • Locally, node scripts/bench-render.mjs --runs 3 counts dispatches and selector calls through the new seam in every streaming scenario. Local s2 commits ranged 2,053 to 3,141 across runs on a shared host, against 3,053 to 3,069 on the runner, which is why the local gate does not fail on commits.
  • A production vite build contains no __kcPerfMw, __kcSelWrap or VITE_KC_PERF; the profiling bundle's store chunk does.

Related Issues

Fixes #23430

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners November 3, 2026 22:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e3d6b1388bbc11453268155c6ebb9d6af47e8526 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces to #23430 or to a test the profile must satisfy. The one leftover is a second "don't fail this counter" switch with only one caller.

Notes

  • Item 5 — one consumer, generalized: --fail-on a,b takes any list of counters, but its only caller (the profile gate) always passes dispatches,selectorCalls. There is already an "exempt this counter" mechanism, the per-scenario report list in perf-budgets.json. Grep for fail-on found 1 non-doc caller. Subtraction: shrink --fail-on to a boolean flag that fixes the host-independent pair inside bench-render.mjs, and drop the free-form parsing and the unknown-counter error.

What this change ships

Inventory (8 items) — 7 justified

Intent: CI flags a PR that makes the dashboard render or dispatch noticeably more than main does. This is an ADDITION.

  1. A new advisory bench-render CI job, pinned to one runner, uploads a report — justified
  2. The store's benchmark hooks now live in source behind VITE_KC_PERF; the string-rewrite plugin is deleted — justified
  3. The render budgets are re-measured on the CI runner — justified
  4. A budget scenario can list counters under report (print only, never fail) — justified
  5. A --fail-on flag limits which counters can fail — one consumer, generalized
  6. A kirocrew-prepare-pr gate runs the benchmark when its inputs change — justified
  7. A run with zero selector calls now fails as a dropped hook — justified
  8. testing.md documents the hooks, the job and the local gate — justified

[FIRST-PRINCIPLES-REVIEWED] e3d6b13

Description read

[DESCRIPTION-READ] 277919574dafa7e502dce39dfb4a86e7091e5efcf47903a91bb4055c640bb520

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e3d6b1388bbc11453268155c6ebb9d6af47e8526 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e3d6b13

Verdict parsed from the review's SHA-scoped output markers for commit e3d6b1388bbc11453268155c6ebb9d6af47e8526.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e3d6b1388bbc11453268155c6ebb9d6af47e8526 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of e3d6b1388bbc11453268155c6ebb9d6af47e8526 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The PR adds no user-visible control or text. It changes only the CI benchmark, its budgets and docs, and a store hook that exists only in the profiling build, so the missing blind read leaves nothing unreviewed.

[UX-REVIEWED] e3d6b13

Description read

[DESCRIPTION-READ] e648aead3c231bf4f86455d8213851c4c2fd055841c092adb2cb0c8b321ee0bf

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e3d6b1388bbc11453268155c6ebb9d6af47e8526 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only measures, the CI job is advisory, and every part can be reverted cleanly.

Notes

  • The CI job is advisory until its budgets hold on main, but the new kirocrew-prepare-pr gate fails a contributor locally on dispatches,selectorCalls today. Those budgets come from three runs on one runner, and local runs matched them within 4–5% against a 15% tolerance. Either make the local gate report-only too, or say in the description that it is meant to fail before CI's job does.
  • The local gate's path list copies CI's render_bench filter by hand (git diff --quiet "$BASE" -- website/src …). The two lists will drift apart without anyone noticing unless a test pins them equal or both read one shared list.
  • The local gate builds the profiling bundle and runs Playwright on every diff under website/src/**, which covers almost every frontend PR. That adds minutes to the local gate for the commonest kind of change.

[DESIGN-REVIEWED] e3d6b13

Description read

[DESCRIPTION-READ] af0de1a0b4796b17a5a9c88a0abd8913eb2ef089713bb6bdc3b2781bfeb53605

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e3d6b1388bbc11453268155c6ebb9d6af47e8526 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e3d6b13

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e3d6b1388bbc11453268155c6ebb9d6af47e8526 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 3, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-bench-ci-job branch from 7b08597 to d6f5e18 Compare November 4, 2026 00:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 4, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-bench-ci-job branch from d6f5e18 to 103af4b Compare November 4, 2026 03:12
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • span=3c22e0830ecc fixed in 103af4b

The exemption set is gone. test/test_prepare_pr_profiles.py matches origin/main again, so the npm-script assertion covers every script CI runs, bench:render included.
profiles/kirocrew.json gains a gate that runs npm --prefix website run bench:render when the diff since the merge base touches an input of the CI render_bench filter, and prints that it skipped otherwise. The browser it needs is already in setup[].

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 4, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-bench-ci-job branch from 103af4b to 4780808 Compare November 4, 2026 03:42
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • span=582dc586b8f3 fixed in 4780808

bench-render.mjs gains --fail-on, which names the counters --check may fail on and prints the rest as report only. An unknown counter name throws before the browser starts.
The prepare-pr profile gate now runs bench:render with --fail-on dispatches,selectorCalls, so host-speed-dependent commits and renders never fail a contributor run. Dispatches and selector calls follow the frames sent and matched the runner within 5% where gated.
website/docs/testing.md and the script header now say a faster host counts more commits and renders, not fewer.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 4, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-bench-ci-job branch from 4780808 to 54a0efe Compare November 4, 2026 03:57
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • span=93fb6e0bcb97 fixed in 54a0efe

The bench-render job now carries job-level continue-on-error: true. A budget overrun reddens its own check and keeps the bench-render-report artifact, but no longer fails the CI run that pr-readiness.yml reads, so it blocks no merge.
The job comment and website/docs/testing.md say it is report-only and that dropping continue-on-error is the follow-up once the budgets have held on main.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 4, 2026
Add a bench-render job that runs npm run bench:render on diffs touching
website/src or the benchmark's files, pinned to kirocrew-linux-4-core,
and uploads the JSON report. The kirocrew-prepare-pr gate floor runs
the same script when the diff touches the same inputs.

Replace the profiling build's exact-string store patch with a real seam
in store/index.ts (window.__kcPerfMw, window.__kcSelWrap), gated on
VITE_KC_PERF so other builds drop it. A run now fails when it counts no
selector calls, so a dropped seam cannot pass as a low count.

Set the budgets from the runner's 3-run medians. s1x5 selector calls
vary by more than the tolerance there, so they are report-only.

Part of #23430
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/perf-bench-ci-job branch from 54a0efe to e3d6b13 Compare November 4, 2026 06:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 4, 2026
@Pearcekieser
Pearcekieser enabled auto-merge (squash) November 4, 2026 07:45

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at e3d6b13. I read all 7 files.

  • Store seams: website/src/store/index.ts reads the profiling hooks only when import.meta.env.VITE_KC_PERF === '1', which only the profiling build config sets, so the production bundle behaves as before (Bundle Size Gate is green). This replaces the old string rewrite in the profiling build, and the PR removes that rewrite's pattern.
  • Run guards: the new checks in bench-render.mjs make a broken seam fail the run instead of counting zero.
  • Workflow change, read line by line: the new bench-render job pins every action by sha, checks out with persist-credentials: false, uses no secrets, widens no permissions, adds no trigger, and runs in this repository only. It is continue-on-error, so it reports without blocking a merge, as described. The new render_bench paths-filter bucket only chooses when the job runs.

The benchmark job ran and passed on this head, all AI lanes pass, and all 96 checks are green.

@Pearcekieser
Pearcekieser merged commit 590ff1d into main Nov 4, 2026
97 checks passed
@Pearcekieser
Pearcekieser deleted the pearcekieser/perf-bench-ci-job branch November 4, 2026 09:20
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Web perf: gateway-free render benchmark with counter budgets

2 participants