Skip to content

perf(website): gateway-free render benchmark with counter budgets - #23431

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/perf-bench-ci
Oct 31, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/perf-bench-ci

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Anyone can measure, before and after a change, how many React commits, renders, store dispatches and selector calls the dashboard spends on streaming traffic, with one command and no gateway.

CI checks the web bundle's size and nothing about render work. The only render measurements so far came from scripts that drive a live pod with a seeded home, so they cannot run in CI and are awkward to rerun for a perf PR.

Why it matters

Several perf fixes are queued for the streaming path: composer re-renders, store equality guards, the active-stream cascade and markdown parsing. Without a shared benchmark, each PR measures its own way and nothing stops the next regression. One script with fixed scenarios gives every one of those PRs a before and after number, and the budget file gives a CI job something to check.

Not a goal

  • The CI job that runs the check. It follows in a separate PR, so perf work can use the script now.
  • Any change to app code. The only store change lives in the profiling build's transform. The prod build and website/dist are untouched.
  • Wall-time or frame-rate budgets. They depend on the runner.
  • Fixing anything the benchmark finds.

What changed (motivation → approach → change)

npm run bench:render builds the SPA with react-dom/profiling into node_modules/.cache/kc-perf/dist, then runs scripts/bench-render.mjs against it. The build config adds two store seams to that build only: a middleware that counts dispatches and a useAppSelector wrapper that counts selector calls. The script serves the build with serve-dist and answers /api/** with stub-dashboard-api, using a seeded world from lib/perf-fixture.mjs: 222 sessions in 8 folders, a 100-row open transcript, and the command bar app enabled as on a default install. The stub gains an onWebSocket option, so the script can push frames into the mocked dashboard socket. Four scenarios, idle, s1x1, s1x5 and s2, each send 300 ticks of 100 ms. They carry agentic background turns or a prose answer in the open chat, plus the gateway's 5 s status frame, which keeps the 20 s silence watchdog from reconnecting.

lib/render-probe.mjs runs in the page. It walks each commit and attributes every render to a mount, a changed hook or selector, a context, changed props, or a parent re-render. A run is refused when the page logs an error, when no chunk reached the store, or when the streamed answer stays short, because each of those lowers the counts and would pass any upper bound. --check perf-budgets.json compares the 3-run medians with the budgets and exits 1 when a counter is more than 15% over. package.json gains the two bench:render scripts and no dependencies; Playwright and react-dom are already there.

Backwards compatibility

Compatible: stubDashboardApi's new onWebSocket option defaults to the old no-op, so every existing capture script behaves as before.

Removes nothing: the diff only adds files, one npm script pair, one stub option and one docs section.

Tests

src/test/renderProbe.test.ts mounts an 8-component tree on react-dom/profiling under the probe and drives 3 updates. It asserts the cause the probe reports for each component: parent(same props), props:o~ for a deep-equal object, props:fƒ for a new function, props:v! for a changed value, ctx:TestCtx!, uSES#0 [snap:getSnap]! for a store read, useState#0!, and no render for a memo child whose props are unchanged. A second test checks that nothing is counted while counting is off. Six runs with --sequence.shuffle, 12 tests, all passed. An earlier version that loaded React once per test failed 4 of 5 shuffled runs, because react-dom keeps the DevTools hook it saw at first load. The file now loads React once.

Manual verification

These are 3-run medians on this host, desktop 1440x900. The reference column is the pod profiling run this script replaces, measured at a9898a0.

Scenario, counter pod run at a9898a0 this script at a9898a0 this script on main
s1x5 dispatches 3,102 3,165 3,173
s1x5 selector calls 1,853,817 1,819,815 1,913,509
s1x5 commits 219 68 * 248
s1x1 dispatches 631 641 647
s1x1 selector calls 375,174 367,867 389,303
s2 dispatches 314 311 320
s2 selector calls 241,307 221,877 244,118
s2 commits 3,643 2,799 3,068

* The a9898a0 column ran on an earlier fixture, before two changes. First, every streaming row sat below the fold there. The sidebar does not re-render an offscreen row on its slot's status, so 180 status commits were missing. The pod home had one of the five rows on screen, so the fixture now pins perf-s000 in the first folder. Second, the fixture now sends the 5 s status frame, which adds 6 dispatches per scenario. The main column has both changes. On main, s1x5 commits come out 13% above the pod run. Most of the extra 29 commits are idle background work, react-query observer updates against the stub, which also show in idle: 29 to 31 commits here against 10 on the pod.

Dispatches and selector calls land within 4% of the pod run on main. s2 commits vary 2,719 to 3,579 between runs on one host, because they follow the frame rate. The s2 budget allows for that. npm run bench:render passes on main. The same run against a budget file with idle.commits: 20 exits 1 with Over budget by more than 15%: idle.commits.

The selector-counting wrapper is cached per selector, because react-redux 9 memoizes a selector's result by the selector's identity and a new wrapper on every render discarded that memo. After that fix, a 3-run re-measure on this host gave idle 5,162 selector calls, s1x1 389,618, s1x5 1,917,871 and s2 256,678, and perf-budgets.json holds budgets from that run. s2 commits were 3,545 to 3,638 and its renders 661,034 to 673,584 in the same runs.

Rebased on main at 2719ba9, which carries several store-identity and React Compiler PRs. The counts dropped, so perf-budgets.json now holds a fresh 3-run baseline from that main: idle 4 commits, 1,059 renders, 3,777 selector calls; s1x1 27, 17,393, 287,740; s1x5 207, 21,306, 454,648; s2 3,121, 390,738, 228,966. Dispatches did not move. The s2 commit budget stays at 3,450 to absorb its frame-rate spread.

A mobile run (--viewport mobile --cpu 4, s1x5 and s2) completes. It waits on the store rather than a sidebar row, because the mobile sidebar is a closed drawer. At --cpu 4, s2 sends 60 ticks and gets 41 dispatches, because chunks that land in one frame flush together. The run completes. The 29 vitest files that reference a changed path pass, 854 tests.

semgrep is not installed on this host, so the repo's semgrep rules were not run locally. CI's SAST job runs them.

Related Issues

Part of #23430. The issue stays open for the CI job that runs npm run bench:render, including how that job handles the s2 frame-rate spread and the store/index.ts string patch.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners October 23, 2026 10:31
@Pearcekieser
Pearcekieser requested a review from buluoray October 23, 2026 10:31
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision ce908d2bf2e78809f95124bdba341278c2fc7ee1 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed ce908d2bf2e78809f95124bdba341278c2fc7ee1 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ce908d2

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed ce908d2bf2e78809f95124bdba341278c2fc7ee1 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ce908d2

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

question (blocking): PR Readiness has waited on Design Review since 13:03Z with the status "1 readiness check(s) still pending; waiting on Design Review (not started)".

Every other lane finished on head a871e52. Opus and GPT posted no blocking findings, First Principles posted a skip, and CI, Fast Gate and UX Review are green. The Fork Design Review workflow starts only from a Fast Gate workflow_run and has no manual dispatch, so I can't restart it from the fork side. Could a maintainer re-run the Fork Design Review run for this head, or record a design override if the lane has nothing to judge here? The diff adds a benchmark script, its fixture and probe, a budget file, a test and a docs section. It changes no app code.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): pushed f382d48, a same-tree retrigger of a871e52 with only the commit date changed, because Fork Design Review never started on the previous head.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ✅ PASS

UX-level review of ce908d2bf2e78809f95124bdba341278c2fc7ee1 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff only adds dev tooling: a profiling build, a benchmark script, test fixtures, a test and a docs section. It adds or changes no dashboard control, string or state a user sees.

[UX-REVIEWED] ce908d2

Description read

[DESCRIPTION-READ] e6c6e93bce32e7a81d2405a72b0623beff9f19cbb2ddbaab476a210ff34634e4

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of ce908d2bf2e78809f95124bdba341278c2fc7ee1 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this adds a dev-only benchmark with no app-code or UI change, and the one shared-helper change (onWebSocket) defaults to the old no-op.

Notes

  • perf-budgets.json doesn't match the baseline in the description, which says "s1x1 27, 17,393, 287,740; s1x5 207, 21,306, 454,648". The file caps s1x1 at commits: 10, renders: 6000, selectorCalls: 199200 and s1x5 at renders: 9800, selectorCalls: 219000. Allowing 15% over, every one of those stated medians is still over its cap. So either npm run bench:render fails on main, which contradicts the claim that it passes there, or the file came from a later run the description doesn't mention. Re-measure and make the file and the description agree before the follow-up CI job relies on these numbers.
  • The store/index.ts seams are injected by matching exact source text. The build fails loudly when that text drifts, which is the right failure. A single explicit window.__kcPerfMw/__kcSelWrap hook point in the store would remove the text matching, but that touches app code, so it belongs in the Web perf: gateway-free render benchmark with counter budgets #23430 follow-up, not here.

[DESIGN-REVIEWED] ce908d2

Description read

[DESCRIPTION-READ] 69051ad4a751a71a334ea0ce736fb9aab16d5d1c798e865424e914ea53c2d3e1

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 24, 2026
@Pearcekieser

Pearcekieser commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: per-selector wrapper cache in the profiling store seam

  • span=5ec268dc1ff2 selector wrapper identity

issue (blocking): fixed in 938e295.

fixed: __kcSelWrap now returns one cached wrapper per selector from a WeakMap, so react-redux 9 keeps its selector memo across renders.
A new test in src/test/renderProbe.test.ts asserts the same selector gets the same wrapper and a different selector a different one.
Re-measured 3 runs: idle selector calls 5,272 to 5,162, streaming scenarios under 1% change; budgets in perf-budgets.json re-based on that run.

@Pearcekieser

Pearcekieser commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: per-selector wrapper cache in the profiling store seam

  • span=4b3d3fbf8b96 selector wrapper identity

issue (blocking): fixed in 938e295.

fixed: __kcSelWrap caches one wrapper per original selector in a WeakMap, so a stable selector keeps its identity and react-redux's reference cache holds.
src/test/renderProbe.test.ts asserts wrapper identity per selector.
selectorCalls re-measured after the fix and perf-budgets.json re-based (idle 5,200; s2 250,000).

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): agreed. App code and the prod build are unchanged; the store seams exist only in the profiling build under node_modules/.cache/kc-perf/dist, and stubDashboardApi's new onWebSocket option defaults to the old no-op.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): agreed. The diff adds developer tooling only and changes no rendered UI, so there is nothing to screenshot.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 24, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): rebased on main, no behaviour change. Main had changed website/package.json since this branch's green, and UX Review never started on 938e295, so this push re-runs every lane.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): rebased on main 2719ba9 and re-baselined the budgets.

Main had changed website/package.json since the last green. After the rebase, main's counts were well below the old budgets because several store-identity and React Compiler PRs had landed, so a large regression would still have passed. perf-budgets.json now holds a fresh 3-run baseline from that main:

Scenario Commits Renders Dispatches Selector calls
idle 4 1,059 6 3,777
s1x1 27 17,393 647 287,740
s1x5 207 21,306 3,179 454,648
s2 3,121 390,738 316 228,966

The s2 commit budget stays at 3,450 to allow for its frame-rate spread. The benchmark code is unchanged, and the answers to the earlier review still apply.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 25, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): exact-string store seams in store/index.ts.

Deferred to the CI-job PR, now item 3 in #23430. That PR adds a real no-op seam to store/index.ts, a window.__kcPerfMw middleware lookup and a __kcSelWrap wrapper that do nothing when unset.
This PR stays out of app code, which is why it uses the build-time patch. The patch throws a named error when either string misses, so until the seam lands a store refactor breaks the bench build loudly instead of counting wrongly.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): s2 commits on a slower CI runner.

Agreed. Budgets are upper bounds and a slower host batches more frames per commit, so s2 commits fall there and could hide a regression of under 15%.
Nothing gates these budgets in this PR. #23430 item 4 requires the CI job to pin the runner class the budgets were measured on, or to gate s2 on dispatches and selector calls only. Those two varied under 4% between runs here.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

question (non-blocking): could a maintainer re-run Cross-Platform Portability run 40175055240 for 849dfe9? It was cancelled during actions/checkout at 08:02Z, before any test ran, and a fork account needs admin rights to re-run it. The rest of the PR has not changed since the 21:13Z answers to the changes-requested review.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): rebased on main 6785e25, the run check now accepts batched stream frames, and the budgets are re-baselined.

Since #23712, main delivers each frame of stream traffic as one stream/flush action, so the old chat/sseChatMessage:chunk count was 0. The run check required that one action type, so it refused every streaming run on current main. It now accepts any dispatch, whichever action carries the frames. The existing store-state checks still reject a run whose frames never reached the store: background rows in chat.slotMessages, and the streamed answer length for s2.

The rebase also re-runs the lanes that were stale or incomplete on 849dfe9: the green that predated a website/package.json change on main, the UX Review that never started, and the Cross-Platform Portability run cancelled during checkout.

Fresh 3-run medians on 6785e25, now in perf-budgets.json:

Scenario Commits Renders Dispatches Selector calls
idle 4 1,049 6 3,776
s1x1 9 5,981 492 196,156
s1x5 189 9,780 1,348 218,949
s2 3,162 385,032 315 227,357

Batching cut s1x5 dispatches from 3,179 to 1,348 and its selector calls from 454,648 to 218,949. The answers to the earlier review still stand.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 25, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

question (non-blocking): could a maintainer re-run the fork Security Scope Review for ce908d2? Its check (run 121263291982) reports [scope-floor:unsettled]: the generate job did not finish, so no review leg ran. The same-repo Security Scope candidates job on this head passed. Re-running needs maintainer rights from a fork.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

question (non-blocking): two maintainer actions still block this PR on ce908d2. First, the fork Security Scope Review needs a re-run: its setup job never finished, so it reports scope-floor:unsettled and readiness waits on it. Second, the changes-requested review needs a fresh look. The 2026-10-24 21:13Z replies answer both of its points, linking #23430 and replying to each Design note, and the code has not changed since. Nothing else is pending from my side.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 30, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at ce908d2.

The blocker from my r124 review is cleared: Related Issues now links #23430, and its "Remaining after #23431" section covers the CI job, the s2 frame-rate spread, the exact-string store patch and runner pinning. The benchmark is dev-only. The store seams exist only in the profiling build and fail the build loudly on drift (vite.profiling.config.mjs L31-L53), and both matched strings are still present in current main. Runs that never reached the store are refused (bench-render.mjs L168-L176), and the static server binds 127.0.0.1 (serve-dist.mjs L78). Every check on this head is green, including the re-run Security Scope Review, and the branch merges cleanly into main.

AI findings judged non-blocking

  • Design Review, "perf-budgets.json does not match the description's baseline": the file matches your 2026-10-25 re-baseline on 6785e25 (perf-budgets.json). Only the Manual verification paragraph still quotes the older 2719ba9 numbers, and nothing gates the budgets yet. Please update that paragraph so the squash message is accurate.

Design's second note (the exact-string store patch) is tracked as item 3 in #23430.

— crew-pr-reviewer (r137)

@iamwhatever
iamwhatever merged commit ca1f4da into kirodotdev:main Oct 31, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants