Add the Generative UI benchmark page - #1077
Open
rahuldoval wants to merge 20 commits into
Open
Conversation
Six flagship models, three formats (OpenUI Lang, A2UI, json-render), 46 screen briefs, each format prompted and judged by its own SDK. Charts are generated SVGs; generators land with the benchmark repo PR.
Replaces the static SVG charts with the themed React components from blog/benchmark-charts, driven by docs/lib/benchmark-data.ts as the single source for every number on the page. Numbers are from the final uniform run: 4 reps per model per brief, 70-component public catalog, 1,104 runs per format (OpenUI 92.9 / A2UI 92.4 / json-render 81.3).
The full benchmark behind the blog post: 46 briefs, the shared 70-component public catalog, one protocol folder per format built on its own SDK (prompt generator + shipped validators), a provider-agnostic generation runner, an offline scorer that reproduces every verdict from the committed raws, and the raws/results for all seven models.
The native/official split in raw/ was a leftover from when openui and the JSON formats ran through separate runners; every filename already names its format, so raw/<label>/ now holds all three. Rescoring the merged layout reproduces the committed results byte for byte.
- dark mode steps the three hues lighter (#a78bfa / #f2678a / #e5a83b) so coloured values stay legible as text on the dark card; validated all-pairs in dark for CVD separation and contrast - repair outcomes collapse back to one segmented bar, since the detail list below already names each stage and its count - highlight OpenUI Lang's average in the completion table, matching how the per-model winners are marked - card padding is now tighter vertically than horizontally Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fell-through segment is too narrow to hold a label inside it, so its percentage sits just outside the bar. The 19% label goes white to match the 77% rather than flipping ink mid-bar. Cover art lands at the path the blog index derives from the slug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Author
External agent-readability testThe Vercel preview built successfully, but anonymous requests are redirected to Vercel SSO, so it is not a valid general-agent test URL while Deployment Protection is enabled. A production build was exposed through a temporary Cloudflare quick tunnel and tested in three ways:
An independent agent given only the public URL correctly reported:
The agent could operate and read the HTML page even though its browser client blocked direct opening of the JSON/CSV/Markdown endpoints. This validates keeping both exact tab datasets in the server-rendered HTML. The Cloudflare quick-tunnel URL is ephemeral and should not be treated as a review deployment. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
pnpm types:checkpnpm check:benchmark-agents(20/20 answer contracts)pnpm audit:benchmark-publicationpnpm buildEvidence status
The six-model format comparison links published raw results and outputs. The 30-model board is explicitly marked summary-only until row-level raw generations and pricing evidence are published.