feat: add experiment results to the overview by joshblack · Pull Request #242 · primer/agent-eval

joshblack · 2026-09-10T14:58:06Z

This PR adds experiment results to the overview alongside the existing design-system benchmark. Experiment pages now show how treatments perform in the latest run, with per-scenario comparisons and direct links to walkthroughs, test results, and transcripts.

The run viewer already supported switching models and treatments, but only showed the first matching trial. This also adds trial selection so repeated trials can be inspected individually.

Changelog

New

Add an experiments section to / with each experiment's description, latest run, and treatment results.
Add latest-run treatment comparisons and per-scenario results to /experiments/:id, while preserving dated run history.
Add direct scenario output links and a trial selector in the shared run viewer.
Add website tests for aggregation, grouping, missing data, latest-run selection, run history, and loading errors, including assertions that older bundles are not read for the overview.
Add shared scenario-anchor generation and regression tests for rendered links, Unicode, spaces, slashes, and literal percent escapes.

Changed

Group comparisons by treatment, model, and reasoning effort. Test pass rates use summed passed/total test counts; resource usage is averaged per recorded trial.
Show trial and scenario counts to make differences in coverage visible. Empty latest runs remain empty rather than falling back to older results.
Read only the newest available experiment bundle for the homepage instead of loading every dated run.
Reset to the first trial when changing models or treatments, so returning to a previous combination does not restore a stale trial.
Keep scenario fragment targets consistent between native browser navigation and Next.js, without percent-escape collisions.
Document the routes, metric definitions, and output-navigation flow.
Merge the latest main, preserving its Judges tab and website tests alongside the trial selector.
Render Primer's focus-visible root markers on the server to avoid pre-hydration attribute mismatches without suppressing hydration warnings.

Removed

None.

Visual comparisons

Both versions use the same synthetic result bundles for the noop experiment. These screenshots illustrate the UI, not measured model performance. No benchmark bundles were loaded, so the benchmark's existing empty state is visible.

The original before/after pairs predate the merge of judge support from main. The integrated Judges view is shown below them.

Before / After: overview

The existing benchmark overview is preserved. Experiment summaries, latest-run links, and treatment results are now shown below it.

Before	After

Before / After: experiment details

The experiment page previously listed dated runs only. It now includes latest-run treatment comparisons, per-scenario results, and direct output links above the run history.

Before	After

Before / After: scenario output and trial selection

The before view always displays the first matching trial. The after view has Trial 2 selected, so its summary and transcript are different even though the underlying result bundle is identical.

Before	After

After review feedback: Judges tab and trial selection

The Judges tab from main is preserved and follows the selected trial. This uses a separate synthetic fixture to verify that switching treatments or models resets to the first matching trial, including when a model does not have the previous treatment.

Validation

Formatting, full build, JavaScript lint, package lint, tests, and type checks passed.
Agentic workflows compile without generated changes using the CI-pinned gh-aw v0.87.9.
Verified the overview-to-experiment-to-scenario flow in a browser, including treatment/trial selection, test and transcript updates, historical runs, empty states, and mobile layout.
Verified round-trip model/treatment changes reset trial selection, and that transcripts and judge results follow the selected trial.
Verified native browser fragment navigation and rendered link/target agreement for special-character scenario IDs.
Verified static export with and without experiment result bundles.
Verified clean hydration on overview and experiment run pages under normal and early polyfill initialization, interactive transcript tabs, and keyboard focus with and without JavaScript.

This is a website-only change and does not change the published @primer/agent-eval API, so no changeset is included.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f75a8076-a032-4c34-8135-97855d56fd80

Copilot

Copilot review overview

🟡 Changes recommended

The new trial-selection UI has state/control issues that can lead to inconsistent selection behavior, and the homepage overview loader currently does potentially expensive full run loading per experiment.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 3

New issues introduced by this change (3)

Severity	Finding
	`website/src/app/components/RunDetailsPage.tsx` — When switching models, `selectedTrial` is not reset to a trial that exists in the newly selected…
	`website/src/app/components/RunDetailsPage.tsx` — The Treatment select is controlled by `selectedResult.treatment`, but updates `selectedTreatment`…
	`website/src/experiment-page-data.ts` — `getExperimentsOverview()` calls `listForExperiment(experiment.id)`, which loads all dated runs…

What changed in this PR

This PR extends the website to surface experiment results alongside the existing design-system benchmark, adds latest-run treatment + per-scenario summaries to experiment pages, and enhances the shared run viewer with per-trial selection for repeated trials.

Changes:

Add experiment result aggregation/summarization and render it on / and /experiments/:id.
Add a trial selector (and scenario anchor IDs) to the shared run details viewer.
Add website-focused Vitest coverage for the new aggregation + page-data behaviors and wire it into the root Vitest projects.

File	Description
website/vitest.config.ts	Adds a dedicated Vitest project/config for website tests.
website/src/test/experiment.ts	Introduces helpers to build experiment run/result fixtures for tests.
website/src/experiment-results.ts	Implements aggregation for treatment/scenario summaries from a run bundle.
website/src/experiment-results.test.ts	Adds unit tests validating aggregation rules and edge cases.
website/src/experiment-page-data.ts	Adds data loaders for experiment pages and homepage overview summaries.
website/src/experiment-page-data.test.ts	Tests newest-run selection, empty/latest-run behavior, and error propagation.
website/src/app/page.tsx	Renders experiments overview on the homepage alongside benchmarks.
website/src/app/experiments/[id]/page.tsx	Switches to shared `getExperimentPageData` loader.
website/src/app/experiments/[id]/components/Page.tsx	Displays “Latest results” section on experiment pages.
website/src/app/components/RunDetailsPage.tsx	Adds trial selection and scenario anchors to the run viewer.
website/src/app/components/ExperimentResults.tsx	Adds overview + latest-results UI tables for experiment summaries.
website/README.md	Documents updated routes and metric definitions/output-navigation flow.
vitest.config.ts	Registers the website Vitest project in the root config.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Merge main, reset trial selection when models or treatments change, and load only the latest experiment bundle for the overview. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f75a8076-a032-4c34-8135-97855d56fd80

Copilot

Copilot review overview

🟡 Changes recommended

The new website Vitest tests use a non-existent matcher (toHaveBeenCalledExactlyOnceWith), which will fail test execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 · 1

New issues introduced by this change (2)

Severity	Finding
	`website/src/runs.test.ts` — `toHaveBeenCalledExactlyOnceWith` is not a Vitest matcher (and there’s no website vitest setup…
	`website/src/app/components/RunDetailsPage.tsx` — The scenario anchor ID is built from the raw `scenarioId`, but links to scenario output use…

Issues resolved since last review (3)

Severity	Finding
	`website/src/experiment-page-data.ts` — `getExperimentsOverview()` calls `listForExperiment(experiment.id)`, which loads all dated runs… View resolved comment
	`website/src/app/components/RunDetailsPage.tsx` — The Treatment select is controlled by `selectedResult.treatment`, but updates `selectedTreatment`… View resolved comment
	`website/src/app/components/RunDetailsPage.tsx` — When switching models, `selectedTrial` is not reset to a trial that exists in the newly selected… View resolved comment

Suppressed comments (2)

website/src/runs.test.ts:69

toHaveBeenCalledExactlyOnceWith is not a Vitest matcher (and there’s no website vitest setup extending expect), so this assertion will fail at runtime. Use toHaveBeenCalledTimes(1) plus toHaveBeenCalledWith(...) instead.

  expect(read).toHaveBeenCalledExactlyOnceWith(latest)

website/src/runs.test.ts:100

toHaveBeenCalledExactlyOnceWith is not a Vitest matcher (and there’s no website vitest setup extending expect), so this assertion will fail at runtime. Use toHaveBeenCalledTimes(1) plus toHaveBeenCalledWith(...) instead.

  expect(read).toHaveBeenCalledExactlyOnceWith(latest)

Share fragment and target generation, cover encoded scenario IDs, and simplify run-loader mock assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f75a8076-a032-4c34-8135-97855d56fd80

Copilot

Copilot review overview

🟢 Approval recommended

The changes are cohesive, well-tested (including edge cases like empty latest runs and invalid bundles), and I didn’t find any correctness or maintainability issues that block merging.

Review tier: Lite
Findings: None

Issues resolved since last review (2)

Severity	Finding
	`website/src/app/components/RunDetailsPage.tsx` — The scenario anchor ID is built from the raw `scenarioId`, but links to scenario output use… View resolved comment
	`website/src/runs.test.ts` — `toHaveBeenCalledExactlyOnceWith` is not a Vitest matcher (and there’s no website vitest setup… View resolved comment

Render Primer's deterministic focus-visible markers in the root layout so pre-hydration polyfill initialization does not cause an attribute mismatch. Add server-rendering coverage without suppressing hydration warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f75a8076-a032-4c34-8135-97855d56fd80

Copilot

Copilot review overview

🔵 Needs a closer look

It introduces broad UI + data-loading behavior changes (homepage aggregation, latest-run resolution, fragment/anchor semantics, and run viewer state) that warrant final human verification despite good test coverage.

Review tier: Lite
Findings: None

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

website/src/runs.test.ts:20

The test creates temporary fixtures under a repo-relative .agents/tmp directory. Other tests in this repo use os.tmpdir() with fs.mkdtemp(...), which avoids leaving behind project-local directories and is more robust when the checkout is read-only or shared.

Consider switching this test to use os.tmpdir() directly and removing the explicit .agents/tmp mkdir.

feat: add experiment results to the overview

8656951

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f75a8076-a032-4c34-8135-97855d56fd80

Copilot AI lite review requested due to automatic review settings September 10, 2026 14:58

Copilot started reviewing on behalf of joshblack September 10, 2026 14:58 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Comment thread website/src/app/components/RunDetailsPage.tsx

Comment thread website/src/app/components/RunDetailsPage.tsx

Comment thread website/src/experiment-page-data.ts

Copilot AI review requested due to automatic review settings September 10, 2026 16:04

Copilot started reviewing on behalf of joshblack September 10, 2026 16:05 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Comment thread website/src/runs.test.ts Outdated

Comment thread website/src/app/components/RunDetailsPage.tsx Outdated

fix: make scenario result anchors consistent

6259a76

Share fragment and target generation, cover encoded scenario IDs, and simplify run-loader mock assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f75a8076-a032-4c34-8135-97855d56fd80

Copilot AI review requested due to automatic review settings September 10, 2026 16:27

Copilot started reviewing on behalf of joshblack September 10, 2026 16:28 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 10, 2026 16:35

Copilot started reviewing on behalf of joshblack September 10, 2026 16:35 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat: add experiment results to the overview - #242

feat: add experiment results to the overview#242
joshblack wants to merge 4 commits into
mainfrom
add-experiment-results-overview

joshblack commented Sep 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joshblack commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog

New

Changed

Removed

Visual comparisons

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshblack commented Sep 10, 2026 •

edited

Loading