Skip to content

feat: add a file explorer to trial results - #244

Open
joshblack wants to merge 4 commits into
mainfrom
add-trial-results-file-explorer
Open

feat: add a file explorer to trial results#244
joshblack wants to merge 4 commits into
mainfrom
add-trial-results-file-explorer

Conversation

@joshblack

@joshblack joshblack commented Sep 10, 2026

Copy link
Copy Markdown
Member

This PR adds a Code tab to benchmark and experiment trial results so we can inspect the generated workspace without downloading the result bundle.

The explorer shows the final saved workspace, including starter files, rather than a diff of the agent's changes. File previews are highlighted on the server and exported as individual JSON assets, so the explorer also works on GitHub Pages.

Changelog

New

  • Add an expandable, keyboard-accessible file tree with read-only UTF-8 previews and a responsive layout.
  • Add Shiki syntax highlighting with GitHub light and dark themes that follow the website's color mode. Unknown file types remain plain text.
  • Add explicit states for missing workspaces, empty files, binary files, preview limits, loading, and failed requests with retry.
  • Add coverage for workspace loading, path containment, symbolic links, size limits, language detection, source escaping, exact whitespace preservation, preview routes, and lightweight run payloads.

Changed

  • Include workspace previews in benchmark and experiment result details while preserving the existing Walkthrough, Tests, Judges, and Transcript tabs.
  • Fetch a preview only when its file is selected. Run pages contain file metadata and preview URLs instead of every file's contents and highlighted React nodes.
  • Keep Shiki, its grammars, and its runtime on the server. Highlighting runs at build time for static exports and on request during local development.
  • Render returned tokens as escaped React text rather than injected HTML, and cancel stale requests when selecting another file.
  • Reset the explorer when switching models or treatments.
  • Accommodate Primer's focus-visible polyfill attributes on the root HTML element without suppressing hydration warnings inside the page.
  • Apply directory exclusions only to directories, preserving regular files with names such as dist.
  • Include files exactly 50 directory levels deep and truncate only deeper directories.

Removed

None.

Review notes

This is a website-only change and does not change the published @primer/agent-eval API or result format. No changeset is needed.

Previews are limited to 256 KiB per file and 2 MiB per workspace. The tree is limited to 2,000 entries and 50 directory levels. Dependency/build directories and Git metadata are omitted; symbolic links are not followed. Review workspace contents before publishing, since previewable files are included in the website.

The initial Server Component implementation eagerly serialized previews for every trial, producing approximately 109 MB of HTML for the saved design-system run. The same run now exports approximately 4.3 MB of HTML with no highlighting tokens. Previews use statically exported GET responses instead of Server Component slots, avoiding that work during the initial run render while keeping highlighting out of the browser.

Verification

  • Run the CI workflow compiler with the pinned v0.87.9 release.
  • Run formatting, lint, package lint, builds, type checks, and tests from ci.yml.
  • Exercise the explorer on benchmark and experiment pages, including keyboard navigation, trial switching, binary/empty files, and mobile layout.
  • Verify initial run rendering and Code navigation request no previews, and file selection requests only the selected file.
  • Verify syntax colors, unknown file types, source preservation, failed-request retry, and stale-request cancellation.
  • Confirm client JavaScript contains no Shiki runtime or grammars, and the run page has no pre-rendered highlighting tokens.
  • Verify the root hydration warning is absent on full navigation and reload.
  • Cover directory-only exclusions and the exact depth boundary, and run website tests with a 256-file-descriptor limit.
Before / After

These screenshots compare the plain-text explorer at 310b122 with Shiki highlighting. Both versions use the same representative local trial fixtures (not real evaluation runs). On-demand loading preserves the highlighted preview's appearance.

Experiment results

Before After
Experiment file explorer with plain-text source Experiment file explorer with Shiki syntax highlighting

Benchmark results

Before After
Benchmark file explorer with plain-text source Benchmark file explorer with Shiki syntax highlighting

Dark mode

File explorer with dark-mode syntax highlighting

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a49f4691-6bee-4479-98d8-adf6e8f60d38
Copilot AI lite review requested due to automatic review settings September 10, 2026 15:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The workspace loader currently excludes files whose names match excluded directories and has an apparent off-by-one depth limit, plus a test that may be flaky due to high parallel file creation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 3 Medium severity

New issues introduced by this change (3)
Severity Finding
Medium severity website/​src/​workspace-files.test.ts — This test creates 2,001 files using Promise.all, which can open a large number of file…
Medium severity website/​src/​workspace-files.tsMAX_DEPTH is described as “50 directory levels”, but the current depth &gt;= MAX_DEPTH guard stops…
Medium severity website/​src/​workspace-files.ts — The exclusion filter drops any entry whose name matches an excluded directory, even when that entry…
What changed in this PR

Adds a new website-only “Code” tab to benchmark/experiment trial results that renders a read-only, keyboard-accessible file tree and UTF-8 previews for the saved workspace (artifacts.workspaceDirectory) at build time (static-export friendly).

Changes:

  • Introduces workspace loading + safety/limits (path containment, symlink handling, byte/entry/depth caps) and wires it into run details.
  • Adds a new FileExplorer UI component and integrates it as a new tab in the run details page.
  • Adds Vitest coverage for workspace loading edge cases and documents the feature/limits in the website README.
File Description
website/​src/​workspace-files.ts New loader that reads and bounds workspace file trees + previews from result artifacts.
website/​src/​workspace-files.test.ts Tests for workspace loading, limits, symlink/path safety, and integration into experiment run details.
website/​src/​runs.ts Plumbs workspaceDirectory through normalized experiment output results.
website/​src/​run-details.ts Integrates getWorkspaceFiles into benchmark/experiment run details; factors artifact resolution into shared helper.
website/​src/​artifacts.ts New shared artifact path resolution/containment helpers.
website/​src/​app/​components/​RunDetailsPage.tsx Adds the new “Code” tab and renders the explorer per trial (reset via key).
website/​src/​app/​components/​FileExplorer.tsx New client component rendering the file tree and read-only previews.
website/​src/​app/​components/​FileExplorer.module.css Responsive two-pane layout styling for tree + preview.
website/​README.md Documents the new Code tab behavior, exclusions, and preview/tree limits.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread website/src/workspace-files.test.ts Outdated
Comment thread website/src/workspace-files.ts Outdated
Comment thread website/src/workspace-files.ts
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a49f4691-6bee-4479-98d8-adf6e8f60d38
Copilot AI review requested due to automatic review settings September 10, 2026 15:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Workspace candidate resolution returns early on some invalid candidates, preventing fallback to later valid candidates and causing incorrect “unavailable” outcomes in certain relocation/legacy scenarios.

Review tier: Lite
Findings: None

Issues resolved since last review (3)
Severity Finding
Medium severity website/​src/​workspace-files.ts — The exclusion filter drops any entry whose name matches an excluded directory, even when that entry… View resolved comment
Medium severity website/​src/​workspace-files.tsMAX_DEPTH is described as “50 directory levels”, but the current depth &gt;= MAX_DEPTH guard stops… View resolved comment
Medium severity website/​src/​workspace-files.test.ts — This test creates 2,001 files using Promise.all, which can open a large number of file… View resolved comment
Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

website/src/workspace-files.ts:145

  • Inside the for (const candidate ...) loop, returning immediately on an out-of-root resolution (or a non-directory) prevents trying later candidates from getArtifactCandidates() (e.g., legacy vs relocated artifacts). This can cause false negatives when the first candidate is invalid but a subsequent candidate is valid; it also differs from getArtifactDataUrl, which continues to other candidates on failure. Consider tracking these failure modes and only returning an unavailable reason after all candidates have been tried.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a49f4691-6bee-4479-98d8-adf6e8f60d38
Copilot AI review requested due to automatic review settings September 10, 2026 16:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changes are cohesive, well-tested, and the workspace previewing path includes appropriate safety/limit handling for static and server rendering.

Review tier: Lite
Findings: None

Files not reviewed (1)
  • pnpm-lock.yaml: Generated file

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a49f4691-6bee-4479-98d8-adf6e8f60d38
Copilot AI review requested due to automatic review settings September 10, 2026 16:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

FileExplorer uses workspace file paths directly in DOM id attributes, which can be invalid (e.g., whitespace) and break TreeView accessibility/behavior.

Review tier: Lite
Findings: None

Files not reviewed (1)
  • pnpm-lock.yaml: Generated file
Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

website/src/app/components/FileExplorer.tsx:22

  • TreeView.Item uses an id derived from entry.path. Workspace paths can contain spaces (and other whitespace), which makes the resulting HTML id invalid and can break TreeView’s ARIA relationships / keyboard navigation for those entries. Encode/sanitize the path portion before using it as a DOM id.

This issue also appears on line 35 of the same file.

website/src/app/components/FileExplorer.tsx:35

  • TreeView.Item uses an id derived from entry.path. Workspace paths can contain spaces (and other whitespace), which makes the resulting HTML id invalid and can break TreeView’s ARIA relationships / keyboard navigation for those entries. Encode/sanitize the path portion before using it as a DOM id.
        id={`${id}-${entry.path}`}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants