feat: add Experiment report templates

[Cedric Vidal (cedricvidal)]

Summary

Adds reusable, project-scoped Experiment report templates as the next layer on the Experiment successor stack.

• Extends the existing report-template catalog with run | experiment subjects, immutable content versions, enablement, and manual / always / linked-run triggers.
• Freezes exact template-version identity per Experiment execution and schedules configurable reports only after the built-in pp-readout, with independent dependency state, exact-execution claims, stale-message protection, and append-only retries.
• Adds project-safe API and CLI parity for template lifecycle, Experiment selection/revisioning, report history, and selected-report retry.
• Adds Portal template creation/edit/version history, Experiment report configuration categories, multi-type/version report selection, and retry while preserving the PR B Option A layout and stable report shell.
• Adds Cosmos-compatible migration 035, OpenAPI updates, prompt-evaluation coverage, architecture/website documentation, and focused storage/orchestration/UI tests.

Demos

Before

N/A — this is a new Experiment report-template workflow.

After

Creation and immutable versioning, multi-template Experiment configuration, independent processing categories, exact report history, and safe retry:

experiment-report-templates-walkthrough.mp4

Template immutable version history

Multi-type report history and retry

Testing

• pnpm test — passed: 292 files, 3,509 tests.
• pnpm --filter shared build && pnpm --filter db-migrations build && pnpm --filter api build && pnpm --filter readout-handler build && pnpm --filter report-generator build && pnpm --filter cli build && pnpm --filter portal build — passed.
• pnpm --filter api generate:openapi — passed; committed snapshots/specification regenerated.
• pnpm eval:static-prompts:validate-data — passed.
• pnpm eval:static-prompts:smoke — passed offline smoke.
• pnpm --dir website build — passed.
• git diff --check — passed.
• Compose full-profile build under the exclusive lease was attempted once and failed because the local Podman daemon reported no space left on device while creating /var/tmp/libpod_builder…; the single targeted retry stopped before build because the reduced profile omitted the required entra-local-init dependency. No pruning or unrelated Docker resources were touched.
• Host Vite walkthrough used the normal Scope shell with deterministic browser-level API mocks. Two headed, attention-cued takes were recorded; take 2 was retained and verified by full decode and representative frame inspection. Final media: H.264, yuv420p, 1440×900, 25 fps, faststart (moov before mdat), 38.12 seconds.

Documentation and compatibility

Updated docs/architecture/experiments.md, docs/architecture/app-design.md, docs/architecture/post-processing.md, docs/architecture/prompt-evaluations.md, and the website Experiment guide.

Migration 035 backfills legacy report templates as enabled subject: run templates with immutable version 1 records and adds Cosmos-compatible lookup indexes. pp-readout remains the built-in Experiment readout; no historical artifact is reinterpreted as a configurable template report.

Checklist

• If Portal features changed, keep CLI capabilities in sync.
• If Portal components changed, update their Storybook stories.
• If database changes require a migration, include up() / down() and keep it CosmosDB-compatible.
• If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed. N/A — no dependency changes.
• Video showing the behavior before the suggested change. N/A — new workflow.
• Video showing the behavior after the suggested change.

[github-actions]

Test Results (Node.js 22)

test: Run #279

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3509	3508	1	0	0	0	0	1m56s

❌ Some tests failed!

Name	Failure Message
❌ ExperimentStore CAS contracts > plans chunked-to-chunked growth and chunked-to-inline shrink independently	Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #293

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3516	3515	1	0	0	0	0	1m57s

❌ Some tests failed!

Name	Failure Message
❌ ExperimentStore CAS contracts > plans chunked-to-chunked growth and chunked-to-inline shrink independently	Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #298

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3514	3513	1	0	0	0	0	1m58s

❌ Some tests failed!

Name	Failure Message
❌ ExperimentStore CAS contracts > plans chunked-to-chunked growth and chunked-to-inline shrink independently	Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".

Github Test Reporter