Skip to content

feat: add canonical Experiment foundation - #1512

Draft
Cedric Vidal (cedricvidal) wants to merge 25 commits into
cedricvidal-microsoft/observation-criteria-pp-taxonomyfrom
cedricvidal-experiment-foundation
Draft

feat: add canonical Experiment foundation#1512
Cedric Vidal (cedricvidal) wants to merge 25 commits into
cedricvidal-microsoft/observation-criteria-pp-taxonomyfrom
cedricvidal-experiment-foundation

Conversation

@cedricvidal

@cedricvidal Cedric Vidal (cedricvidal) commented Oct 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Adds the canonical Experiment foundation as the sole experiment aggregate across shared persistence, API coordination, scheduler integration, CLI, Portal, Storybook, OpenAPI, migrations, deployment, tests, and operations.

  • Implements bounded Cosmos Mongo storage with deterministic inline/chunked fixed payloads, immutable archived executions, CAS transitions, rollover/retry/repair, and migration 034.
  • Adds project-scoped Experiment create/list/get/execution-history/retry APIs and exact execution dispatch identities.
  • Adds matching CLI and Portal creation and operational surfaces while preserving permanent ordinary request submission.
  • Infers empty-versus-composed creation in the Portal from scenario selection while keeping the API payload explicitly discriminated.
  • Aligns the Experiments list with the shared Runs-list full-height, edge-to-edge ListLayout, filter rail, DataTable, column customization, pagination, and action layout while preserving Experiment-specific status and navigation; Experiments now immediately precedes Runs in the primary sidebar.
  • Aligns Experiment Detail with Run Detail: current-by-default attempt selection, direct historical links, one selected attempt at a time, and an inset shared sortable/customizable/paginated exact-run table that remains visually distinct from its enclosing section.
  • Removes Submission resource routes and surfaces; legacy request-batch submissionId correlation remains compatibility-only.
  • Documents the architecture, creation workflow, and migration from deprecated composition inputs.

Demos

Before

N/A — this introduces the canonical Experiment workflow.

After

pr-a-experiment-progressive-actions.mp4

Exhaustive action-driven walkthrough of Experiment composition: empty state → task reveal → prompt helper, codebase, resources, and AGENTS.md → advanced options, iteration policy, and gate configuration → Quick Start profile locking/versioning → variation expansion and profile swap → occurrence multiplication → manual mode → agent/model-dependent reasoning and tool capabilities → exact run-count CTA → task reset and collapse.

Testing

  • Passed on the current head: pnpm exec vitest run apps/portal/src/components/Layout.test.tsx apps/portal/src/pages/ExperimentsList.test.tsx apps/portal/src/pages/ExperimentsList.play.test.tsx (17/17).
  • Passed: pnpm exec vitest run apps/portal/src/pages/ExperimentDetail.test.tsx apps/portal/src/pages/ExperimentDetail.play.test.tsx (4/4).
  • Passed: pnpm exec vitest run apps/api/src/routes/experiments.test.ts apps/api/src/experiment-coordinator.test.ts apps/cli/src/commands/experiment.test.ts (14/14).
  • Passed: pnpm vitest run apps/portal/src/pages/SubmitRun.test.tsx apps/portal/src/components/ui/stories.play.test.tsx (34/34).
  • Passed: pnpm --filter api build.
  • Passed: pnpm --filter cli build:tsc.
  • Passed on the current head: pnpm --filter portal lint.
  • Passed on the current head: pnpm --filter portal build.
  • Passed on the current head: pnpm --dir apps/portal exec storybook build -o /tmp/pr-a-launcher-revert-storybook.
  • Passed on the current head: cd website && pnpm test && pnpm build (7/7 tests and production build).
  • Passed earlier on this exact branch lineage: Experiment backend-focused tests (265/265), CLI-focused tests (19/19), Portal/Storybook-focused tests (53/53), website tests (7/7), website production build, Portal production build, and OpenAPI generation/validation.
  • Full repository suite: 3,412/3,416. The four remaining failures reproduce independently of PR A: unavailable local pwsh, an unchanged prompt fixture omitting suggestedSubject: "run", and two post-processor tests constructing without the database read by unchanged base code.
  • Clean exact-head Compose smoke passed for migration 034, MongoDB, Redis, Azurite, API, scheduler, and Portal: health endpoints succeeded, Portal returned 200, scheduler loops started, and logs were clean.
  • Clean-stack canonical CLI smoke passed for empty create, list, get, execution history, and retry.
  • Copilot worker no-cache image completion is environment-limited by the shared Podman VM disk while committing its large toolchain layer. No valid GitHub/Copilot authorization was available, so no agent-run success is claimed.

Documentation and compatibility

  • Added docs/architecture/experiments.md and updated architecture/navigation references.
  • Added website Experiment creation and request-composition migration documentation with API and CLI mappings.
  • POST /requests, run submit, and /runs/new remain first-class. Only overlapping legacy composition inputs are conditionally deprecated; there is no Sunset date.
  • Experiment storage requires migration 034 and persists storageProfileVersion = "cosmos-mongo-2mb-v1".

Checklist

  • If Portal features changed, keep CLI capabilities in sync.
  • If Portal components changed, update their Storybook stories.
  • If database changes require a migration, include up() / down() and keep it CosmosDB-compatible.
  • If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed. N/A — no dependencies changed.
  • Video showing the behavior before the suggested change. N/A — new workflow.
  • Video showing the behavior after the suggested change.

@cedricvidal
feat(submissions): add attempt state foundation
a1612d4 
Introduce stable submission identities, deterministic request/run vectors, resumable CAS rollover with immutable history, named-handler dependency evaluation, Cosmos-safe indexes, and attempt-scoped worker claim/completion helpers. Align the observation handler identity with the pp-observations dependency contract.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 99f84f04-d3b0-40f1-a254-4dfbd21e4d87
@cedricvidal
feat(submissions): orchestrate native submission attempts
36fb224 
Finalize every native submit as an exact submission vector, recover interrupted final writes, reconcile and archive vector drift, dispatch only current attempts, and expose project-scoped submission detail, history, and retry APIs with polling repair.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 99f84f04-d3b0-40f1-a254-4dfbd21e4d87
@cedricvidal
feat(submissions): add Portal and CLI workflows
0220a11 
Navigate successful submits to a Submission Detail shell, link grouped runs and run detail back to their parent submission, surface readiness and blocking pairs, and add CLI list/get/history/retry parity with Storybook coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 99f84f04-d3b0-40f1-a254-4dfbd21e4d87
@cedricvidal
style(submissions): normalize file endings
a88da97 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 99f84f04-d3b0-40f1-a254-4dfbd21e4d87
@cedricvidal
feat(shared): add canonical experiment storage
57c7634 
Introduce the Experiment aggregate, bounded request payload schemas, deterministic Cosmos-safe chunk routing/layout, persistence CAS and repair primitives, focused tests, and migration indexes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8fe76b9e-1205-4910-b51b-e8fa51538125
@cedricvidal
refactor(experiments): remove submission model
e251641 
Replace imported Submission persistence surfaces with Experiment correlation and handler types.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
feat(experiments): add canonical API runtime
ed1ca2c 
Replace legacy Submission API and scheduler surfaces with project-scoped Experiment creation, execution history, retry, exact queue claims, recovery, and OpenAPI contracts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b05e74ff-44da-4e78-9f05-0e256009e7ee
@cedricvidal
fix(requests): preserve legacy batch correlation
aedbc18 
Restore submissionId as a request-batch correlation contract while keeping Experiment as the only aggregate. Require project scope for nested Experiment execution reads and publish the shared composition deprecation identifier.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b05e74ff-44da-4e78-9f05-0e256009e7ee
@cedricvidal
feat(shared): add experiment payload CAS updates
48f244b 
Add representation-preserving request projection updates with root publication claims, versioned bucket CAS, resumable partial-failure outcomes, and inline/chunked concurrency coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8fe76b9e-1205-4910-b51b-e8fa51538125
@cedricvidal
feat: add Experiment clients and documentation
a55c388 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 4e02b3c3-5ec3-44f8-b88d-642673088e41
@cedricvidal
refactor(experiments): move workflow policy to API
a0bf54e 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
docs: document Experiment architecture
23d6f58 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 4e02b3c3-5ec3-44f8-b88d-642673088e41
@cedricvidal
docs(experiments): align canonical runtime contract
44ff1d6 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
fix(portal): restore criteria badge import
bca9d7a 
Unblock the required clean full-stack Docker build by restoring the import for the already-rendered CriteriaBadge component.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
fix(experiments): align canonical client contracts
2c453c3 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@github-actions github-actions Bot added type: documentation Documentation additions, corrections, and improvements. area: cli Scope command-line interface and terminal workflows. type: dependencies Dependency additions, removals, and version updates. area: infrastructure Cloud resources, hosting, networking, and shared infrastructure. language: javascript Work involving JavaScript code, tooling, or dependencies. area: portal Scope web portal, pages, and user-facing components. area: post-processing ATIF generation and the run-output post-processing pipeline. topic: testing Test coverage, test infrastructure, and validation quality. topic: ui Portal, visual presentation, layout, and interface components. area: scheduler labels Oct 16, 2026
@cedricvidal
fix(portal): allow explicit empty experiments
4ba14c7 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
fix(portal): infer empty experiment creation
3e5dc95 
Remove the explicit-empty checkbox, progressively disclose run composition from scenario selection, and keep Experiment API creation discriminated. Add component and Storybook interaction coverage and update the creation guides.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
fix(experiments): remove unsupported tags
26fa3bf 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
fix(portal): distinguish experiment icon
a007bcd 
Use the Atom icon for Experiment navigation and empty states while preserving the Criteria flask identity. Add focused layout and Storybook assertions for the distinct icons.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
feat(portal): align experiment detail with run history
f7cba92 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
refactor(portal): unify experiment list layout
ae417a3 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
fix(portal): inset experiment runs table
2678960 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@cedricvidal
Cedric Vidal (cedricvidal) added this pull request to stack #1518 October 17, 2026 00:58
This was referenced Oct 17, 2026
Draft
Draft
@github-actions

github-actions Bot commented Oct 17, 2026

Copy link
Copy Markdown

Test Results (Node.js 22)

test: Run #278

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
3429 3422 7 0 0 0 0 1m17s

❌ Some tests failed!

Name Failure Message
❌ API Endpoints > POST /api/v1/criteria > creates an observation criterion with a taxonomy element expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > defaults an observation's subject to 'run' expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > creates an observation with an explicit subject:'iteration' override expected 400 to be 201 // Object.is equality
❌ PostProcessor > re-processes stale versions and stamps success for the same run attempt Cannot read properties of undefined (reading 'collection')
❌ PostProcessor > claims a run left in the scheduler's 'queued' state and drives it to done Cannot read properties of undefined (reading 'collection')

See all failed tests here

Github Test Reporter

@cedricvidal
fix(portal): align experiments list shell
457cacf 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@github-actions

github-actions Bot commented Oct 17, 2026

Copy link
Copy Markdown

Test Results (Node.js 22)

test: Run #288

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
3434 3427 7 0 0 0 0 1m46s

❌ Some tests failed!

Name Failure Message
❌ API Endpoints > POST /api/v1/criteria > creates an observation criterion with a taxonomy element expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > defaults an observation's subject to 'run' expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > creates an observation with an explicit subject:'iteration' override expected 400 to be 201 // Object.is equality
❌ PostProcessor > re-processes stale versions and stamps success for the same run attempt Cannot read properties of undefined (reading 'collection')
❌ PostProcessor > claims a run left in the scheduler's 'queued' state and drives it to done Cannot read properties of undefined (reading 'collection')

See all failed tests here

Github Test Reporter

@cedricvidal
feat(portal): add experiment creation launcher
994d4c0 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@github-actions

github-actions Bot commented Oct 17, 2026

Copy link
Copy Markdown

Test Results (Node.js 22)

test: Run #291

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
3436 3429 7 0 0 0 0 1m45s

❌ Some tests failed!

Name Failure Message
❌ API Endpoints > POST /api/v1/criteria > creates an observation criterion with a taxonomy element expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > defaults an observation's subject to 'run' expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > creates an observation with an explicit subject:'iteration' override expected 400 to be 201 // Object.is equality
❌ PostProcessor > re-processes stale versions and stamps success for the same run attempt Cannot read properties of undefined (reading 'collection')
❌ PostProcessor > claims a run left in the scheduler's 'queued' state and drives it to done Cannot read properties of undefined (reading 'collection')

See all failed tests here

Github Test Reporter

@cedricvidal
Revert "feat(portal): add experiment creation launcher"
7af2be5 
This reverts commit 994d4c0.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 92d03055-789e-48b7-9a44-6de5b357f486
@github-actions

github-actions Bot commented Oct 17, 2026

Copy link
Copy Markdown

Test Results (Node.js 22)

test: Run #295

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
3434 3427 7 0 0 0 0 1m43s

❌ Some tests failed!

Name Failure Message
❌ API Endpoints > POST /api/v1/criteria > creates an observation criterion with a taxonomy element expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > defaults an observation's subject to 'run' expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > creates an observation with an explicit subject:'iteration' override expected 400 to be 201 // Object.is equality
❌ PostProcessor > re-processes stale versions and stamps success for the same run attempt Cannot read properties of undefined (reading 'collection')
❌ PostProcessor > claims a run left in the scheduler's 'queued' state and drives it to done Cannot read properties of undefined (reading 'collection')

See all failed tests here

Github Test Reporter

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@manekinekko Wassim Chegham (manekinekko) Awaiting requested review from manekinekko manekinekko will be requested when the pull request is marked ready for review manekinekko is a code owner

Assignees

No one assigned

Labels

area: cli Scope command-line interface and terminal workflows. area: infrastructure Cloud resources, hosting, networking, and shared infrastructure. area: portal Scope web portal, pages, and user-facing components. area: post-processing ATIF generation and the run-output post-processing pipeline. area: scheduler language: javascript Work involving JavaScript code, tooling, or dependencies. topic: testing Test coverage, test infrastructure, and validation quality. topic: ui Portal, visual presentation, layout, and interface components. type: dependencies Dependency additions, removals, and version updates. type: documentation Documentation additions, corrections, and improvements.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cedricvidal