feat: add submission foundation

[Cedric Vidal (cedricvidal)]

Summary

Adds PR A of the submission/readout stack:

• introduces stable SubmissionDocument identity with an exact immutable request/run vector per embedded current attempt and immutable submission-runs history;
• adds resumable claim/archive/CAS rollover, polling repair, bulk-retry reconciliation holds, and attempt-scoped stale worker rejection;
• finalizes every native single/count/profile/bulk submit into a recoverable submission and exposes list/detail/history/current-retry APIs;
• defines the PR B pp-readout registration, queue, claim, supersession, and completion contracts over authoritative pp-observations state;
• adds Submission Detail navigation, parent-submission links, Storybook coverage, and scope submission CLI parity.

This draft intentionally stops at the submission foundation. The readout worker/report generation implementation belongs in stacked PR B.

Demos

Before

N/A — submissions had no stable detail surface or CLI commands on the base branch.

After

N/A — no recording was produced for this foundation draft.

Testing

• Passed: pnpm vitest run packages/shared/src/submissions/submission-state.test.ts packages/shared/src/submissions/submission-coordinator.test.ts packages/shared/src/scripts/register-handler.test.ts packages/db-migrations/src/034-create-submission-indexes.test.ts packages/db-migrations/src/required-migrations.test.ts apps/api/src/routes/submissions.test.ts apps/portal/src/lib/enrichment.test.ts apps/portal/src/components/SubmissionStatusCard.test.tsx apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts apps/scheduler/src/handler-dispatcher.test.ts — 73/73 tests.
• Passed: pnpm --filter shared build && pnpm --filter telemetry build && pnpm --filter api build && pnpm --filter scheduler build && pnpm --filter cli build && pnpm --filter db-migrations build && pnpm --filter post-processor build.
• Passed: pnpm --filter portal build-storybook.
• Passed: pnpm --filter api generate:openapi && pnpm --dir website build.
• Passed: pnpm vitest run apps/judge/src/agent-trajectory.test.ts apps/judge/src/observation-trajectory.integration.test.ts — 39/39 discovered tests.
• Passed: pnpm cli submission --help; confirmed list, get, runs, and retry.
• Partial baseline: pnpm vitest run apps/api/src/endpoints.test.ts — 168/171 passed. The three failing observation-criterion creation tests return 400 instead of 201 and are unchanged from the stacked base behavior.
• Blocked by stacked base: pnpm --filter portal build fails at apps/portal/src/pages/CriteriaList.tsx:168 because CriteriaBadge is not imported; that file is unchanged in this branch.
• Not run: backend ESLint scripts because the workspace command cannot find the eslint executable (spawn ENOENT).

Documentation and compatibility

• Updated submission lifecycle and PR B contracts in docs/architecture/app-design.md and docs/architecture/overview.md.
• Updated post-processing/criteria documentation for the authoritative pp-observations handler identity.
• Regenerated website/src/openapi/scope-openapi.json.
• Adds required migration 034-create-submission-indexes; it uses Cosmos-compatible index helpers and a no-op down migration.
• The observation handler registration changes from pp-taxonomy / taxonomy to pp-observations / observations. Deploy the updated registration manifest with the matching post-processor build.
• No dependency manifests or lockfiles changed.

Checklist

• If Portal features changed, keep CLI capabilities in sync.
• If Portal components changed, update their Storybook stories.
• If database changes require a migration, include up() / down() and keep it CosmosDB-compatible.
• If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed. (N/A — no dependency changes.)
• Video showing the behavior before the suggested change (N/A — new foundation).
• Video showing the behavior after the suggested change (not produced for this draft).

[github-actions]

Test Results (Node.js 22)

test: Run #272

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3386	3378	8	0	0	0	0	1m33s

❌ Some tests failed!

Name	Failure Message
❌ API Endpoints > POST /api/v1/criteria > creates an observation criterion with a taxonomy element	expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > defaults an observation's subject to 'run'	expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > creates an observation with an explicit subject:'iteration' override	expected 400 to be 201 // Object.is equality
❌ PostProcessor > re-processes stale versions and stamps success for the same run attempt	Cannot read properties of undefined (reading 'collection')
❌ PostProcessor > claims a run left in the scheduler's 'queued' state and drives it to done	Cannot read properties of undefined (reading 'collection')

See all failed tests here

Github Test Reporter

[Cedric Vidal (cedricvidal)]

Historical recovery reference: the canonical successor work is now #1512 (Experiment foundation) followed by #1513 (attempt-scoped Experiment readouts), registered in native stack #1518. This PR remains open and unchanged for recovery/history.