feat: GEPA prompt optimizer with Scope adapter (AGENTS.md optimization) by cedricvidal · Pull Request #1031 · microsoft/scope

Cedric Vidal (cedricvidal) · 2026-07-29T09:28:16Z

Summary

Adds the GEPA prompt optimizer as the final layer on the canonical Experiment
lifecycle stack.

Adds apps/gepa-optimizer/, a Python 3.12 + uv app that evaluates candidate
AGENTS.md prompts through real, project-scoped Scope runs.
Creates or targets one canonical Experiment per optimization and publishes
each evaluated candidate batch through revision-guarded membership updates.
Consumes exact Experiment revision, execution, and report identities while
preserving bounded concurrency, derived criteria scoring, reflective
evidence, systemic/per-example failures, idempotent-read retries,
non-retried submissions, and post-hoc candidate lineage.
Adds GEPA-specific CI, root scripts, configuration reference, architecture
documentation, and unit tests without duplicating superseded Experiment API,
schema, CLI, Portal, or migration surfaces.
Keeps the standalone CLI self-contained by importing observation taxonomy
metadata through a focused shared subpath instead of the server-heavy shared
root; this repairs an inherited stacked-base ioredis bundle failure.

Demos

Before

N/A — optimizer/automation layer with no new user-visible Portal or CLI surface.

After

N/A — optimizer/automation layer with no new user-visible Portal or CLI surface.

Testing

pnpm test:gepa — passed, 44 tests.
pnpm exec vitest run scripts/ci-workflow.test.ts — passed, 22 tests.
pnpm build:gepa — passed; locked uv environment resolved and audited.
cd apps/gepa-optimizer && uv run python -m compileall -q src tests && uv sync --locked --check — passed.
pnpm headers:check — passed, 1055 in-scope files.
pnpm notice:check — passed after restoring the frozen pnpm dependency store; NOTICE files are current.
pnpm build:cli && node --no-global-search-paths apps/cli/dist/scope.mjs --version — passed; isolated bundle prints 0.0.0-dev.
pnpm vitest run --config vitest.integration.config.ts apps/cli/src/bundle.integration.test.ts apps/cli/src/utils/update-check.integration.test.ts — passed, 24 tests.
pnpm vitest run packages/shared/src/experiments/experiment-store.test.ts -t "plans chunked-to-chunked growth and chunked-to-inline shrink independently" — passed; 51ms.
pnpm vitest run apps/portal/src/pages/ExperimentDetail.test.tsx -t "starts an asynchronous deletion from the confirmation dialog" — passed; 1.34s.
pnpm vitest run --exclude scripts/build-windows-worker.test.ts — passed under full-suite load, 293 files / 3,523 tests; the complete ExperimentStore and ExperimentDetail files passed 14/14 and 16/16.
pnpm vitest run --coverage --exclude scripts/build-windows-worker.test.ts — passed under CI-like coverage load, 293 files / 3,523 tests; ExperimentStore completed in 2.34s and ExperimentDetail in 11.59s with all individual tests inside their bounded timeout.
pnpm test — both stabilized files passed under full-suite load; the run otherwise had one unrelated local spawnSync pwsh ETIMEDOUT in scripts/build-windows-worker.test.ts.
pnpm --filter portal lint && pnpm --filter shared build — passed. pnpm --filter shared lint could not start because this checkout's local install does not expose the eslint executable.
Live smoke — blocked locally because http://localhost:3100 was unavailable and apps/gepa-optimizer/.env was absent.

Documentation and compatibility

Updated apps/gepa-optimizer/README.md, ENV_VARIABLES.md,
docs/architecture/app-design.md, docs/architecture/experiments.md, and
root AGENTS.md.
Targets the canonical Experiment create, membership, revision, execution, and
report contracts from the base branch.
Exposes the existing observation taxonomy module as a focused package
subpath; no breaking API/CLI changes and no migration or upgrade step.

Checklist

If Portal features changed, keep CLI capabilities in sync. N/A — no Portal features changed.
If Portal components changed, update their Storybook stories. N/A — no Portal components changed.
If database changes require a migration, include up() / down() and keep it CosmosDB-compatible. N/A — no database changes.
If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed. uv.lock is committed and pnpm notice:check passes.
Video showing the behavior before the suggested change. N/A — no user-visible surface.
Video showing the behavior after the suggested change. N/A — no user-visible surface.

Add AGENTS.md support and GEPA experiment grouping to the shared layer: - TaskPromptDocument gains optional `type` ('task' | 'agents.md') and `contentBlobUrl`; `text` becomes optional. computeTaskPromptId is type-aware and backward-compatible (task/undefined keep identical _ids). - TaskPromptStore: size-based storage (inline <= PROMPT_INLINE_MAX_BYTES, else blob at prompts/{id}.txt), findOrCreate({type}), getAll({type}), resolvePromptText(doc), 256 KB hard cap. BlobStorage.uploadText added. - RequestDocument + schemas gain experimentId, agentsMdPromptId, agentsMdParentIds (multi-parent lineage). GroupByKey adds 'experiment'. - Tests: 35 passing covering type namespacing, backward compat, blob offload on create-miss, resolvePromptText, max-size reject, getAll filter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ompts Wire the shared typed-prompt and experiment capabilities into the API: - POST /api/v1/requests accepts experimentId, agentsMd, agentsMdParentIds; agentsMd is resolved via findOrCreate(type: 'agents.md') and stored as agentsMdPromptId across all construction paths (variation, count>1, single); retry/resubmit/import preserve lineage. - GET /api/v1/requests: ?experimentId= filter + groupBy=experiment. - Task prompts: ?type= list filter, create with type, new GET /:id/content (resolvePromptText), extract-features scoped by prompt type. TaskPromptStore constructed with blobStorage. - grouping.ts buildGroupingPipeline supports 'experiment'. - Tests: typed task prompts, /:id/content, experiment/AGENTS.md request creation + lineage, experiment filter/group. OpenAPI snapshot updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Features are now scoped by prompt type so a feature only applies to prompts of its own type: - PromptFeatureStore: getAll({type}), create({type}), seed carries type. - Schemas: CreatePromptFeatureInput + PromptFeatureResponse add optional type. - API: GET /prompt-features?type= filter (task includes legacy untyped), create persists type, seed $setOnInsert carries type, extract-from-text scopes candidate features by ?type=. task-prompts extract-features already scopes by the prompt's own type. - FS provider loads optional `type` from feature YAML. - Tests: 6 store tests + API filter/create tests. OpenAPI snapshot updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add AGENTS.md delivery to the shared queue processor so all workers honor a request's optional agentsMdPromptId: - New PromptClient.getText(id) REST client resolves a prompt's plain-text body via GET /api/v1/task-prompts/:id/content (server downloads blob if the doc is blob-backed), with clear errors on 404 / non-OK / missing text. - writeAgentsMd() resolves the AGENTS.md text once before the run and writes <workspace>/AGENTS.md (constant for the whole run). No-ops when the request has no agentsMdPromptId; throws (fails the run loudly via the base processor's catch) when set but apiBaseUrl is missing or resolution fails. Tests: PromptClient success/404/non-OK/missing-text; writeAgentsMd no-op, missing-apiBaseUrl throw, and writes resolved body into a temp workspace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

GEPA experiments group a seed request and all follow-up requests under a shared experimentId. Both experiment grouping (groupBy=experiment) and AGENTS.md lineage reconstruction filter requests by this field, so add a sparse single-field index to keep those queries efficient. - 018-add-experiment-id-index.ts: try/catch-wrapped createIndex, log-only non-destructive down(). - Registered in required-migrations.ts. - Unit tests cover index creation, idempotent already-exists, and no-op down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…pt features - SubmitRun: optional AGENTS.md textarea + experimentId input - RunDetail: render AGENTS.md as Markdown with lineage/experiment badges - RunsList: add "Experiment" group-by option (client-side grouping by experimentId) - PromptFeatureWizard/List: typed features (task | agents.md) with filter + column - api.ts/types.ts: new request fields, TaskPromptType, getTaskPromptContent Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- run submit: --agents-md <text|@file>, --experiment-id - run list: --experiment-id filter, --group-by (incl. experiment) - task-prompt list/create: --type (task | agents.md); guard blob-backed prompts - prompt-feature list/create: --type filter + create, Type column - Tests for all new options (run/task-prompt/prompt-feature) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add apps/gepa-optimizer, a Python+uv app that drives the GEPA reflective prompt-optimization algorithm to optimize an AGENTS.md file, scoring each candidate with a real Scope benchmark run (Copilot CLI worker + Claude Haiku 4.5). - ScopeAdapter(GEPAAdapter): submit one Scope run per dataset example with agentsMd + experimentId, poll to completion, score from criteriaResults, build per-turn trajectories and reflective dataset (judge + criteria feedback for every turn). Bounded concurrency (default 10, the Linux copilot KEDA ceiling) via asyncio + httpx. - Scoring is derived, never persisted: denominator = requested criteria (skipped/missing count as failed); mean|final|weighted across turns; non-succeeded/empty runs score 0.0, never NaN. - Failure model: per-example errors -> 0.0; systemic majority -> abort, so infrastructure failure never poisons optimization. Idempotent GETs retry with backoff; POSTs are not retried. - Best-effort lineage reconstructed post-hoc from GEPA result.parents. - Preflight validates criterion ids before burning agent runs. - 36 pytest unit tests (scoring, reflective, lineage, client via httpx.MockTransport, adapter via stubbed client). - CI: new gepa job (uv sync --locked + pytest), change filter, ci-summary wiring; root scripts test:gepa/build:gepa; README app table + polyglot exception documented (mirrors apps/gateway). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… optimizer env - app-design.md: typed prompts (task|agents.md), size-based inline/blob storage, experiment grouping, AGENTS.md delivery, scoring-not-persisted - ENV_VARIABLES.md: PROMPT_INLINE_MAX_BYTES + GEPA optimizer (SCOPE_*/GEPA_*) vars - AGENTS.md: gepa-optimizer in repo tree + Key Components section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…riment flags Captures --type (prompt-feature/task-prompt create), --agents-md, --experiment-id, and --group-by added in the CLI step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

TaskPromptDocument.text is now optional (blob-backed docs carry no inline text). Guard all consumers (pickers, list, preview panel) with fallbacks; the preview panel shows a 'stored in blob' note when the body is absent. Fixes tsc build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…on 018 Migration 018-add-experiment-id-index was added to REQUIRED_MIGRATIONS in the db-migration step; update the applied/pending lists and totalApplied counts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…y AGENTS.md Replace the two generic Express demo tasks with a single, self-contained optimization target: a Medium-style Django blogging platform backed by a nonrelational cloud database. The seed AGENTS.md is now EMPTY on purpose so GEPA discovers the guidance from scratch. Each run requests the full ancestor closure of the target leaf (uses_azure_documentdb): uses_database -> uses_document_oriented_db, plus uses_azure, both feeding uses_azure_documentdb. Requesting the closure (not just the leaf) gives GEPA graded partial-credit signal instead of a sparse 0/1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The DataTable's getGroupKey switch was missing the 'experiment' case, so selecting Group by > Experiment collapsed every run into a single empty-keyed group instead of grouping by experimentId. Add the missing case and a new Experiment column (visible by default) that shows the short experiment id with the full id on hover. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

compute_run_score gated on outcome == "succeeded", returning 0.0 for any run that did not pass *every* criterion. But Scope only marks a run "succeeded" when all criteria pass; a run that completes and is judged with partial criteria gets outcome "finished". Gating on "succeeded" collapsed GEPA's optimization landscape to a flat 0.0 until a candidate happened to pass all criteria, destroying the partial-credit gradient the reflective loop relies on (a seed passing 3/4 criteria scored 0.0 instead of 0.75). Accept both "succeeded" and "finished" as scoreable outcomes (run completed and was judged); keep "failed" (errored/abandoned) at 0.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Cedric Vidal (cedricvidal) · 2026-07-29T09:28:18Z

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1839

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
1921	1921	0	0	0	0	0	1m25s

🎉 All tests passed!

Github Test Reporter

github-actions · 2026-10-17T00:31:04Z

Test Results (Node.js 22)

test: Run #274

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3531	3529	2	0	0	0	0	2m3s

❌ Some tests failed!

Name	Failure Message
❌ ExperimentStore CAS contracts > plans chunked-to-chunked growth and chunked-to-inline shrink independently	Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
❌ ExperimentDetail > starts an asynchronous deletion from the confirmation dialog	Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".

Github Test Reporter

Import observation taxonomy metadata through a focused shared subpath so the standalone CLI bundle does not retain server-only Redis loading. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70

github-actions · 2026-10-17T01:00:55Z

Test Results (Node.js 22)

test: Run #276

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3531	3529	2	0	0	0	0	1m58s

❌ Some tests failed!

Name	Failure Message
❌ ExperimentStore CAS contracts > plans chunked-to-chunked growth and chunked-to-inline shrink independently	Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
❌ ExperimentDetail > starts an asynchronous deletion from the confirmation dialog	Test timed out in 5000ms. If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".

Github Test Reporter

Reduce synthetic membership work while preserving chunked layout coverage, and bound the asynchronous deletion interaction under full-suite load. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70

github-actions · 2026-10-17T01:18:29Z

Test Results (Node.js 22)

test: Run #285

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3533	3533	0	0	0	0	0	1m17s

🎉 All tests passed!

Github Test Reporter

Apply the existing ten-second bound to the interaction-heavy ExperimentDetail suite so coverage runs remain stable under hosted runner contention. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70

github-actions · 2026-10-17T01:30:57Z

Test Results (Node.js 22)

test: Run #287

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3533	3533	0	0	0	0	0	1m56s

🎉 All tests passed!

Github Test Reporter

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70

github-actions · 2026-10-17T03:57:59Z

Test Results (Node.js 22)

test: Run #300

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3536	3536	0	0	0	0	0	1m57s

🎉 All tests passed!

Github Test Reporter

Cedric Vidal (cedricvidal) added 15 commits June 1, 2026 23:01

test(cli): regenerate command-tree snapshot for new typed-prompt/expe…

9b1b8e1

…riment flags Captures --type (prompt-feature/task-prompt create), --agents-md, --experiment-id, and --group-by added in the CLI step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Cedric Vidal (cedricvidal) added the author: cedricvidal label Jul 29, 2026

Cedric Vidal (cedricvidal) mentioned this pull request Jul 29, 2026

feat: AGENTS.md prompt support (closes #732) #1161

Closed

Cedric Vidal (cedricvidal) removed this pull request from stack #1497 October 17, 2026 00:30

Cedric Vidal (cedricvidal) changed the base branch from cedricvidal-experiment-support to cedricvidal-experiment-lifecycle October 17, 2026 00:30

Cedric Vidal (cedricvidal) added this pull request to stack #1518 October 17, 2026 00:58

Cedric Vidal (cedricvidal) mentioned this pull request Oct 17, 2026

feat: add submission-level generated readouts #1495

Draft

5 of 6 tasks

github-actions Bot added area: portal Scope web portal, pages, and user-facing components. topic: ui Portal, visual presentation, layout, and interface components. labels Oct 17, 2026

Merge corrected experiment lifecycle UX into GEPA

1744485

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ed7b459-4ebc-4313-892c-fc5f23d5bf70

Navigation Menu

Repository navigation

feat: GEPA prompt optimizer with Scope adapter (AGENTS.md optimization) - #1031

feat: GEPA prompt optimizer with Scope adapter (AGENTS.md optimization)#1031
Cedric Vidal (cedricvidal) wants to merge 25 commits into
cedricvidal-experiment-lifecyclefrom
cedricvidal-microsoft/gepa-scope-adapter

Cedric Vidal (cedricvidal) commented Jul 29, 2026 •

edited

Loading

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Cedric Vidal (cedricvidal) commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Demos

Before

After

Testing

Documentation and compatibility

Checklist

Uh oh!

Cedric Vidal (cedricvidal) commented Jul 29, 2026

Test Results (Node.js 22)

test: Run #1839

🎉 All tests passed!

Uh oh!

github-actions Bot commented Oct 17, 2026

Test Results (Node.js 22)

test: Run #274

❌ Some tests failed!

Uh oh!

github-actions Bot commented Oct 17, 2026

Test Results (Node.js 22)

test: Run #276

❌ Some tests failed!

Uh oh!

github-actions Bot commented Oct 17, 2026

Test Results (Node.js 22)

test: Run #285

🎉 All tests passed!

Uh oh!

github-actions Bot commented Oct 17, 2026

Test Results (Node.js 22)

test: Run #287

🎉 All tests passed!

Uh oh!

github-actions Bot commented Oct 17, 2026

Test Results (Node.js 22)

test: Run #300

🎉 All tests passed!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Cedric Vidal (cedricvidal) commented Jul 29, 2026 •

edited

Loading