From 1fb829c008fadb5fa02f4662d4be328d0a930083 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Sun, 28 Jun 2026 19:16:42 -0700 Subject: [PATCH 01/17] feat(observations): observation criteria + pp-taxonomy handler, CLI, config, docs Add kind:gate|observation discriminator + hard-coded taxonomyElementId to criteria; record-only per-iteration boolean observations evaluated post-run by the pp-taxonomy handler (reuses judge via JudgeClient, dependsOn pp-atif) and stored as turn.observationResults. Request-level opt-in via observations[]. CLI, migration 021, deploy wiring, config seeds, and docs included. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ENV_VARIABLES.md | 8 +- README.md | 1 + .../openapi-snapshot.test.ts.snap | 93 ++++++++++++++++ apps/api/src/endpoints.test.ts | 28 +++++ apps/api/src/routes/criteria.ts | 16 ++- apps/api/src/routes/requests/index.ts | 51 ++++++++- .../__snapshots__/cli-commands.test.ts.snap | 5 + apps/cli/src/commands/criteria.ts | 36 ++++++- apps/cli/src/commands/run.ts | 7 +- apps/cli/src/run-get-action.ts | 45 +++++++- apps/cli/src/utils/yaml-mappers.ts | 9 +- apps/judge/src/index.ts | 27 ++++- .../post-processor/handler.taxonomy.yaml | 17 +++ .../src/handlers/taxonomy-handler.test.ts | 80 ++++++++++++++ .../src/handlers/taxonomy-handler.ts | 90 ++++++++++++++++ apps/workers/post-processor/src/index.ts | 2 + .../criteria/auth_configured_correctly.yaml | 8 ++ .../dependency_added_then_removed.yaml | 9 ++ config/criteria/idiomatic_api_usage.yaml | 8 ++ docker-compose.dev.yml | 4 + docker-compose.yml | 19 ++++ docs/architecture/app-design.md | 7 +- docs/architecture/criteria-provider.md | 4 + docs/architecture/overview.md | 1 + docs/architecture/post-processing.md | 21 +++- .../migrations/021-backfill-criteria-kind.ts | 47 ++++++++ .../db-migrations/src/required-migrations.ts | 1 + .../src/criteria/criteria-provider-api.ts | 4 + .../src/criteria/criteria-provider-fs.ts | 13 +++ .../src/criteria/criteria-store.test.ts | 59 +++++++++++ .../shared/src/criteria/criteria-store.ts | 100 +++++++++++++++++- packages/shared/src/criteria/index.ts | 1 + .../src/criteria/observation-taxonomy.ts | 43 ++++++++ packages/shared/src/judge/judge-client.ts | 4 + packages/shared/src/schemas/criteria.ts | 12 ++- packages/shared/src/schemas/request.ts | 3 + packages/shared/src/schemas/schemas.test.ts | 27 +++++ packages/shared/src/storage/blob-storage.ts | 23 ++++ packages/shared/src/types/types.ts | 54 ++++++++++ 39 files changed, 964 insertions(+), 23 deletions(-) create mode 100644 apps/workers/post-processor/handler.taxonomy.yaml create mode 100644 apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts create mode 100644 apps/workers/post-processor/src/handlers/taxonomy-handler.ts create mode 100644 config/criteria/auth_configured_correctly.yaml create mode 100644 config/criteria/dependency_added_then_removed.yaml create mode 100644 config/criteria/idiomatic_api_usage.yaml create mode 100644 packages/db-migrations/src/migrations/021-backfill-criteria-kind.ts create mode 100644 packages/shared/src/criteria/observation-taxonomy.ts diff --git a/ENV_VARIABLES.md b/ENV_VARIABLES.md index ec5d268e1..3001c4b16 100644 --- a/ENV_VARIABLES.md +++ b/ENV_VARIABLES.md @@ -341,7 +341,13 @@ How often the post-processor dispatcher polls for completed runs needing post-pr **Default:** `post-processor-queue` **Type:** string -Azure Storage Queue name used by both the scheduler (to enqueue post-processing work) and the post-processor worker (to dequeue). Must match between the two services. +Azure Storage Queue name used by both the scheduler (to enqueue post-processing work) and the post-processor worker (to dequeue). Must match between the two services. The `pp-taxonomy` observation handler shares this queue. + +### JUDGE_URL +**Default:** falls back to `JUDGE_SERVICE_URL`, then `http://localhost:3002` +**Type:** URL string + +Base URL of the judge service used by the `pp-taxonomy` post-process handler to evaluate observation criteria per iteration (snapshot + ATIF trajectory → boolean + evidence). The handler reuses the shared `JudgeClient`, so `JUDGE_CLIENT_TIMEOUT` / `JUDGE_CLIENT_RETRIES` apply. In Docker/K8s the worker reads `JUDGE_SERVICE_URL` from `worker-config`; no extra var needed. See issue #1156 and [docs/architecture/post-processing.md](docs/architecture/post-processing.md). ### SCHEDULER_URL **Default:** (not set) diff --git a/README.md b/README.md index 38c22b8dd..e8df7301b 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,7 @@ | **coder-acp-copilot** | GitHub Copilot agent worker (ACP SDK) | | **coder-acp-claude-code** | Claude Code agent worker (ACP SDK) | | **report-generator** | Post-run evaluation report generator (Copilot SDK) | +| **post-processor** | DAG post-run handlers: `pp-atif` (ATIF trajectory) + `pp-taxonomy` (boolean observations via judge) | | **model-scanners** | Feature detection for Copilot and Anthropic models | | **version-checkers** | Poll for new releases of agents/tools (`acp-copilot`, `claude-code`, `vscode-electron`) | diff --git a/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap b/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap index d99d079de..3d5d038c9 100644 --- a/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap +++ b/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap @@ -512,6 +512,12 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "judgeFeedback": { "type": "string", }, + "observationResults": { + "items": { + "$ref": "#/components/schemas/CriterionResult", + }, + "type": "array", + }, "passed": { "type": "boolean", }, @@ -653,9 +659,25 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "pattern": "^[a-z][a-z0-9_]*$", "type": "string", }, + "kind": { + "default": "gate", + "enum": [ + "gate", + "observation", + ], + "type": "string", + }, "prompt": { "type": "string", }, + "taxonomyElementId": { + "enum": [ + "dimension:idiomatic-use", + "dimension:dependency-currency", + "dimension:configuration-correctness", + ], + "type": "string", + }, }, "required": [ "id", @@ -873,6 +895,12 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "model": { "type": "string", }, + "observations": { + "items": { + "type": "string", + }, + "type": "array", + }, "persona": { "$ref": "#/components/schemas/Persona", }, @@ -1020,9 +1048,24 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "id": { "type": "string", }, + "kind": { + "enum": [ + "gate", + "observation", + ], + "type": "string", + }, "prompt": { "type": "string", }, + "taxonomyElementId": { + "enum": [ + "dimension:idiomatic-use", + "dimension:dependency-currency", + "dimension:configuration-correctness", + ], + "type": "string", + }, }, "required": [ "id", @@ -1068,9 +1111,24 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "id": { "type": "string", }, + "kind": { + "enum": [ + "gate", + "observation", + ], + "type": "string", + }, "prompt": { "type": "string", }, + "taxonomyElementId": { + "enum": [ + "dimension:idiomatic-use", + "dimension:dependency-currency", + "dimension:configuration-correctness", + ], + "type": "string", + }, "updatedAt": { "format": "date-time", "type": [ @@ -2430,6 +2488,12 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "model": { "type": "string", }, + "observations": { + "items": { + "type": "string", + }, + "type": "array", + }, "persona": { "$ref": "#/components/schemas/Persona", }, @@ -3087,9 +3151,24 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` }, "type": "array", }, + "kind": { + "enum": [ + "gate", + "observation", + ], + "type": "string", + }, "prompt": { "type": "string", }, + "taxonomyElementId": { + "enum": [ + "dimension:idiomatic-use", + "dimension:dependency-currency", + "dimension:configuration-correctness", + ], + "type": "string", + }, }, "type": "object", }, @@ -4132,6 +4211,20 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "type": "string", }, }, + { + "description": "Filter by criterion kind", + "in": "query", + "name": "kind", + "required": false, + "schema": { + "description": "Filter by criterion kind", + "enum": [ + "gate", + "observation", + ], + "type": "string", + }, + }, ], "responses": { "200": { diff --git a/apps/api/src/endpoints.test.ts b/apps/api/src/endpoints.test.ts index 0cabc44eb..34a89df39 100644 --- a/apps/api/src/endpoints.test.ts +++ b/apps/api/src/endpoints.test.ts @@ -192,6 +192,34 @@ describe("API Endpoints", () => { expect(res.status).toBe(400); }); + + it("creates an observation criterion with a taxonomy element", async () => { + (mocks.criteriaCollection.findOne as any).mockResolvedValue(null); + + const res = await request(app) + .post("/api/v1/criteria") + .send({ + id: "obs_crit", + prompt: "Did the agent use current deps?", + dependsOn: [], + kind: "observation", + taxonomyElementId: "dimension:dependency-currency", + }); + + expect(res.status).toBe(201); + expect(res.body).toHaveProperty("kind", "observation"); + expect(res.body).toHaveProperty("taxonomyElementId", "dimension:dependency-currency"); + }); + + it("rejects an unknown taxonomyElementId", async () => { + (mocks.criteriaCollection.findOne as any).mockResolvedValue(null); + + const res = await request(app) + .post("/api/v1/criteria") + .send({ id: "bad_tax", prompt: "x", dependsOn: [], kind: "observation", taxonomyElementId: "dimension:nope" }); + + expect(res.status).toBe(400); + }); }); describe("POST /api/v1/criteria/seed", () => { diff --git a/apps/api/src/routes/criteria.ts b/apps/api/src/routes/criteria.ts index 3d1279a9f..40aee5d0c 100644 --- a/apps/api/src/routes/criteria.ts +++ b/apps/api/src/routes/criteria.ts @@ -210,11 +210,13 @@ apiRoute(ctx.app, ctx.registry, { q: z.string().optional(), ids: z.string().optional().describe("Comma-separated criterion IDs to include"), ancestors: z.enum(["true", "false"]).optional().describe("When true and ids is set, also include dependency ancestors"), + kind: z.enum(["gate", "observation"]).optional().describe("Filter by criterion kind"), }), response: z.array(CriteriaResponseSchema), handler: async (req, res) => { const q = req.query.q; const idsParam = req.query.ids; + const kind = req.query.kind; // `q` (regex search) and `ids` (exact set + optional ancestor resolution) // are mutually exclusive: applying the regex filter first would silently @@ -231,6 +233,12 @@ apiRoute(ctx.app, ctx.registry, { { prompt: { $regex: q, $options: "i" } }, ]; } + // kind:"gate" includes legacy rows where kind is absent (backfilled to gate). + if (kind === "gate") { + filter.$and = [{ $or: [{ kind: "gate" }, { kind: { $exists: false } }] }]; + } else if (kind === "observation") { + filter.kind = "observation"; + } let criteria = await ctx.criteriaCollection.find(filter).toArray(); criteria.sort((a, b) => a.id.localeCompare(b.id)); @@ -419,9 +427,9 @@ apiRoute(ctx.app, ctx.registry, { 409: { description: "Criterion already exists" }, }, handler: async (req, res) => { - const { id, prompt, dependsOn = [], gates } = req.body; + const { id, prompt, dependsOn = [], gates, kind, taxonomyElementId } = req.body; try { - const doc = await getCriteriaStore().create({ id, prompt, dependsOn, gates }); + const doc = await getCriteriaStore().create({ id, prompt, dependsOn, gates, kind, taxonomyElementId }); res.status(201).json(doc); } catch (err) { if (!sendStoreError(res, err)) throw err; @@ -444,9 +452,9 @@ apiRoute(ctx.app, ctx.registry, { }, handler: async (req, res) => { const { id } = req.params; - const { prompt, dependsOn, gates } = req.body; + const { prompt, dependsOn, gates, kind, taxonomyElementId } = req.body; try { - const updated = await getCriteriaStore().update(id, { prompt, dependsOn, gates }); + const updated = await getCriteriaStore().update(id, { prompt, dependsOn, gates, kind, taxonomyElementId }); res.json(updated); } catch (err) { if (!sendStoreError(res, err)) throw err; diff --git a/apps/api/src/routes/requests/index.ts b/apps/api/src/routes/requests/index.ts index b6fcd4b1d..143735329 100644 --- a/apps/api/src/routes/requests/index.ts +++ b/apps/api/src/routes/requests/index.ts @@ -95,6 +95,9 @@ async function validateGatesForSubmit( if (!doc) { return `Gate '${gc.gate}' references unknown criterion '${cid}'.`; } + if ((doc.kind ?? "gate") !== "gate") { + return `Criterion '${cid}' is an observation and cannot be used as a gate criterion.`; + } if (!isCriterionCompatibleWithGate(doc.gates as GateId[] | undefined, gc.gate)) { return `Criterion '${cid}' is not compatible with the '${gc.gate}' gate.`; } @@ -122,7 +125,32 @@ async function validateGatesForSubmit( } /** - * Materialize free-text gate prompts into typed prompt entities (docs/design/ + * Validate the submit payload's observation-criteria ids: each must be an + * active kind:"observation" criterion. Gate ids are rejected. Empty/absent is + * valid (no observations evaluated). See issue #1156. + */ +async function validateObservationsForSubmit( + ctx: RouteContext, + observations: string[], +): Promise { + if (observations.length === 0) return null; + const ids = [...new Set(observations)]; + const docs = await ctx.criteriaCollection + .find({ id: { $in: ids }, deletedAt: { $exists: false } }) + .toArray(); + const byId = new Map(docs.map((c) => [c.id, c])); + for (const id of ids) { + const doc = byId.get(id); + if (!doc) return `Observation references unknown criterion '${id}'.`; + if ((doc.kind ?? "gate") !== "observation") { + return `Criterion '${id}' is a gate and cannot be used as an observation.`; + } + } + return null; +} + +/** + * Resolve free-text gate prompts into typed prompt entities before validation (see * gates.md §4.3/§4.4). For every non-Select gate that supplies `promptText`, the * text is content-addressed via `taskPromptStore.findOrCreate(text, gate)` — * idempotent and parallel to how the request task prompt is materialized — and @@ -168,7 +196,7 @@ apiRoute(ctx.app, ctx.registry, { response: z.union([RequestResponseSchema, z.array(RequestResponseSchema)]), successStatus: 201, handler: async (req, res) => { - const { scenario: scenarioObj, persona: personaObj, maxIterations, personaInstructions, count = 1, promptFeatureExtractionId, model: requestedModel, reasoningEffort: requestedReasoningEffort, mcpServers: mcpServerSlugs, skills: skillSlugs, extensions: extensionIds, agentVersion: requestedAgentVersion, profileId: requestedProfileSpec, profileVariations, priority: requestedPriority, gates: requestedGates, codebase: codebaseSpec, codebaseRevisionId: requestedCodebaseRevisionId } = req.body; + const { scenario: scenarioObj, persona: personaObj, maxIterations, personaInstructions, count = 1, promptFeatureExtractionId, model: requestedModel, reasoningEffort: requestedReasoningEffort, mcpServers: mcpServerSlugs, skills: skillSlugs, extensions: extensionIds, agentVersion: requestedAgentVersion, profileId: requestedProfileSpec, profileVariations, priority: requestedPriority, gates: requestedGates, observations: requestedObservations, codebase: codebaseSpec, codebaseRevisionId: requestedCodebaseRevisionId } = req.body; let worker = req.query.worker as string | undefined; // Resolve the optional per-run codebase selection into a concrete revision id. @@ -677,6 +705,17 @@ apiRoute(ctx.app, ctx.registry, { } } + // Validate the optional observation-criteria selection (issue #1156). + const observationsProvided = Array.isArray(requestedObservations) && requestedObservations.length > 0; + const resolvedObservations: string[] = observationsProvided ? (requestedObservations as string[]) : []; + if (observationsProvided) { + const obsError = await validateObservationsForSubmit(ctx, resolvedObservations); + if (obsError) { + res.status(400).json({ error: obsError }); + return; + } + } + // Validate count if provided if (typeof count !== "number" || count < 1 || count > 10) { res.status(400).json({ error: "count must be a number between 1 and 10" }); @@ -887,6 +926,12 @@ apiRoute(ctx.app, ctx.registry, { ) : undefined; + // Persisted observation-criteria selection (issue #1156); deduped, evaluated + // by the pp-taxonomy post-processor. Absent/empty = no observations evaluated. + const persistedObservations: string[] | undefined = observationsProvided + ? [...new Set(resolvedObservations)] + : undefined; + // Generate a submission ID to group all runs from this request const submissionId = uuidv4(); @@ -921,6 +966,7 @@ apiRoute(ctx.app, ctx.registry, { ...(profileId ? { profileId } : {}), ...(profileVersionId ? { profileVersionId } : {}), ...(persistedGates ? { gates: persistedGates } : {}), + ...(persistedObservations ? { observations: persistedObservations } : {}), submissionId, // Mint a distinct run id for the first attempt. Blob artifacts // are scoped under `{requestId}/runs/{runId}/...` so retries @@ -981,6 +1027,7 @@ apiRoute(ctx.app, ctx.registry, { ...(profileId ? { profileId } : {}), ...(profileVersionId ? { profileVersionId } : {}), ...(persistedGates ? { gates: persistedGates } : {}), + ...(persistedObservations ? { observations: persistedObservations } : {}), submissionId, // Mint a distinct run id for the first attempt. Blob artifacts // are scoped under `{requestId}/runs/{runId}/...` so retries diff --git a/apps/cli/src/__snapshots__/cli-commands.test.ts.snap b/apps/cli/src/__snapshots__/cli-commands.test.ts.snap index 065ced3aa..f22265fbc 100644 --- a/apps/cli/src/__snapshots__/cli-commands.test.ts.snap +++ b/apps/cli/src/__snapshots__/cli-commands.test.ts.snap @@ -131,7 +131,9 @@ exports[`CLI command registration > full command tree matches snapshot 1`] = ` "--depends-on", "--gates", "--id", + "--kind", "--prompt", + "--taxonomy", "--url", ], }, @@ -179,7 +181,9 @@ exports[`CLI command registration > full command tree matches snapshot 1`] = ` "--depends-on", "--gates", "--id", + "--kind", "--prompt", + "--taxonomy", "--url", ], }, @@ -661,6 +665,7 @@ exports[`CLI command registration > full command tree matches snapshot 1`] = ` "--message", "--model", "--no-stream", + "--observations", "--persona", "--profile", "--profile-variations-file", diff --git a/apps/cli/src/commands/criteria.ts b/apps/cli/src/commands/criteria.ts index 17d181422..7e890f9ac 100644 --- a/apps/cli/src/commands/criteria.ts +++ b/apps/cli/src/commands/criteria.ts @@ -46,7 +46,7 @@ criteria process.exit(1); } - const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[] }>; + const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string }>; if (items.length === 0) { if (!isMachineReadable(format)) console.log(warnBanner("No criteria found.")); return; @@ -60,6 +60,7 @@ criteria { key: 'id', label: 'ID', tableFormatter: (c: any) => value(c.id), }, + { key: 'kind', label: 'Kind', formatter: (c: any) => c.kind ?? 'gate' }, { key: 'dependsOn', label: 'Deps', formatter: (c: any) => String((c.dependsOn ?? []).length) }, { key: 'gates', label: 'Gates', formatter: (c: any) => formatGateList(c.gates) }, { key: 'prompt', label: 'Prompt', tableFormatter: (c: any) => { @@ -96,6 +97,7 @@ criteria const c = await response.json() as { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; + kind?: string; taxonomyElementId?: string; dependents: string[]; createdAt: string; updatedAt?: string; }; @@ -103,6 +105,8 @@ criteria const fields: DisplayField[] = [ { key: 'id', label: 'ID' }, { key: 'prompt', label: 'Prompt' }, + { key: 'kind', label: 'Kind', formatter: (item: any) => item.kind ?? 'gate' }, + { key: 'taxonomyElementId', label: 'Taxonomy', formatter: (item: any) => item.taxonomyElementId ?? '(none)' }, { key: 'dependsOn', label: 'Depends On', formatter: (item: any) => (item.dependsOn ?? []).join(', ') || '(none)' }, { key: 'gates', label: 'Gates', formatter: (item: any) => formatGateList(item.gates) }, { key: 'dependents', label: 'Dependents', formatter: (item: any) => (item.dependents ?? []).join(', ') || '(none)' }, @@ -124,6 +128,10 @@ criteria console.log(`${label('Depends on:')} ${dimTimestamp('(none — root criterion)')}`); } console.log(`${label('Gates:')} ${value(formatGateList(c.gates))}`); + console.log(`${label('Kind:')} ${value(c.kind ?? 'gate')}`); + if (c.taxonomyElementId) { + console.log(`${label('Taxonomy:')} ${value(c.taxonomyElementId)}`); + } if (c.dependents.length > 0) { console.log(`${label('Dependents:')} ${c.dependents.map(d => value(d)).join(', ')}`); } @@ -142,6 +150,8 @@ criteria .requiredOption("--prompt ", "Evaluation prompt for the judge") .option("-d, --depends-on ", "IDs of parent criteria") .option("--gates ", "Compatible gates (space/comma separated), or all/* for unrestricted") + .option("--kind ", "Criterion kind: gate (default) or observation") + .option("--taxonomy ", "Taxonomy element id (observation only), e.g. dimension:dependency-currency") .option("-u, --url ", "API base URL", getDefaultApiUrl()) .action(async (options) => { try { @@ -154,6 +164,8 @@ criteria } const gates = parseGateListOption(options.gates); if (gates !== undefined) body.gates = gates; + if (options.kind) body.kind = options.kind; + if (options.taxonomy) body.taxonomyElementId = options.taxonomy; const response = await fetch(`${normalizeUrl(options.url)}/api/v1/criteria`, { method: "POST", @@ -182,6 +194,8 @@ criteria .option("--prompt ", "New evaluation prompt") .option("-d, --depends-on ", "New parent criteria IDs (replaces all)") .option("--gates ", "New compatible gates (space/comma separated), or all/* for unrestricted") + .option("--kind ", "Criterion kind: gate or observation") + .option("--taxonomy ", "Taxonomy element id (observation only)") .option("-u, --url ", "API base URL", getDefaultApiUrl()) .action(async (options) => { try { @@ -190,9 +204,11 @@ criteria if (options.dependsOn !== undefined) body.dependsOn = options.dependsOn; const gates = parseGateListOption(options.gates); if (gates !== undefined) body.gates = gates; + if (options.kind) body.kind = options.kind; + if (options.taxonomy) body.taxonomyElementId = options.taxonomy; if (Object.keys(body).length === 0) { - console.error(errorText("Error: provide --prompt, --depends-on, and/or --gates")); + console.error(errorText("Error: provide --prompt, --depends-on, --gates, --kind, and/or --taxonomy")); process.exit(1); } @@ -347,7 +363,7 @@ criteria process.exit(1); } - const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[] }>; + const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string }>; if (items.length === 0) { console.error(errorText("No criteria found to export.")); @@ -383,6 +399,15 @@ criteria if (c.dependsOn && c.dependsOn.length > 0) { doc.depends_on = c.dependsOn; } + if (c.gates && c.gates.length > 0) { + doc.gates = c.gates; + } + if (c.kind && c.kind !== 'gate') { + doc.kind = c.kind; + } + if (c.taxonomyElementId) { + doc.taxonomy_element_id = c.taxonomyElementId; + } return doc; }); @@ -435,7 +460,7 @@ criteria } // Parse all criteria from files (supports multi-document YAML) - const allCriteria: Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[] }> = []; + const allCriteria: Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string }> = []; const parseErrors: string[] = []; for (const file of yamlFiles) { @@ -480,7 +505,8 @@ criteria const deps = (c.dependsOn ?? []).length; const depsStr = deps > 0 ? ` ${dimTimestamp(`(${deps} dep${deps > 1 ? 's' : ''})`)}` : ''; const gatesStr = c.gates ? ` ${dimTimestamp(`[${formatGateList(c.gates)}]`)}` : ''; - console.log(` ${value(c.id)}${depsStr}${gatesStr}`); + const kindStr = c.kind === 'observation' ? ` ${dimTimestamp(``)}` : ''; + console.log(` ${value(c.id)}${depsStr}${gatesStr}${kindStr}`); } if (options.dryRun) { diff --git a/apps/cli/src/commands/run.ts b/apps/cli/src/commands/run.ts index 7da9840ea..605460817 100644 --- a/apps/cli/src/commands/run.ts +++ b/apps/cli/src/commands/run.ts @@ -49,10 +49,11 @@ run .addOption(new Option("--base-profile ", "Deprecated alias for --profile.").hideHelp()) .option("--profile-variations-file ", "Path to JSON file containing profile variation entries") .option("--gates ", "GateConfig[] JSON or path/@path to a JSON file for gated runs") + .option("--observations ", "Observation criteria ids to record post-run (non-gating)") .option("-u, --url ", "API base URL", process.env.SCOPE_API_URL || "http://localhost:3100") .option("--no-stream", "Don't stream logs, just submit") .action(async (options, command) => { - const { scenario, persona, traits, worker, url, stream, maxIterations, model, reasoningEffort, mcpServers: mcpServerSlugs, skills: skillSlugs, codebase: codebaseRef, extensions: extensionIds, agentVersion, profile, baseProfile, profileVariationsFile, gates: gatesOption } = options; + const { scenario, persona, traits, worker, url, stream, maxIterations, model, reasoningEffort, mcpServers: mcpServerSlugs, skills: skillSlugs, codebase: codebaseRef, extensions: extensionIds, agentVersion, profile, baseProfile, profileVariationsFile, gates: gatesOption, observations: observationIds } = options; // `--profile` is the documented flag; `--base-profile` is kept as a hidden // back-compat alias. Both resolve to the same request `profileId`. const profileId = profile ?? baseProfile; @@ -127,6 +128,9 @@ run if (gatesOption) { body.gates = parseGatesOption(gatesOption, maxIterations); } + if (observationIds && observationIds.length > 0) { + body.observations = observationIds; + } if (profileVariationsFile) { if (!profileId) { @@ -176,6 +180,7 @@ run if (result.reasoningEffort) console.log(`${label('Reasoning Effort:')} ${value(result.reasoningEffort)}`); console.log(`${label('Mode:')} ${value(result.mode || 'one-shot')}`); if (Array.isArray(body.gates)) console.log(`${label('Gates:')} ${value(String(body.gates.length))}`); + if (Array.isArray(body.observations)) console.log(`${label('Observations:')} ${value(String(body.observations.length))}`); console.log(`${label('Status:')} ${value(result.status)}`); // Display warnings (e.g. model effort compatibility) diff --git a/apps/cli/src/run-get-action.ts b/apps/cli/src/run-get-action.ts index c3090b494..e2a9c9a8b 100644 --- a/apps/cli/src/run-get-action.ts +++ b/apps/cli/src/run-get-action.ts @@ -5,7 +5,7 @@ * Action handler for `run get` subcommand — extracted for testability. */ import { colorLevel, dimTimestamp, errorText, successText, label, value, banner, warnBanner, criterionIcon } from "./utils/style.js"; -import { GATE_METADATA, GATE_ORDER, type ConversationTurn, type GateId, type GateRunSummary, type RequestDocument } from "shared"; +import { GATE_METADATA, GATE_ORDER, TAXONOMY_ELEMENT_METADATA, type ConversationTurn, type GateId, type GateRunSummary, type RequestDocument, type TaxonomyElementId } from "shared"; import { formatData, isMachineReadable } from "./utils/formatters.js"; import type { OutputFormat, DisplayField } from "./utils/types.js"; @@ -172,6 +172,49 @@ export async function runGetAction(options: RunGetOptions): Promise { } } + // Observations (kind:"observation" criteria), grouped by taxonomy dimension. + const turnsWithObs = (rs?.turns ?? []).filter((t) => (t.observationResults?.length ?? 0) > 0); + if (turnsWithObs.length > 0) { + // Build criterionId → taxonomyElementId map from the criteria store (best effort). + const taxById = new Map(); + try { + const cr = await fetch(`${normalizeUrl(options.url)}/api/v1/criteria?kind=observation`); + if (cr.ok) { + const obs = await cr.json() as Array<{ id: string; taxonomyElementId?: TaxonomyElementId }>; + for (const o of obs) taxById.set(o.id, o.taxonomyElementId); + } + } catch { /* grouping is best-effort */ } + + // detected/evaluated tally per criterion id across iterations. + const tally = new Map(); + for (const t of turnsWithObs) { + for (const r of t.observationResults ?? []) { + const e = tally.get(r.criterionId) ?? { detected: 0, evaluated: 0 }; + e.evaluated += 1; + if (r.passed) e.detected += 1; + tally.set(r.criterionId, e); + } + } + + const byDim = new Map(); + for (const id of tally.keys()) { + const dim = taxById.get(id) ?? "unclassified"; + const arr = byDim.get(dim) ?? []; + arr.push(id); + byDim.set(dim, arr); + } + + console.log(`\n${banner('─── Observations ───')}`); + for (const [dim, ids] of byDim) { + const dimLabel = dim === "unclassified" ? "Unclassified" : (TAXONOMY_ELEMENT_METADATA[dim as TaxonomyElementId]?.label ?? dim); + console.log(` ${label(dimLabel)}`); + for (const id of ids.sort()) { + const { detected, evaluated } = tally.get(id)!; + console.log(` ${value(id)} — ${detected}/${evaluated} iterations detected`); + } + } + } + // Error if (rs?.error) { console.log(`${label('Error:')} ${errorText(rs.error)}`); diff --git a/apps/cli/src/utils/yaml-mappers.ts b/apps/cli/src/utils/yaml-mappers.ts index 04bc54fb6..cad0c23dd 100644 --- a/apps/cli/src/utils/yaml-mappers.ts +++ b/apps/cli/src/utils/yaml-mappers.ts @@ -12,7 +12,7 @@ export function mapYamlCriterion( filename: string, _docIndex: number, options: { includeGates?: boolean } = {}, -): { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[] } | null { +): { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string } | null { const id = doc.id as string | undefined; const prompt = doc.prompt as string | undefined; if (!id || !prompt) return null; @@ -24,11 +24,18 @@ export function mapYamlCriterion( const gatesRaw = doc.gates as string | string[] | undefined; const gates = options.includeGates ? parseGateListOption(gatesRaw) : undefined; + const kindRaw = doc.kind as string | undefined; + const kind = kindRaw ? String(kindRaw).trim() : undefined; + const taxRaw = (doc.taxonomy_element_id ?? doc.taxonomyElementId) as string | undefined; + const taxonomyElementId = taxRaw ? String(taxRaw).trim() : undefined; + return { id: id.trim(), prompt: prompt.trim(), ...(dependsOn && dependsOn.length > 0 ? { dependsOn } : {}), ...(gates !== undefined ? { gates } : {}), + ...(kind ? { kind } : {}), + ...(taxonomyElementId ? { taxonomyElementId } : {}), }; } diff --git a/apps/judge/src/index.ts b/apps/judge/src/index.ts index e9718131a..0dea452cd 100644 --- a/apps/judge/src/index.ts +++ b/apps/judge/src/index.ts @@ -52,7 +52,7 @@ app.post( const startTime = Date.now(); try { - const { snapshotUrl, criteria, conversationHistory, personaInstructions, requestId, gate, toolCallsUrl } = req.body; + const { snapshotUrl, criteria, conversationHistory, personaInstructions, requestId, gate, toolCallsUrl, atifUrl } = req.body; // Validate required fields if (!snapshotUrl || typeof snapshotUrl !== "string") { @@ -106,6 +106,31 @@ app.post( } } + // Load the ATIF trajectory (normalized superset of tool calls) when + // provided — used by pp-taxonomy to evaluate observations against the + // full agent trajectory, not just the end-state snapshot. Exposed to the + // judge as a synthetic tool output so trajectory-only observations (e.g. + // a dependency added then removed mid-run) are detectable. Issue #1156. + if (atifUrl && typeof atifUrl === "string") { + try { + const atif = await blobStorage.downloadJson(atifUrl); + toolCalls = [ + ...toolCalls, + { + id: "atif-trajectory", + name: "agent_trajectory_atif", + arguments: {}, + response: typeof atif === "string" ? atif : JSON.stringify(atif), + }, + ]; + console.log(`[judge] Loaded ATIF trajectory from ${atifUrl}`); + } catch (err) { + console.warn( + `[judge] Failed to load ATIF from ${atifUrl}: ${err instanceof Error ? err.message : String(err)}`, + ); + } + } + // Build onProgress callback that publishes criterion results via Redis const onProgress = (requestId && logPublisher) ? (result: import("shared").CriterionResult) => { diff --git a/apps/workers/post-processor/handler.taxonomy.yaml b/apps/workers/post-processor/handler.taxonomy.yaml new file mode 100644 index 000000000..37411625f --- /dev/null +++ b/apps/workers/post-processor/handler.taxonomy.yaml @@ -0,0 +1,17 @@ +# Post-process handler registration metadata for the observation evaluator. +# Read by the generic register-handler script and POSTed to the +# scheduler's POST /handlers/register endpoint on deploy. +# +# pp-taxonomy records per-iteration boolean observations (kind:"observation" +# criteria) by reusing the judge. It depends on pp-atif so each turn's ATIF +# trajectory is available, giving the judge codebase (snapshot) + trajectory +# (ATIF) access. Shares the post-processor-queue with pp-atif (same worker). +# See issue #1156. +_id: pp-taxonomy +type: post-process-handler +version: 1 +queue: post-processor-queue +selector: taxonomy +autoBackfill: true +dependsOn: + - pp-atif diff --git a/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts b/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts new file mode 100644 index 000000000..18831592a --- /dev/null +++ b/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts @@ -0,0 +1,80 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import { describe, it, expect, vi, beforeEach } from "vitest"; +import { JudgeInfrastructureError, type JudgeClient } from "shared"; +import { TaxonomyHandler } from "./taxonomy-handler.js"; +import type { HandlerContext, PostProcessorMessage } from "../types.js"; + +function makeCtx(doc: unknown): HandlerContext & { collection: { updateOne: ReturnType } } { + return { + blobStorage: {} as any, + collection: { + findOne: vi.fn().mockResolvedValue(doc), + updateOne: vi.fn().mockResolvedValue({ modifiedCount: 1 }), + } as any, + log: vi.fn().mockResolvedValue(undefined), + } as any; +} + +const msg: PostProcessorMessage = { type: "taxonomy", requestId: "req-1", runId: "run-1" }; + +describe("TaxonomyHandler", () => { + let judge: { evaluate: ReturnType }; + + beforeEach(() => { + judge = { evaluate: vi.fn() }; + }); + + it("has type 'taxonomy'", () => { + expect(new TaxonomyHandler(judge as unknown as JudgeClient).type).toBe("taxonomy"); + }); + + it("no-ops when no observations are selected", async () => { + const ctx = makeCtx({ _id: "req-1", run: { observations: [], turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }); + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + expect(judge.evaluate).not.toHaveBeenCalled(); + expect(ctx.collection.updateOne).not.toHaveBeenCalled(); + }); + + it("writes judge criteriaResults to each turn's observationResults and sends atifUrl", async () => { + judge.evaluate.mockResolvedValue({ + passed: true, + feedback: "ok", + criteriaResults: [{ criterionId: "dep_removed", passed: true, feedback: "left-pad removed", evaluated: true }], + }); + const ctx = makeCtx({ + _id: "req-1", + run: { + observations: ["dep_removed"], + turns: [{ iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }], + }, + }); + + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + + expect(judge.evaluate).toHaveBeenCalledWith( + expect.objectContaining({ snapshotUrl: "s://1", criteria: ["dep_removed"], atifUrl: "a://1" }), + ); + const call = judge.evaluate.mock.calls[0][0]; + expect(call.gate).toBeUndefined(); + expect(call.toolCallsUrl).toBeUndefined(); + expect(ctx.collection.updateOne).toHaveBeenCalledWith( + { _id: "req-1", "run.turns.iteration": 1 }, + { $set: { "run.turns.$.observationResults": [{ criterionId: "dep_removed", passed: true, feedback: "left-pad removed", evaluated: true }] } }, + ); + }); + + it("skips iterations without a snapshotUrl", async () => { + judge.evaluate.mockResolvedValue({ passed: true, feedback: "", criteriaResults: [] }); + const ctx = makeCtx({ _id: "req-1", run: { observations: ["c1"], turns: [{ iteration: 1 }] } }); + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + expect(judge.evaluate).not.toHaveBeenCalled(); + }); + + it("throws on judge infrastructure error so the handler is marked failed", async () => { + judge.evaluate.mockRejectedValue(new JudgeInfrastructureError("down", { httpStatus: 503 })); + const ctx = makeCtx({ _id: "req-1", run: { observations: ["c1"], turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }); + await expect(new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx)).rejects.toThrow("down"); + }); +}); diff --git a/apps/workers/post-processor/src/handlers/taxonomy-handler.ts b/apps/workers/post-processor/src/handlers/taxonomy-handler.ts new file mode 100644 index 000000000..3701c4cd8 --- /dev/null +++ b/apps/workers/post-processor/src/handlers/taxonomy-handler.ts @@ -0,0 +1,90 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import { JudgeClient, isRetryableJudgeError } from "shared"; +import type { ConversationTurn, CriterionResult } from "shared"; +import type { PostProcessHandler, PostProcessorMessage, HandlerContext } from "../types.js"; + +/** Minimal shape of the run document fields this handler reads/writes. */ +interface RunDoc { + run?: { + _id?: string; + observations?: string[]; + turns?: ConversationTurn[]; + }; +} + +/** + * Taxonomy Handler (pp-taxonomy): records per-iteration boolean observations. + * + * For each iteration turn with a `snapshotUrl`, reuses the judge service to + * evaluate the run's selected observation criteria (`run.observations`) against + * the codebase snapshot AND the agent trajectory (ATIF) — sending `atifUrl` in + * place of `toolCallsUrl`. Results are written to `turn.observationResults`, + * mirroring gate `criteriaResults`. Never gates, never steers the agent. + * Empty/absent `observations` → no-op. See issue #1156. + */ +export class TaxonomyHandler implements PostProcessHandler { + readonly type = "taxonomy"; + private readonly judge: JudgeClient; + + constructor(judge?: JudgeClient, judgeUrl = process.env.JUDGE_URL || process.env.JUDGE_SERVICE_URL || "http://localhost:3002") { + this.judge = judge ?? new JudgeClient(judgeUrl); + } + + async process(message: PostProcessorMessage, ctx: HandlerContext): Promise { + const { requestId, iteration } = message; + + const doc = (await ctx.collection.findOne({ _id: requestId } as any)) as RunDoc | null; + const observations = doc?.run?.observations ?? []; + const turns = doc?.run?.turns ?? []; + + if (observations.length === 0) { + await ctx.log("info", "No observations selected, skipping taxonomy evaluation"); + return; + } + if (turns.length === 0) { + await ctx.log("info", "No turns found, skipping taxonomy evaluation"); + return; + } + + const turnsToProcess = iteration ? turns.filter((t) => t.iteration === iteration) : turns; + let processedCount = 0; + + for (const turn of turnsToProcess) { + if (!turn.snapshotUrl) { + await ctx.log("info", `Iteration ${turn.iteration}: no snapshot, skipping`); + continue; + } + + try { + const result = await this.judge.evaluate({ + snapshotUrl: turn.snapshotUrl, + criteria: observations, + conversationHistory: turns, + ...(turn.atifUrl ? { atifUrl: turn.atifUrl } : {}), + requestId, + }); + + const observationResults: CriterionResult[] = result.criteriaResults ?? []; + + await ctx.collection.updateOne( + { _id: requestId, "run.turns.iteration": turn.iteration } as any, + { $set: { "run.turns.$.observationResults": observationResults } } as any, + ); + + processedCount++; + await ctx.log("info", `Iteration ${turn.iteration}: ${observationResults.length} observations recorded`); + } catch (err) { + const errMsg = err instanceof Error ? err.message : String(err); + if (isRetryableJudgeError(err)) { + await ctx.log("error", `Iteration ${turn.iteration}: judge infrastructure error, failing handler: ${errMsg}`); + throw err; + } + await ctx.log("warn", `Iteration ${turn.iteration}: observation evaluation failed: ${errMsg}`); + } + } + + await ctx.log("info", `Taxonomy evaluation complete: ${processedCount}/${turnsToProcess.length} iterations processed`); + } +} diff --git a/apps/workers/post-processor/src/index.ts b/apps/workers/post-processor/src/index.ts index b6b6ce3c9..c6a756cfb 100644 --- a/apps/workers/post-processor/src/index.ts +++ b/apps/workers/post-processor/src/index.ts @@ -4,6 +4,7 @@ import "dotenv/config"; import { PostProcessor, type PostProcessorConfig } from "./post-processor.js"; import { AtifHandler } from "./handlers/atif-handler.js"; +import { TaxonomyHandler } from "./handlers/taxonomy-handler.js"; const config: PostProcessorConfig = { mongoUri: process.env.MONGO_CONNECTION_STRING || process.env.AZURE_COSMOS_CONNECTION_STRING || "mongodb://localhost:27017", @@ -24,6 +25,7 @@ const processor = new PostProcessor(config); // Register handlers processor.registerHandler(new AtifHandler()); +processor.registerHandler(new TaxonomyHandler()); processor.start().catch((err) => { console.error("[post-processor] Fatal error:", err); diff --git a/config/criteria/auth_configured_correctly.yaml b/config/criteria/auth_configured_correctly.yaml new file mode 100644 index 000000000..8c4b62410 --- /dev/null +++ b/config/criteria/auth_configured_correctly.yaml @@ -0,0 +1,8 @@ +id: auth_configured_correctly +prompt: | + Authentication, connection, environment, and deployment settings are + configured properly (e.g. credentials/connection strings sourced from + environment or secret stores, correct endpoints, no hard-coded secrets, valid + deployment configuration). +kind: observation +taxonomy_element_id: dimension:configuration-correctness diff --git a/config/criteria/dependency_added_then_removed.yaml b/config/criteria/dependency_added_then_removed.yaml new file mode 100644 index 000000000..1706e1e64 --- /dev/null +++ b/config/criteria/dependency_added_then_removed.yaml @@ -0,0 +1,9 @@ +id: dependency_added_then_removed +prompt: | + During the run the agent installed a package and later uninstalled or removed + it before finishing (e.g. added left-pad then removed it). This is only + derivable from the agent's trajectory (mid-run tool calls), not from the final + codebase snapshot, because the dependency is absent in the end state. Inspect + the trajectory of tool calls to detect a transient add-then-remove. +kind: observation +taxonomy_element_id: dimension:dependency-currency diff --git a/config/criteria/idiomatic_api_usage.yaml b/config/criteria/idiomatic_api_usage.yaml new file mode 100644 index 000000000..13ec3e949 --- /dev/null +++ b/config/criteria/idiomatic_api_usage.yaml @@ -0,0 +1,8 @@ +id: idiomatic_api_usage +prompt: | + The agent used the product's recommended patterns and best practices when + calling its APIs (idiomatic usage), rather than working around them or using + deprecated/low-level approaches. Judge the quality of HOW the product is used, + not whether it is used at all. +kind: observation +taxonomy_element_id: dimension:idiomatic-use diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index daa4d9c9f..fe8c064ad 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -184,6 +184,10 @@ services: build: target: dev + register-handler-taxonomy: + build: + target: dev + post-processor: build: target: dev diff --git a/docker-compose.yml b/docker-compose.yml index 0bbc99234..aea2d76f7 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -659,6 +659,23 @@ services: restart: "no" entrypoint: ["npx", "tsx", "packages/shared/src/scripts/register-handler.ts"] + # Registers the pp-taxonomy handler (observation evaluator) — shares the + # post-processor worker + post-processor-queue, depends on pp-atif. Issue #1156. + register-handler-taxonomy: + build: + context: . + dockerfile: apps/workers/post-processor/Dockerfile + profiles: [post-processor] + environment: + <<: *common-env + HANDLER_YAML_PATH: apps/workers/post-processor/handler.taxonomy.yaml + SCHEDULER_URL: http://scheduler:8080 + depends_on: + scheduler: + condition: service_started + restart: "no" + entrypoint: ["npx", "tsx", "packages/shared/src/scripts/register-handler.ts"] + post-processor: build: context: . @@ -681,6 +698,8 @@ services: condition: service_started register-handler-post-processor: condition: service_completed_successfully + register-handler-taxonomy: + condition: service_completed_successfully volumes: mongodb_data: diff --git a/docs/architecture/app-design.md b/docs/architecture/app-design.md index a5babb604..6f1b528a7 100644 --- a/docs/architecture/app-design.md +++ b/docs/architecture/app-design.md @@ -115,7 +115,7 @@ Currently all versions of an agent share a single queue (e.g., `queue-coder-acp- ### Run submission flow -1. User submits via Portal or CLI with: **task**, **criteria** (required), **worker**, **model** (required), and optionally **agentVersion**, a **codebase** selection, and/or a per-gate **`gates`** configuration (see [Gates](#gates--multi-phase-evaluation-pipeline)) +1. User submits via Portal or CLI with: **task**, **criteria** (required), **worker**, **model** (required), and optionally **agentVersion**, a **codebase** selection, record-only **`observations`** criteria, and/or a per-gate **`gates`** configuration (see [Gates](#gates--multi-phase-evaluation-pipeline)) 2. API resolves `agentVersion`: explicit selection → validate active; omitted → latest active by `createdAt` 3. API resolves `model`: explicit → validate against `supportedModels`; omitted → `defaultModel` 4. API looks up `AgentVersion.queueName` and routes message to that queue @@ -146,6 +146,11 @@ Criteria are reusable evaluation rules stored in the database and optionally def - **AI-generated prompts** — natural language behavior descriptions can be converted to evaluation prompts via LLM - **Traits** — reusable labels for filtering and composition (e.g., `has_azure`, `has_node`) - **Gate compatibility** — a `gates: GateId[]` list controls which [gates](#gates--multi-phase-evaluation-pipeline) a criterion may be selected for (empty = all); the list is downward-closed along the DAG +- **Criterion kind** — `kind: "gate" | "observation"` distinguishes criteria that gate the agent from record-only observations. Observation criteria can optionally set `taxonomyElementId` to one of the hard-coded observation dimensions; run detail derives dimension grouping by joining each per-turn `observationResults[].criterionId` back to the criterion metadata. +- **Kind** — `kind: "gate" | "observation"` (default `gate`). **Gate** criteria steer the coding agent through the gate DAG. **Observation** criteria (issue #1156) are record-only: they produce a boolean + evidence per iteration, never feed back to the agent, never pass/fail a gate, and never enter the gate DAG. Dependencies may only reference same-kind criteria. +- **Taxonomy classification** — observation criteria carry an optional `taxonomyElementId` from a hard-coded enum of the three R&A Readout quality dimensions (`dimension:idiomatic-use`, `dimension:dependency-currency`, `dimension:configuration-correctness`). Classification is static/design-time, not per-run. + +Observation evaluation happens off the coding-agent critical path in the `pp-taxonomy` post-processor (see [post-processing](post-processing.md)). A run carries the chosen `observations: string[]` (observation-criteria ids); each evaluated turn stores `observationResults: CriterionResult[]` (same shape as gate `criteriaResults`). Dimension grouping is derived at read time by joining `criterionId` → criterion `taxonomyElementId`. See [`ENV_VARIABLES.md`](../../scope-mt-app/ENV_VARIABLES.md) for related configuration options. diff --git a/docs/architecture/criteria-provider.md b/docs/architecture/criteria-provider.md index de2fc8284..08e623fbb 100644 --- a/docs/architecture/criteria-provider.md +++ b/docs/architecture/criteria-provider.md @@ -160,6 +160,10 @@ bypassed or drift between callers: - **Gate-compatibility invariant** — enforced from both sides: a child's gates must be a subset of every parent's gates, and narrowing a parent may not strand an existing dependent. +- **Same-kind dependency partition** — a criterion's `dependsOn` may only reference + criteria of the same `kind`. This keeps the gate DAG closed under + `resolveWithAncestors` (observations never leak into a gate's evaluation set) and + lets `kind:"observation"` criteria form their own optional dependency chains. - **Referential integrity on delete** — deleting a criterion that still has dependents is rejected. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 9eb7cafbe..ea118f342 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -80,6 +80,7 @@ flowchart TB | `workers/coder-acp-claude-code` | Claude Code agent via Agent Client Protocol (ACP) | | `workers/coder-acp-copilot` | GitHub Copilot agent via Agent Client Protocol (ACP) | | `workers/report-generator` | Copilot SDK worker that generates post-run markdown reports | +| `workers/post-processor` | DAG-based post-run handlers: `pp-atif` (trajectory ATIF) and `pp-taxonomy` (per-iteration boolean observations via the judge) | | `gateway` | AI Gateway — shared Rust TLS-intercepting proxy with plugin architecture (HAR capture, future: token refresh, rate limiting) | ## Data Flow diff --git a/docs/architecture/post-processing.md b/docs/architecture/post-processing.md index 910812ac0..baeed6e44 100644 --- a/docs/architecture/post-processing.md +++ b/docs/architecture/post-processing.md @@ -56,6 +56,24 @@ Two paths trigger post-processing: The `iteration` query parameter is mandatory for the ATIF endpoints. ATIF files are also included in archive exports as `iteration-{N}.atif.trajectory.json`. +## Taxonomy Observations (`pp-taxonomy`) + +The `pp-taxonomy` handler records **observation criteria** — per-iteration boolean +outcomes + evidence about the codebase and the agent's trajectory, **without gating +or steering** the coding agent (issue #1156). It is a DAG node `dependsOn: [pp-atif]`, +because some observations are only derivable from the trajectory (ATIF), not the +final snapshot. + +### Flow + +1. Reads the run's `observations: string[]` (observation-criteria ids chosen at submit). Empty/absent ⇒ no-op, handler `done`. +2. For each iteration with a `snapshotUrl`, calls the existing judge via the shared `JudgeClient` (`POST /api/v1/evaluate`) with `snapshotUrl` + `atifUrl` + the observation-criteria ids — no persona, no gate, no taxonomyElementId (the classifier lives on the criterion). ATIF supersedes per-turn `toolCallsUrl`, so only ATIF is sent. +3. Writes the returned per-criterion `{criterionId, passed, feedback}` to that turn's **`observationResults`** — same shape/storage as gate `criteriaResults`. No new blob, no `run.taxonomyUrl`. +4. Stamps `run.handlerStatus["pp-taxonomy"] = { status: "done", version: N }` and notifies the scheduler. + +Dimension grouping is **derived at read time** by joining each result's `criterionId` → criterion `taxonomyElementId` (the three R&A Readout dimensions; unset = unclassified). It never feeds back to the agent and never affects pass/fail. (#1053 later extends this worker with a richer scorecard artifact.) + + ## Handler Interface The post-processor uses a registry pattern for extensibility: @@ -85,6 +103,7 @@ Handlers are registered at startup in `index.ts`: ```typescript const processor = new PostProcessor(config); processor.registerHandler(new AtifHandler()); +processor.registerHandler(new TaxonomyHandler()); processor.start(); ``` @@ -140,7 +159,7 @@ flowchart LR **Where it runs:** - **Docker Compose:** one `register-handler-` init service per handler (e.g. `pp-atif`) runs the generic script via `tsx`, gated on the scheduler being up. The worker waits for its registration service to complete. -- **Kubernetes:** a `register-handler-post-processor` Job runs the compiled script (`node packages/shared/dist/scripts/register-handler.js`) on each deploy. Only `pp-atif` is registered in K8s today because it is the only post-process handler deployed to the cluster. The scheduler exposes a `Service` (`scheduler.scoped.svc.cluster.local:8080`) so Jobs and workers can reach `/handlers/register` and the notify endpoints. +- **Kubernetes:** a `register-handler-post-processor` Job runs the compiled script (`node packages/shared/dist/scripts/register-handler.js`) on each deploy. `pp-atif` and `pp-taxonomy` are registered in K8s via their respective registration Jobs (`register-handler-taxonomy` points its `HANDLER_YAML_PATH` at `handler.taxonomy.yaml`). The scheduler exposes a `Service` (`scheduler.scoped.svc.cluster.local:8080`) so Jobs and workers can reach `/handlers/register` and the notify endpoints. To **add or change a handler**, edit its `handler.yaml` (bump `version`, adjust `dependsOn`, etc.) — no code or migration changes are needed for registration. diff --git a/packages/db-migrations/src/migrations/021-backfill-criteria-kind.ts b/packages/db-migrations/src/migrations/021-backfill-criteria-kind.ts new file mode 100644 index 000000000..743d9a38d --- /dev/null +++ b/packages/db-migrations/src/migrations/021-backfill-criteria-kind.ts @@ -0,0 +1,47 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/** + * Migration: Backfill criteria kind. + * + * The observation-criteria feature (issue #1156) introduces a `kind` field on + * criteria — `"gate"` (default) gates the coding-agent session via the DAG, + * while `"observation"` is record-only. To preserve existing behaviour every + * legacy criterion must be a gate, so we backfill rows whose `kind` field is + * absent or null with `"gate"`. Mirrors 018-backfill-criteria-gates.ts. + * + * down() reverses only the rows this migration set to exactly "gate", + * unsetting the field to restore the pre-migration state. + */ + +import type { Db } from "mongodb"; +import type { MigrationInterface } from "mongo-migrate-ts"; +import { batchUpdate } from "../batch-update.js"; + +export class BackfillCriteriaKind implements MigrationInterface { + async up(db: Db): Promise { + const criteria = db.collection("criteria"); + + const count = await batchUpdate( + criteria, + { $or: [{ kind: { $exists: false } }, { kind: null }] }, + { $set: { kind: "gate" } }, + "[021] backfill criteria.kind=gate", + ); + + console.log(`[021] Backfilled kind=gate on ${count} criteria`); + } + + async down(db: Db): Promise { + const criteria = db.collection("criteria"); + + const count = await batchUpdate( + criteria, + { kind: "gate" }, + { $unset: { kind: "" } }, + "[021-down] unset criteria.kind", + ); + + console.log(`[021-down] Unset kind on ${count} criteria`); + } +} diff --git a/packages/db-migrations/src/required-migrations.ts b/packages/db-migrations/src/required-migrations.ts index e6d8e82ae..f5742e603 100644 --- a/packages/db-migrations/src/required-migrations.ts +++ b/packages/db-migrations/src/required-migrations.ts @@ -36,4 +36,5 @@ export const REQUIRED_MIGRATIONS: readonly string[] = [ "019-backfill-task-prompt-type.ts", "020-backfill-handler-status.ts", "020-create-codebase-indexes.ts", + "021-backfill-criteria-kind.ts", ]; diff --git a/packages/shared/src/criteria/criteria-provider-api.ts b/packages/shared/src/criteria/criteria-provider-api.ts index f66c6cc70..080dd1c9e 100644 --- a/packages/shared/src/criteria/criteria-provider-api.ts +++ b/packages/shared/src/criteria/criteria-provider-api.ts @@ -203,10 +203,14 @@ export class RestApiCriteriaProvider implements CriteriaProvider { function mapToCriteriaConfig(data: Record): CriteriaConfig { const dependsOn = (data.dependsOn as string[] | undefined) ?? []; const gates = data.gates as CriteriaConfig["gates"] | undefined; + const kind = data.kind as CriteriaConfig["kind"] | undefined; + const taxonomyElementId = data.taxonomyElementId as CriteriaConfig["taxonomyElementId"] | undefined; return { id: String(data.id), prompt: String(data.prompt), dependsOn: dependsOn.length > 0 ? dependsOn : undefined, ...(gates && gates.length > 0 ? { gates } : {}), + ...(kind ? { kind } : {}), + ...(taxonomyElementId ? { taxonomyElementId } : {}), }; } diff --git a/packages/shared/src/criteria/criteria-provider-fs.ts b/packages/shared/src/criteria/criteria-provider-fs.ts index 9922327ea..9c5042700 100644 --- a/packages/shared/src/criteria/criteria-provider-fs.ts +++ b/packages/shared/src/criteria/criteria-provider-fs.ts @@ -68,6 +68,19 @@ export class FileSystemCriteriaProvider implements CriteriaProvider { criteria.gates = gates.map((g: any) => String(g).trim()) as CriteriaConfig["gates"]; } + const kind = data.kind; + if (kind !== undefined) { + if (kind !== "gate" && kind !== "observation") { + throw new Error(`'kind' must be 'gate' or 'observation' in ${file}`); + } + criteria.kind = kind; + } + + const taxonomyElementId = data.taxonomy_element_id || data.taxonomyElementId; + if (taxonomyElementId !== undefined) { + criteria.taxonomyElementId = String(taxonomyElementId).trim() as CriteriaConfig["taxonomyElementId"]; + } + if (this.registry.has(criteria.id)) { throw new Error(`Duplicate criteria id '${criteria.id}' found in ${file}`); } diff --git a/packages/shared/src/criteria/criteria-store.test.ts b/packages/shared/src/criteria/criteria-store.test.ts index ce6c7645e..07908886e 100644 --- a/packages/shared/src/criteria/criteria-store.test.ts +++ b/packages/shared/src/criteria/criteria-store.test.ts @@ -311,3 +311,62 @@ describe("CriteriaStore revives soft-deleted ids on re-create", () => { }); }); + +describe("CriteriaStore kind + taxonomy (issue #1156)", () => { + it("defaults kind to 'gate' on create when absent", async () => { + const store = new CriteriaStore(fakeCollection()); + const created = await store.create({ id: "g1", prompt: "p" }); + expect(created.kind).toBe("gate"); + expect(created.taxonomyElementId).toBeUndefined(); + }); + + it("persists kind:'observation' + taxonomyElementId", async () => { + const store = new CriteriaStore(fakeCollection()); + const created = await store.create({ + id: "obs1", + prompt: "p", + kind: "observation", + taxonomyElementId: "dimension:idiomatic-use", + }); + expect(created.kind).toBe("observation"); + expect(created.taxonomyElementId).toBe("dimension:idiomatic-use"); + }); + + it("rejects taxonomyElementId on a gate criterion", async () => { + const store = new CriteriaStore(fakeCollection()); + await expect( + store.create({ id: "g2", prompt: "p", kind: "gate", taxonomyElementId: "dimension:idiomatic-use" }), + ).rejects.toThrow(CriteriaValidationError); + }); + + it("rejects a gate depending on an observation (same-kind rule)", async () => { + const col = fakeCollection([ + { id: "obs", prompt: "o", dependsOn: [], kind: "observation", createdAt: new Date() }, + ]); + const store = new CriteriaStore(col); + await expect( + store.create({ id: "g3", prompt: "p", kind: "gate", dependsOn: ["obs"] }), + ).rejects.toThrow(CriteriaValidationError); + }); + + it("allows an observation depending on an observation", async () => { + const col = fakeCollection([ + { id: "obs_parent", prompt: "o", dependsOn: [], kind: "observation", createdAt: new Date() }, + ]); + const store = new CriteriaStore(col); + await expect( + store.create({ id: "obs_child", prompt: "c", kind: "observation", dependsOn: ["obs_parent"] }), + ).resolves.toBeTruthy(); + }); + + it("resolveObservations accepts observation ids and rejects gate ids", async () => { + const col = fakeCollection([ + { id: "obs", prompt: "o", dependsOn: [], kind: "observation", createdAt: new Date() }, + { id: "gate", prompt: "g", dependsOn: [], kind: "gate", createdAt: new Date() }, + ]); + const store = new CriteriaStore(col); + const resolved = await store.resolveObservations(["obs"]); + expect(resolved.map((c) => c.id)).toEqual(["obs"]); + await expect(store.resolveObservations(["gate"])).rejects.toThrow(CriteriaValidationError); + }); +}); diff --git a/packages/shared/src/criteria/criteria-store.ts b/packages/shared/src/criteria/criteria-store.ts index 34893c7cb..de51748df 100644 --- a/packages/shared/src/criteria/criteria-store.ts +++ b/packages/shared/src/criteria/criteria-store.ts @@ -2,7 +2,7 @@ // Licensed under the MIT License. import { Collection } from 'mongodb'; -import { CriteriaConfig, CriteriaDocument, GateId } from '../types/types.js'; +import { CriteriaConfig, CriteriaDocument, CriterionKind, GateId, TaxonomyElementId } from '../types/types.js'; import { DependencyGraph } from '../graph/dependency-graph.js'; import { gatesSatisfyInvariant } from '../gates/gates.js'; import { @@ -40,8 +40,11 @@ export class CriteriaStore { prompt: string; dependsOn?: string[]; gates?: GateId[]; + kind?: CriterionKind; + taxonomyElementId?: TaxonomyElementId; }): Promise { - const { id, prompt, dependsOn = [], gates } = input; + const { id, prompt, dependsOn = [], gates, taxonomyElementId } = input; + const kind: CriterionKind = input.kind ?? 'gate'; // Validate ID format if (!/^[a-z][a-z0-9_]*$/.test(id)) { @@ -50,6 +53,13 @@ export class CriteriaStore { ); } + // taxonomyElementId only applies to observation criteria. + if (taxonomyElementId !== undefined && kind !== 'observation') { + throw new CriteriaValidationError( + `taxonomyElementId can only be set on observation criteria (got kind '${kind}')` + ); + } + // Check for duplicates among *active* criteria. const existing = await this.collection.findOne({ id, deletedAt: { $exists: false } }); if (existing) { @@ -72,6 +82,11 @@ export class CriteriaStore { await this.validateNoCycles(id, dependsOn); } + // Validate same-kind dependencies (gate↔gate, observation↔observation). + if (dependsOn.length > 0) { + await this.validateSameKindDependencies(id, dependsOn, kind); + } + // Validate the downward-closed gate-compatibility invariant. await this.validateGateCompatibility(id, dependsOn, gates); @@ -80,6 +95,8 @@ export class CriteriaStore { prompt: prompt.trim(), dependsOn, ...(gates !== undefined && { gates }), + kind, + ...(taxonomyElementId !== undefined && { taxonomyElementId }), createdAt: new Date(), }; @@ -121,7 +138,7 @@ export class CriteriaStore { /** Update a criterion's prompt, dependencies and/or gate compatibility */ async update( id: string, - patch: { prompt?: string; dependsOn?: string[]; gates?: GateId[] } + patch: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId } ): Promise { const existing = await this.get(id); if (!existing) { @@ -147,10 +164,28 @@ export class CriteriaStore { await this.validateDependentsRemainCompatible(id, effectiveGates); } + // Validate same-kind dependencies against the resulting kind + dependsOn. + const effectiveKind: CriterionKind = patch.kind ?? existing.kind ?? 'gate'; + const resultingDependsOn = patch.dependsOn ?? existing.dependsOn ?? []; + if (resultingDependsOn.length > 0) { + await this.validateSameKindDependencies(id, resultingDependsOn, effectiveKind); + } + + // taxonomyElementId only applies to observation criteria. + const effectiveTaxonomy = + patch.taxonomyElementId !== undefined ? patch.taxonomyElementId : existing.taxonomyElementId; + if (effectiveTaxonomy !== undefined && effectiveKind !== 'observation') { + throw new CriteriaValidationError( + `taxonomyElementId can only be set on observation criteria (got kind '${effectiveKind}')` + ); + } + const update: Record = { updatedAt: new Date() }; if (patch.prompt !== undefined) update.prompt = patch.prompt.trim(); if (patch.dependsOn !== undefined) update.dependsOn = patch.dependsOn; if (patch.gates !== undefined) update.gates = patch.gates; + if (patch.kind !== undefined) update.kind = patch.kind; + if (patch.taxonomyElementId !== undefined) update.taxonomyElementId = patch.taxonomyElementId; await this.collection.updateOne( { id, deletedAt: { $exists: false } }, @@ -213,7 +248,14 @@ export class CriteriaStore { ); } - collected.set(id, { id: doc.id, prompt: doc.prompt, dependsOn: doc.dependsOn, ...(doc.gates !== undefined && { gates: doc.gates }) }); + collected.set(id, { + id: doc.id, + prompt: doc.prompt, + dependsOn: doc.dependsOn, + ...(doc.gates !== undefined && { gates: doc.gates }), + ...(doc.kind !== undefined && { kind: doc.kind }), + ...(doc.taxonomyElementId !== undefined && { taxonomyElementId: doc.taxonomyElementId }), + }); if (doc.dependsOn) { for (const parentId of doc.dependsOn) { @@ -240,6 +282,8 @@ export class CriteriaStore { prompt: c.prompt, dependsOn: c.dependsOn, ...(c.gates !== undefined && { gates: c.gates }), + ...(c.kind !== undefined && { kind: c.kind }), + ...(c.taxonomyElementId !== undefined && { taxonomyElementId: c.taxonomyElementId }), })); const edges: { from: string; to: string }[] = []; @@ -268,6 +312,8 @@ export class CriteriaStore { prompt: config.prompt, dependsOn: config.dependsOn || [], ...(config.gates !== undefined && { gates: config.gates }), + kind: config.kind ?? 'gate', + ...(config.taxonomyElementId !== undefined && { taxonomyElementId: config.taxonomyElementId }), createdAt: new Date(), } as any); inserted++; @@ -288,6 +334,52 @@ export class CriteriaStore { } } + /** + * Validate that every dependency shares the criterion's kind. Keeps the gate + * DAG closed under resolveWithAncestors and lets observation criteria form + * their own dependency chains. Missing parents are caught by + * validateDependencies, so they are skipped here. See issue #1156. + */ + private async validateSameKindDependencies( + criterionId: string, + dependsOn: string[], + kind: CriterionKind, + ): Promise { + for (const parentId of dependsOn) { + const parent = await this.get(parentId); + if (!parent) continue; + const parentKind = parent.kind ?? 'gate'; + if (parentKind !== kind) { + throw new CriteriaValidationError( + `'${criterionId}' (kind '${kind}') cannot depend on '${parentId}' (kind '${parentKind}'). ` + + `A criterion may only depend on same-kind criteria.`, + ); + } + } + } + + /** + * Resolve observation-criteria ids, asserting each is an active + * kind:"observation" criterion. Used by submit validation + pp-taxonomy. + * Throws CriteriaValidationError on a missing or non-observation id. + */ + async resolveObservations(ids: string[]): Promise { + const resolved: CriteriaDocument[] = []; + for (const id of ids) { + const doc = await this.get(id); + if (!doc) { + throw new CriteriaValidationError(`Observation criterion '${id}' does not exist`); + } + if ((doc.kind ?? 'gate') !== 'observation') { + throw new CriteriaValidationError( + `Criterion '${id}' is not an observation (kind '${doc.kind ?? 'gate'}')` + ); + } + resolved.push(doc); + } + return resolved; + } + /** Validate that adding edges would not introduce a cycle */ private async validateNoCycles( criterionId: string, diff --git a/packages/shared/src/criteria/index.ts b/packages/shared/src/criteria/index.ts index 4cca92abd..413c36636 100644 --- a/packages/shared/src/criteria/index.ts +++ b/packages/shared/src/criteria/index.ts @@ -7,3 +7,4 @@ export * from "./criteria-provider.js"; export * from "./criteria-provider-fs.js"; export * from "./criteria-provider-api.js"; export * from "./criteria-provider-factory.js"; +export * from "./observation-taxonomy.js"; diff --git a/packages/shared/src/criteria/observation-taxonomy.ts b/packages/shared/src/criteria/observation-taxonomy.ts new file mode 100644 index 000000000..3781e36c6 --- /dev/null +++ b/packages/shared/src/criteria/observation-taxonomy.ts @@ -0,0 +1,43 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import { + TAXONOMY_ELEMENT_IDS, + TaxonomyElementId, + isTaxonomyElementId, +} from "../types/types.js"; + +/** Re-export the hard-coded taxonomy constants from the shared barrel. */ +export { TAXONOMY_ELEMENT_IDS, isTaxonomyElementId }; +export type { TaxonomyElementId }; + +/** Static, human-facing metadata for a taxonomy element. */ +export interface TaxonomyElementMetadata { + id: TaxonomyElementId; + label: string; + description: string; +} + +/** + * Hard-coded metadata for every observation taxonomy element, keyed by id — + * the three quality dimensions from the R&A Readout design. See issue #1156. + */ +export const TAXONOMY_ELEMENT_METADATA: Record = { + "dimension:idiomatic-use": { + id: "dimension:idiomatic-use", + label: "Idiomatic use", + description: + "Did the agent follow the product's recommended patterns and best practices " + + "(quality of how the product is used, not whether it is used at all)?", + }, + "dimension:dependency-currency": { + id: "dimension:dependency-currency", + label: "Dependency currency", + description: "Did the agent use current versions, packages, and namespaces?", + }, + "dimension:configuration-correctness": { + id: "dimension:configuration-correctness", + label: "Configuration correctness", + description: "Are auth, connection, environment, and deployment settings configured properly?", + }, +}; diff --git a/packages/shared/src/judge/judge-client.ts b/packages/shared/src/judge/judge-client.ts index f94e3c856..c07cfa82f 100644 --- a/packages/shared/src/judge/judge-client.ts +++ b/packages/shared/src/judge/judge-client.ts @@ -17,6 +17,10 @@ export interface JudgeEvaluateRequest { gate?: GateId; /** Blob URL of this iteration's captured tool calls/outputs (build/test/run output). */ toolCallsUrl?: string; + /** Blob URL of this iteration's ATIF trajectory JSON. Normalized superset of + * tool calls; sent by pp-taxonomy in place of toolCallsUrl so observations + * can be evaluated against the full agent trajectory. See issue #1156. */ + atifUrl?: string; } /** diff --git a/packages/shared/src/schemas/criteria.ts b/packages/shared/src/schemas/criteria.ts index 29b50ddc3..efe0761c1 100644 --- a/packages/shared/src/schemas/criteria.ts +++ b/packages/shared/src/schemas/criteria.ts @@ -3,11 +3,13 @@ import { z } from "zod"; import { extendZodWithOpenApi } from "@asteasolutions/zod-to-openapi"; -import { GATES } from "../types/types.js"; +import { GATES, CRITERION_KINDS, TAXONOMY_ELEMENT_IDS } from "../types/types.js"; extendZodWithOpenApi(z); export const GateIdSchema = z.enum(GATES); +export const CriterionKindSchema = z.enum(CRITERION_KINDS); +export const TaxonomyElementIdSchema = z.enum(TAXONOMY_ELEMENT_IDS); export const CreateCriteriaInputSchema = z .object({ @@ -15,6 +17,8 @@ export const CreateCriteriaInputSchema = z prompt: z.string(), dependsOn: z.array(z.string()).optional(), gates: z.array(GateIdSchema).optional(), + kind: CriterionKindSchema.default("gate"), + taxonomyElementId: TaxonomyElementIdSchema.optional(), }) .openapi("CreateCriteriaInput"); @@ -23,6 +27,8 @@ export const UpdateCriteriaInputSchema = z prompt: z.string().optional(), dependsOn: z.array(z.string()).optional(), gates: z.array(GateIdSchema).optional(), + kind: CriterionKindSchema.optional(), + taxonomyElementId: TaxonomyElementIdSchema.optional(), }) .openapi("UpdateCriteriaInput"); @@ -32,6 +38,8 @@ export const CriteriaResponseSchema = z prompt: z.string(), dependsOn: z.array(z.string()).optional(), gates: z.array(GateIdSchema).optional(), + kind: CriterionKindSchema.optional(), + taxonomyElementId: TaxonomyElementIdSchema.optional(), createdAt: z.coerce.date(), updatedAt: z.coerce.date().optional(), deletedAt: z.coerce.date().optional(), @@ -44,6 +52,8 @@ export const CriteriaGraphNodeSchema = z prompt: z.string(), dependsOn: z.array(z.string()).optional(), gates: z.array(GateIdSchema).optional(), + kind: CriterionKindSchema.optional(), + taxonomyElementId: TaxonomyElementIdSchema.optional(), }) .openapi("CriteriaGraphNode"); diff --git a/packages/shared/src/schemas/request.ts b/packages/shared/src/schemas/request.ts index 513e54996..1abc04062 100644 --- a/packages/shared/src/schemas/request.ts +++ b/packages/shared/src/schemas/request.ts @@ -70,6 +70,7 @@ export const ConversationTurnSchema = z passed: z.boolean(), timestamp: z.coerce.date(), criteriaResults: z.array(CriterionResultSchema).optional(), + observationResults: z.array(CriterionResultSchema).optional(), harUrl: z.string().optional(), videoUrls: z.array(z.string()).optional(), tokenUsage: TokenUsageSchema.optional(), @@ -130,6 +131,7 @@ export const CreateRequestInputSchema = z profileVariations: z.array(z.string()).optional(), priority: z.number().int().optional(), gates: z.array(GateConfigSchema).optional(), + observations: z.array(z.string()).optional(), }) .openapi("CreateRequestInput"); @@ -157,6 +159,7 @@ export const RequestResponseSchema = z submissionId: z.string().optional(), priority: z.number().int().default(0), gates: z.array(GateConfigSchema).optional(), + observations: z.array(z.string()).optional(), gateSummaries: z.array(GateRunSummarySchema).optional(), // Per-attempt state lives in the run sub-document. run: z diff --git a/packages/shared/src/schemas/schemas.test.ts b/packages/shared/src/schemas/schemas.test.ts index c8e8adceb..0878e3cf3 100644 --- a/packages/shared/src/schemas/schemas.test.ts +++ b/packages/shared/src/schemas/schemas.test.ts @@ -515,6 +515,33 @@ describe("criteria schemas", () => { it("rejects missing prompt", () => { expect(() => CreateCriteriaInputSchema.parse({ id: "abc" })).toThrow(); }); + + it("defaults kind to 'gate' when absent", () => { + const result = CreateCriteriaInputSchema.parse({ id: "abc", prompt: "p" }); + expect(result.kind).toBe("gate"); + }); + + it("accepts kind:'observation' with a valid taxonomyElementId", () => { + const result = CreateCriteriaInputSchema.parse({ + id: "obs", + prompt: "p", + kind: "observation", + taxonomyElementId: "dimension:idiomatic-use", + }); + expect(result.kind).toBe("observation"); + expect(result.taxonomyElementId).toBe("dimension:idiomatic-use"); + }); + + it("rejects an unknown taxonomyElementId", () => { + expect(() => + CreateCriteriaInputSchema.parse({ + id: "obs", + prompt: "p", + kind: "observation", + taxonomyElementId: "dimension:nope", + }), + ).toThrow(); + }); }); describe("CriteriaResponseSchema", () => { diff --git a/packages/shared/src/storage/blob-storage.ts b/packages/shared/src/storage/blob-storage.ts index 19a7e3db2..99e0bab46 100644 --- a/packages/shared/src/storage/blob-storage.ts +++ b/packages/shared/src/storage/blob-storage.ts @@ -485,4 +485,27 @@ export class BlobStorage { const blockBlobClient = this.containerClient.getBlockBlobClient(blobName); return blockBlobClient.downloadToBuffer(); } + + /** + * Downloads and parses a JSON blob. Accepts either a full blob URL (Azure or + * Azurite) or a bare blob name within the snapshots container. Used by the + * judge to load ATIF trajectory JSON for observation evaluation. Issue #1156. + */ + async downloadJson(urlOrBlobName: string): Promise { + await this.ensureContainer(); + let blobName = urlOrBlobName; + if (urlOrBlobName.startsWith("http://") || urlOrBlobName.startsWith("https://")) { + const url = new URL(urlOrBlobName); + const containerPrefix = `/${SNAPSHOTS_CONTAINER}/`; + const containerIndex = url.pathname.indexOf(containerPrefix); + if (containerIndex === -1) { + throw new Error( + `Blob URL does not contain container '${SNAPSHOTS_CONTAINER}': ${urlOrBlobName}`, + ); + } + blobName = decodeURIComponent(url.pathname.substring(containerIndex + containerPrefix.length)); + } + const buf = await this.downloadBlobToBuffer(blobName); + return JSON.parse(buf.toString("utf-8")) as T; + } } diff --git a/packages/shared/src/types/types.ts b/packages/shared/src/types/types.ts index f233c21cb..34e708e16 100644 --- a/packages/shared/src/types/types.ts +++ b/packages/shared/src/types/types.ts @@ -62,6 +62,42 @@ export function isGateId(value: unknown): value is GateId { return typeof value === "string" && (GATES as readonly string[]).includes(value); } +/** + * Discriminator for what a criterion is used for. `"gate"` (default) gates the + * coding-agent session via the DAG; `"observation"` is record-only — it produces + * a per-iteration boolean + evidence and never feeds back, never pass/fails a + * gate, and never enters the gate DAG. See issue #1156. + */ +export type CriterionKind = "gate" | "observation"; + +/** The hard-coded set of criterion kinds. */ +export const CRITERION_KINDS = ["gate", "observation"] as const; + +/** True when `value` is a valid criterion kind. */ +export function isCriterionKind(value: unknown): value is CriterionKind { + return typeof value === "string" && (CRITERION_KINDS as readonly string[]).includes(value); +} + +/** + * Hard-coded taxonomy elements for observation criteria — the three quality + * dimensions from the R&A Readout design. Gates (select/build/test/run/deploy) + * are excluded as they live in the criterion `gates` field. Namespaced ids keep + * a future first-class taxonomy a clean migration. See issue #1156. + */ +export const TAXONOMY_ELEMENT_IDS = [ + "dimension:idiomatic-use", + "dimension:dependency-currency", + "dimension:configuration-correctness", +] as const; + +/** A single taxonomy element id. */ +export type TaxonomyElementId = (typeof TAXONOMY_ELEMENT_IDS)[number]; + +/** True when `value` is a valid taxonomy element id. */ +export function isTaxonomyElementId(value: unknown): value is TaxonomyElementId { + return typeof value === "string" && (TAXONOMY_ELEMENT_IDS as readonly string[]).includes(value); +} + /** * A prompt's type discriminator — one literal per gate. The prompt that drives * a gate must have a `type` equal to that gate's id. @@ -132,6 +168,10 @@ export interface ConversationTurn { passed: boolean; timestamp: Date; criteriaResults?: CriterionResult[]; // Per-criterion breakdown from DAG evaluation + /** Per-iteration observation results (kind:"observation" criteria), recorded + * by the pp-taxonomy post-processor. Same shape as criteriaResults but never + * gates the agent. Empty/absent when no observations were selected. #1156. */ + observationResults?: CriterionResult[]; harUrl?: string; // Blob storage URL to the HAR file for this turn videoUrls?: string[]; // Blob storage URLs to session recording videos for this turn tokenUsage?: TokenUsage; // LLM token usage for this iteration @@ -284,6 +324,13 @@ export interface RequestDocument { * docs/design/gates.md §4.3). Gates run in `GATE_ORDER`. */ gates?: GateConfig[]; + /** + * Observation-criteria ids (kind:"observation") selected at submit. The + * pp-taxonomy post-processor evaluates exactly this set after the run + * completes and writes per-turn `observationResults`. Empty/absent = none. + * See issue #1156. + */ + observations?: string[]; /** * Per-gate execution summary, populated as the run progresses. Derived * from `run.turns` but stored explicitly for cheap querying. @@ -597,6 +644,13 @@ export interface CriteriaConfig { * (applies only to NEW criteria; legacy rows are backfilled to ["select"] * by migration 018). See docs/design/gates.md §4.2. */ gates?: GateId[]; + /** Whether this criterion gates the agent or is a record-only observation. + * Default "gate" applied at store/schema layer; legacy rows backfilled to + * "gate" by migration. See issue #1156. */ + kind?: CriterionKind; + /** Design-time classification under a hard-coded taxonomy element. Only set + * on observation criteria. Static — never stored per run. See issue #1156. */ + taxonomyElementId?: TaxonomyElementId; } // Per-criterion result from judge evaluation From 19b02998acc79b868ac7042c69ddcd68bd33609b Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Sun, 28 Jun 2026 19:16:50 -0700 Subject: [PATCH 02/17] feat(portal): observation criteria kind/taxonomy UI, submit picker, observations tab Carry criterion kind + taxonomyElementId, request observations, and turn.observationResults through the portal: kind selector + taxonomy picker on criteria pages, observations multi-select on Submit Run, and a Run-detail Observations tab grouped by dimension. Includes Storybook stories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/components/CreateCriterionDialog.tsx | 6 +- .../src/components/CriteriaBadge.stories.tsx | 49 +++++++ apps/portal/src/components/CriteriaBadge.tsx | 48 +++++-- .../src/components/CriteriaFilterBar.tsx | 29 ++++ apps/portal/src/components/CriteriaPicker.tsx | 11 +- .../src/components/CriteriaWizardStep1.tsx | 43 +++++- .../src/components/CriteriaWizardStep2.tsx | 39 +++++- .../ObservationTaxonomySelect.stories.tsx | 31 ++++ .../components/ObservationTaxonomySelect.tsx | 48 +++++++ apps/portal/src/hooks/useCriteriaWizard.ts | 50 +++++-- apps/portal/src/lib/api.ts | 7 +- apps/portal/src/pages/CriteriaList.tsx | 64 ++++++++- apps/portal/src/pages/CriterionDetail.tsx | 70 +++++++++- apps/portal/src/pages/RunDetail.tsx | 132 +++++++++++++++++- apps/portal/src/pages/SubmitRun.tsx | 47 +++++++ apps/portal/src/types.ts | 40 +++++- 16 files changed, 659 insertions(+), 55 deletions(-) create mode 100644 apps/portal/src/components/CriteriaBadge.stories.tsx create mode 100644 apps/portal/src/components/ObservationTaxonomySelect.stories.tsx create mode 100644 apps/portal/src/components/ObservationTaxonomySelect.tsx diff --git a/apps/portal/src/components/CreateCriterionDialog.tsx b/apps/portal/src/components/CreateCriterionDialog.tsx index 188009aac..be51969f3 100644 --- a/apps/portal/src/components/CreateCriterionDialog.tsx +++ b/apps/portal/src/components/CreateCriterionDialog.tsx @@ -15,6 +15,7 @@ import { } from "@/components/ui/dialog"; import { ArrowLeft, Loader2, Sparkles } from "lucide-react"; import type { GateId } from "@/lib/gates"; +import type { CriterionKind } from "@/types"; interface CreateCriterionDialogProps { open: boolean; @@ -25,12 +26,15 @@ interface CreateCriterionDialogProps { defaultGates?: GateId[]; /** Gates that cannot be unselected (e.g. the gate this criterion is created for) */ lockedGates?: GateId[]; + /** Pre-populated criteria kind (defaults to gate) */ + defaultKind?: CriterionKind; } -export function CreateCriterionDialog({ open, onOpenChange, onCreated, defaultGates, lockedGates }: CreateCriterionDialogProps) { +export function CreateCriterionDialog({ open, onOpenChange, onCreated, defaultGates, lockedGates, defaultKind }: CreateCriterionDialogProps) { const wizard = useCriteriaWizard({ initialGates: defaultGates, lockedGates, + initialKind: defaultKind, onSuccess: (id) => { onCreated(id); handleOpenChange(false); diff --git a/apps/portal/src/components/CriteriaBadge.stories.tsx b/apps/portal/src/components/CriteriaBadge.stories.tsx new file mode 100644 index 000000000..e851b3aec --- /dev/null +++ b/apps/portal/src/components/CriteriaBadge.stories.tsx @@ -0,0 +1,49 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { CriteriaBadge, CriteriaKindBadge } from "./CriteriaBadge"; + +const meta = { + title: "Components/CriteriaBadge", + component: CriteriaBadge, + tags: ["ai-generated", "needs-work"], +} satisfies Meta; + +export default meta; +type Story = StoryObj; + +export const GateCriterion: Story = { + args: { + criterionId: "has_unit_tests", + kind: "gate", + evaluated: true, + result: true, + showStateLabel: true, + link: false, + }, +}; + +export const ObservationCriterion: Story = { + args: { + criterionId: "uses_current_sdk", + kind: "observation", + evaluated: true, + result: false, + showStateLabel: true, + link: false, + }, +}; + +export const KindBadges: Story = { + args: { + criterionId: "has_unit_tests", + link: false, + }, + render: () => ( +
+ + +
+ ), +}; diff --git a/apps/portal/src/components/CriteriaBadge.tsx b/apps/portal/src/components/CriteriaBadge.tsx index f05b00b77..d1cc1bbc3 100644 --- a/apps/portal/src/components/CriteriaBadge.tsx +++ b/apps/portal/src/components/CriteriaBadge.tsx @@ -5,6 +5,7 @@ import { Link } from "react-router-dom"; import { Badge } from "@/components/ui/badge"; import { cn } from "@/lib/utils"; import { criterionResultStyle, type CriterionResultState } from "@/lib/criteria-result"; +import type { CriterionKind } from "@/types"; interface CriteriaBadgeProps { criterionId: string; @@ -13,6 +14,7 @@ interface CriteriaBadgeProps { link?: boolean; showStateLabel?: boolean; className?: string; + kind?: CriterionKind; } export function CriteriaBadge({ @@ -22,22 +24,30 @@ export function CriteriaBadge({ link = true, showStateLabel = false, className, + kind, }: CriteriaBadgeProps) { + const kindLabel = kind === "observation" ? "Obs" : kind === "gate" ? "Gate" : undefined; const content = !evaluated ? ( - - {criterionId} - + + + {criterionId} + + {kindLabel && } + ) : (() => { const { colorClass, Icon, label, title } = criterionResultStyle(result); return ( - - - {showStateLabel ? `${criterionId}: ${label}` : criterionId} - + + + + {showStateLabel ? `${criterionId}: ${label}` : criterionId} + + {kindLabel && } + ); })(); @@ -49,3 +59,19 @@ export function CriteriaBadge({ ); } + +export function CriteriaKindBadge({ kind, className }: { kind?: CriterionKind; className?: string }) { + const normalized = kind ?? "gate"; + return ( + + {normalized === "observation" ? "Observation" : "Gate"} + + ); +} diff --git a/apps/portal/src/components/CriteriaFilterBar.tsx b/apps/portal/src/components/CriteriaFilterBar.tsx index 3c1dff4f0..ec34abfad 100644 --- a/apps/portal/src/components/CriteriaFilterBar.tsx +++ b/apps/portal/src/components/CriteriaFilterBar.tsx @@ -4,6 +4,7 @@ import { Filter, X } from "lucide-react"; import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"; import { Badge } from "@/components/ui/badge"; +import type { CriterionKind } from "@/types"; interface CriteriaFilterBarProps { availableCriteria: string[]; @@ -16,6 +17,10 @@ interface CriteriaFilterBarProps { emptyDescription?: string; /** Override the description template when criteria are selected (receives count) */ selectedDescription?: (count: number) => string; + /** Optional kind metadata for rendering gate/observation filter chips. */ + criteriaKinds?: Record; + selectedKinds?: CriterionKind[]; + onToggleKind?: (kind: CriterionKind) => void; } export function CriteriaFilterBar({ @@ -27,12 +32,22 @@ export function CriteriaFilterBar({ emptyDescription = "Click criteria to filter runs and redefine success. Default: all criteria in each run must pass.", selectedDescription = (count: number) => `Success = all ${count} selected criteria pass. Runs without these criteria are excluded.`, + criteriaKinds, + selectedKinds = [], + onToggleKind, }: CriteriaFilterBarProps) { if (availableCriteria.length === 0) { return null; } const selectedSet = new Set(selectedCriteria); + const selectedKindSet = new Set(selectedKinds); + const kindCounts = criteriaKinds + ? { + gate: availableCriteria.filter((id) => (criteriaKinds[id] ?? "gate") === "gate").length, + observation: availableCriteria.filter((id) => criteriaKinds[id] === "observation").length, + } + : undefined; return ( @@ -58,6 +73,20 @@ export function CriteriaFilterBar({ + {kindCounts && onToggleKind && ( +
+ {(["gate", "observation"] as const).map((kind) => ( + onToggleKind(kind)} + > + {kind} ({kindCounts[kind]}) + + ))} +
+ )}
{availableCriteria.map((id) => { const isSelected = selectedSet.has(id); diff --git a/apps/portal/src/components/CriteriaPicker.tsx b/apps/portal/src/components/CriteriaPicker.tsx index 05e04a454..19ed40767 100644 --- a/apps/portal/src/components/CriteriaPicker.tsx +++ b/apps/portal/src/components/CriteriaPicker.tsx @@ -10,7 +10,7 @@ import { Input } from "@/components/ui/input"; import { Skeleton } from "@/components/ui/skeleton"; import { Search, X, Sparkles } from "lucide-react"; import { GATE_METADATA, isCriterionCompatibleWithGate, type GateId } from "@/lib/gates"; -import type { CriteriaDocument } from "@/types"; +import type { CriteriaDocument, CriterionKind } from "@/types"; interface CriteriaPickerProps { selected: string[]; @@ -23,6 +23,8 @@ interface CriteriaPickerProps { trailingAction?: ReactNode; /** Restrict suggestions to criteria compatible with this gate */ gate?: GateId; + /** Restrict suggestions by criterion kind. Missing kind is treated as "gate". */ + kind?: CriterionKind; /** * Additional predicate restricting which criteria are offered as suggestions. * Used to enforce gate-compatibility invariants (e.g. only show criteria that @@ -31,7 +33,7 @@ interface CriteriaPickerProps { filter?: (criterion: CriteriaDocument) => boolean; } -export function CriteriaPicker({ selected, onChange, aiSuggested = [], inputId, trailingAction, gate, filter }: CriteriaPickerProps) { +export function CriteriaPicker({ selected, onChange, aiSuggested = [], inputId, trailingAction, gate, kind, filter }: CriteriaPickerProps) { const [query, setQuery] = useState(""); const [open, setOpen] = useState(false); const [highlightIdx, setHighlightIdx] = useState(0); @@ -49,6 +51,7 @@ export function CriteriaPicker({ selected, onChange, aiSuggested = [], inputId, const suggestions = useMemo(() => { const available = criteria.filter((c) => !selected.includes(c.id) && + (!kind || (c.kind ?? "gate") === kind) && (!gate || isCriterionCompatibleWithGate(c.gates, gate)) && (!filter || filter(c)) ); @@ -57,7 +60,7 @@ export function CriteriaPicker({ selected, onChange, aiSuggested = [], inputId, return available.filter( (c) => c.id.toLowerCase().includes(q) || c.prompt.toLowerCase().includes(q), ); - }, [criteria, selected, query, gate, filter]); + }, [criteria, selected, query, gate, kind, filter]); // Reset highlight when suggestions change useEffect(() => { @@ -212,7 +215,7 @@ export function CriteriaPicker({ selected, onChange, aiSuggested = [], inputId, }} onFocus={() => setOpen(true)} onKeyDown={handleKeyDown} - placeholder={gate ? `Search ${GATE_METADATA[gate].label} criteria…` : "Type to search criteria…"} + placeholder={kind === "observation" ? "Search observation criteria…" : gate ? `Search ${GATE_METADATA[gate].label} criteria…` : "Type to search criteria…"} className="h-9 font-mono text-sm" /> {trailingAction} diff --git a/apps/portal/src/components/CriteriaWizardStep1.tsx b/apps/portal/src/components/CriteriaWizardStep1.tsx index 252ca52ae..c6a182b36 100644 --- a/apps/portal/src/components/CriteriaWizardStep1.tsx +++ b/apps/portal/src/components/CriteriaWizardStep1.tsx @@ -4,7 +4,9 @@ import { Input } from "@/components/ui/input"; import { Label } from "@/components/ui/label"; import { Textarea } from "@/components/ui/textarea"; +import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"; import { GateCompatibilityPicker } from "@/components/GateCompatibilityPicker"; +import { ObservationTaxonomySelect } from "@/components/ObservationTaxonomySelect"; import type { CriteriaWizardState } from "@/hooks/useCriteriaWizard"; interface CriteriaWizardStep1Props { @@ -26,6 +28,10 @@ export function CriteriaWizardStep1({ wizard, idPrefix = "" }: CriteriaWizardSte gates, setGates, lockedGates, + kind, + setKind, + taxonomyElementId, + setTaxonomyElementId, } = wizard; return ( @@ -89,15 +95,42 @@ export function CriteriaWizardStep1({ wizard, idPrefix = "" }: CriteriaWizardSte

- {/* Gate compatibility */} + {/* Criteria kind */}
- - + +

- Select the gates this criterion applies to. Parents must be compatible with every - selected gate. This drives the gate-aware parent/child suggestions on the next step. + Gate criteria drive judge feedback. Observations are recorded after each iteration and never gate the agent.

+ + {/* Gate compatibility */} + {kind === "gate" ? ( +
+ + +

+ Select the gates this criterion applies to. Parents must be compatible with every + selected gate. This drives the gate-aware parent/child suggestions on the next step. +

+
+ ) : ( + + )} ); } diff --git a/apps/portal/src/components/CriteriaWizardStep2.tsx b/apps/portal/src/components/CriteriaWizardStep2.tsx index 9d56df64a..2e4e52cb4 100644 --- a/apps/portal/src/components/CriteriaWizardStep2.tsx +++ b/apps/portal/src/components/CriteriaWizardStep2.tsx @@ -8,6 +8,7 @@ import { Badge } from "@/components/ui/badge"; import { Separator } from "@/components/ui/separator"; import { CriteriaPicker } from "@/components/CriteriaPicker"; import { gatesSatisfyInvariant, GATE_METADATA, orderGateIds } from "@/lib/gates"; +import { TAXONOMY_ELEMENT_METADATA } from "@/types"; import { Loader2, Check, RefreshCw } from "lucide-react"; import type { CriteriaWizardState } from "@/hooks/useCriteriaWizard"; @@ -22,6 +23,8 @@ export function CriteriaWizardStep2({ wizard }: CriteriaWizardStep2Props) { dependsOn, setDependsOn, gates, + kind = "gate", + taxonomyElementId, prompt, setPrompt, aiGenerated, @@ -55,6 +58,15 @@ export function CriteriaWizardStep2({ wizard }: CriteriaWizardStep2Props) { {id} +
+

+ Kind +

+ + {kind === "observation" ? "Observation" : "Gate"} + +
+ {kind === "gate" ? (

Gate compatibility @@ -71,6 +83,16 @@ export function CriteriaWizardStep2({ wizard }: CriteriaWizardStep2Props) { )}

+ ) : ( +
+

+ Observation dimension +

+ + {taxonomyElementId ? TAXONOMY_ELEMENT_METADATA[taxonomyElementId].label : "Unclassified"} + +
+ )} @@ -84,11 +106,14 @@ export function CriteriaWizardStep2({ wizard }: CriteriaWizardStep2Props) { selected={dependsOn} onChange={setDependsOn} aiSuggested={suggestedParents} - filter={(c) => gatesSatisfyInvariant(c.gates, gates) && !acceptedChildren.includes(c.id)} + filter={(c) => + (c.kind ?? "gate") === kind && + (kind === "observation" || gatesSatisfyInvariant(c.gates, gates)) && + !acceptedChildren.includes(c.id) + } />

- Criteria that must pass before this one is evaluated. Only criteria compatible with - every selected gate are shown. List only direct parents — + Criteria that must pass before this one is evaluated. {kind === "gate" ? "Only criteria compatible with every selected gate are shown." : "Only observation criteria are shown."} List only direct parents — the judge automatically evaluates all transitive ancestors in topological order, so you don't need to repeat a parent's own dependencies here.

@@ -105,11 +130,15 @@ export function CriteriaWizardStep2({ wizard }: CriteriaWizardStep2Props) { selected={acceptedChildren} onChange={setAcceptedChildren} aiSuggested={suggestedChildren} - filter={(c) => gatesSatisfyInvariant(gates, c.gates) && !dependsOn.includes(c.id)} + filter={(c) => + (c.kind ?? "gate") === kind && + (kind === "observation" || gatesSatisfyInvariant(gates, c.gates)) && + !dependsOn.includes(c.id) + } />

These criteria will be updated to depend on {id || "this criterion"} after creation. - Only criteria compatible with a subset of the selected gates are shown. + {kind === "gate" ? " Only criteria compatible with a subset of the selected gates are shown." : " Only observation criteria are shown."}

diff --git a/apps/portal/src/components/ObservationTaxonomySelect.stories.tsx b/apps/portal/src/components/ObservationTaxonomySelect.stories.tsx new file mode 100644 index 000000000..aab2ab344 --- /dev/null +++ b/apps/portal/src/components/ObservationTaxonomySelect.stories.tsx @@ -0,0 +1,31 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import { useState } from "react"; +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { ObservationTaxonomySelect } from "./ObservationTaxonomySelect"; +import type { TaxonomyElementId } from "@/types"; + +const meta = { + title: "Components/ObservationTaxonomySelect", + component: ObservationTaxonomySelect, + tags: ["ai-generated", "needs-work"], +} satisfies Meta; + +export default meta; +type Story = StoryObj; + +export const Default: Story = { + args: { + value: "dimension:idiomatic-use", + onChange: () => undefined, + }, + render: () => { + const [value, setValue] = useState("dimension:idiomatic-use"); + return ( +
+ +
+ ); + }, +}; diff --git a/apps/portal/src/components/ObservationTaxonomySelect.tsx b/apps/portal/src/components/ObservationTaxonomySelect.tsx new file mode 100644 index 000000000..1bdfabd7e --- /dev/null +++ b/apps/portal/src/components/ObservationTaxonomySelect.tsx @@ -0,0 +1,48 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import { Label } from "@/components/ui/label"; +import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"; +import { TAXONOMY_ELEMENT_IDS, TAXONOMY_ELEMENT_METADATA, type TaxonomyElementId } from "@/types"; + +interface ObservationTaxonomySelectProps { + value?: TaxonomyElementId; + onChange: (value: TaxonomyElementId | undefined) => void; + id?: string; + label?: string; +} + +export function ObservationTaxonomySelect({ + value, + onChange, + id, + label = "Observation dimension", +}: ObservationTaxonomySelectProps) { + return ( +
+ + +

+ {value ? TAXONOMY_ELEMENT_METADATA[value].description : "Leave unset to group results as unclassified."} +

+
+ ); +} diff --git a/apps/portal/src/hooks/useCriteriaWizard.ts b/apps/portal/src/hooks/useCriteriaWizard.ts index 88575feb6..3033c2d49 100644 --- a/apps/portal/src/hooks/useCriteriaWizard.ts +++ b/apps/portal/src/hooks/useCriteriaWizard.ts @@ -7,6 +7,7 @@ import { api } from "@/lib/api"; import { slugify } from "@/lib/utils"; import { toast } from "sonner"; import { gatesSatisfyInvariant, type GateId } from "@/lib/gates"; +import type { CriterionKind, TaxonomyElementId } from "@/types"; export interface UseCriteriaWizardOptions { /** Pre-populated parent dependency IDs */ @@ -15,11 +16,20 @@ export interface UseCriteriaWizardOptions { initialGates?: GateId[]; /** Gates that cannot be unselected in the gate picker (e.g. inline creation) */ lockedGates?: GateId[]; + initialKind?: CriterionKind; + initialTaxonomyElementId?: TaxonomyElementId; /** Called with the new criterion ID after successful creation */ onSuccess: (id: string) => void; } -export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedGates, onSuccess }: UseCriteriaWizardOptions) { +export function useCriteriaWizard({ + initialDependsOn = [], + initialGates, + lockedGates, + initialKind = "gate", + initialTaxonomyElementId, + onSuccess, +}: UseCriteriaWizardOptions) { const queryClient = useQueryClient(); // Wizard step (1 or 2) @@ -31,6 +41,8 @@ export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedG const [idManuallyEdited, setIdManuallyEdited] = useState(false); const [dependsOn, setDependsOn] = useState(initialDependsOn); const [gates, setGates] = useState(initialGates ?? ["select"]); + const [kind, setKind] = useState(initialKind); + const [taxonomyElementId, setTaxonomyElementId] = useState(initialTaxonomyElementId); // Step 2 fields const [prompt, setPrompt] = useState(""); @@ -53,7 +65,13 @@ export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedG () => existingCriteria.some((c) => c.id === id.trim()), [id, existingCriteria], ); - const canContinue = behavior.trim().length > 0 && id.trim().length > 0 && idValid && !idExists && !criteriaLoading; + const canContinue = + behavior.trim().length > 0 && + id.trim().length > 0 && + idValid && + !idExists && + !criteriaLoading && + (kind === "observation" || !!gates?.length); // Gate compatibility lookup for parent/child suggestion filtering. const criterionGatesById = useMemo(() => { @@ -71,17 +89,17 @@ export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedG useEffect(() => { setDependsOn((prev) => { const next = prev.filter( - (pid) => !criterionGatesById.has(pid) || gatesSatisfyInvariant(criterionGatesById.get(pid), gates), + (pid) => kind === "observation" || !criterionGatesById.has(pid) || gatesSatisfyInvariant(criterionGatesById.get(pid), gates), ); return next.length === prev.length ? prev : next; }); setAcceptedChildren((prev) => { const next = prev.filter( - (cid) => !criterionGatesById.has(cid) || gatesSatisfyInvariant(gates, criterionGatesById.get(cid)), + (cid) => kind === "observation" || !criterionGatesById.has(cid) || gatesSatisfyInvariant(gates, criterionGatesById.get(cid)), ); return next.length === prev.length ? prev : next; }); - }, [gates, criterionGatesById, suggestedParents, suggestedChildren]); + }, [gates, kind, criterionGatesById, suggestedParents, suggestedChildren]); // Auto-suggest ID from behavior (unless manually edited) const handleBehaviorChange = useCallback( @@ -166,8 +184,8 @@ export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedG // Step 1 → Step 2 const handleContinue = useCallback(() => { setStep(2); - generateMutation.mutate({ behavior: behavior.trim(), gates }); - }, [behavior, gates, generateMutation]); + generateMutation.mutate({ behavior: behavior.trim(), gates: kind === "gate" ? gates : undefined }); + }, [behavior, gates, kind, generateMutation]); // Step 2 → Submit const handleCreate = useCallback(() => { @@ -176,17 +194,19 @@ export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedG id: id.trim(), prompt: prompt.trim(), dependsOn: dependsOn.length > 0 ? dependsOn : undefined, - gates, + ...(kind === "gate" ? { gates } : {}), + kind, + ...(kind === "observation" && taxonomyElementId ? { taxonomyElementId } : {}), }); - }, [id, prompt, dependsOn, gates, createMutation]); + }, [id, prompt, dependsOn, gates, kind, taxonomyElementId, createMutation]); // Regenerate prompt const handleRegenerate = useCallback(() => { setSuggestedParents([]); setSuggestedChildren([]); setAcceptedChildren([]); - generateMutation.mutate({ behavior: behavior.trim(), gates }); - }, [behavior, gates, generateMutation]); + generateMutation.mutate({ behavior: behavior.trim(), gates: kind === "gate" ? gates : undefined }); + }, [behavior, gates, kind, generateMutation]); // Reset all state to initial values const reset = useCallback(() => { @@ -196,12 +216,14 @@ export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedG setIdManuallyEdited(false); setDependsOn(initialDependsOn); setGates(initialGates ?? ["select"]); + setKind(initialKind); + setTaxonomyElementId(initialTaxonomyElementId); setPrompt(""); setAiGenerated(false); setSuggestedParents([]); setSuggestedChildren([]); setAcceptedChildren([]); - }, [initialDependsOn, initialGates]); + }, [initialDependsOn, initialGates, initialKind, initialTaxonomyElementId]); return { // State @@ -217,6 +239,10 @@ export function useCriteriaWizard({ initialDependsOn = [], initialGates, lockedG gates, setGates, lockedGates, + kind, + setKind, + taxonomyElementId, + setTaxonomyElementId, prompt, setPrompt, aiGenerated, diff --git a/apps/portal/src/lib/api.ts b/apps/portal/src/lib/api.ts index c3e275f09..281fad7dd 100644 --- a/apps/portal/src/lib/api.ts +++ b/apps/portal/src/lib/api.ts @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. -import type { Run, RunState, CriteriaDocument, CriteriaGraphData, GeneratePromptResponse, AnalysisResponse, PromptFeatureDocument, Report, BulkReportStatus, BulkReportSummary, ReportTemplate, ReportTrigger, ReportTemplateSystemPrompt, KeyDocument, KeyValidationResult, CreateKeyRequest, UpdateKeyRequest, CodingAgent, AgentVersion, McpServerDocument, CreateMcpServerRequest, UpdateMcpServerRequest, BulkResubmitOverrides, Insight, InsightWithReference, TaskPrompt, TaskPromptFeatureExtractionResult, Model, FeatureFlag, SkillDocument, SkillSearchResult, SkillDiscoveryResult, SkillRevisionDocument, CodebaseDocument, CodebaseRevisionDocument, CodebaseSourceType, ExtensionDocument, ExtensionSearchResult, ExtensionVersionInfo, MdpResponse, AccountDocument, CreateAccountRequest, UpdateAccountRequest, ProfileWithVersion, ProfileVersionDocument, ProfileDocument, RunGroup, CursorPaginatedResponse, IterationOp, GateConfig, GateId, PromptType } from "@/types"; +import type { Run, RunState, CriteriaDocument, CriteriaGraphData, GeneratePromptResponse, AnalysisResponse, PromptFeatureDocument, Report, BulkReportStatus, BulkReportSummary, ReportTemplate, ReportTrigger, ReportTemplateSystemPrompt, KeyDocument, KeyValidationResult, CreateKeyRequest, UpdateKeyRequest, CodingAgent, AgentVersion, McpServerDocument, CreateMcpServerRequest, UpdateMcpServerRequest, BulkResubmitOverrides, Insight, InsightWithReference, TaskPrompt, TaskPromptFeatureExtractionResult, Model, FeatureFlag, SkillDocument, SkillSearchResult, SkillDiscoveryResult, SkillRevisionDocument, CodebaseDocument, CodebaseRevisionDocument, CodebaseSourceType, ExtensionDocument, ExtensionSearchResult, ExtensionVersionInfo, MdpResponse, AccountDocument, CreateAccountRequest, UpdateAccountRequest, ProfileWithVersion, ProfileVersionDocument, ProfileDocument, RunGroup, CursorPaginatedResponse, IterationOp, GateConfig, GateId, PromptType, CriterionKind, TaxonomyElementId } from "@/types"; import { qs } from "./url"; import { recordServerDate } from "./serverClock"; @@ -81,6 +81,7 @@ export const api = { /** Submit a new run (or multiple runs if count > 1) */ submitRun: (body: { scenario: { task: string; criteria: string[]; version?: "v1" | "v2" }; + observations?: string[]; worker?: string; model?: string; reasoningEffort?: string; @@ -317,7 +318,7 @@ export const api = { }, /** Create a new criterion */ - createCriterion: (body: { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[] }): Promise => { + createCriterion: (body: { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }): Promise => { return request("/criteria", { method: "POST", body: JSON.stringify(body), @@ -325,7 +326,7 @@ export const api = { }, /** Update an existing criterion */ - updateCriterion: (id: string, body: { prompt?: string; dependsOn?: string[]; gates?: GateId[] }): Promise => { + updateCriterion: (id: string, body: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }): Promise => { return request(`/criteria/${id}`, { method: "PUT", body: JSON.stringify(body), diff --git a/apps/portal/src/pages/CriteriaList.tsx b/apps/portal/src/pages/CriteriaList.tsx index 2f64e390a..3c07c9800 100644 --- a/apps/portal/src/pages/CriteriaList.tsx +++ b/apps/portal/src/pages/CriteriaList.tsx @@ -35,12 +35,16 @@ import { type CustomizeColumnsOption, } from "@/components/list-layout"; import { HelpTooltip } from "@/components/HelpTooltip"; +import { CriteriaKindBadge } from "@/components/CriteriaBadge"; +import { TAXONOMY_ELEMENT_METADATA } from "@/types"; -const FILTER_KEYS = ["gate"] as const; +const FILTER_KEYS = ["gate", "kind"] as const; const COLUMN_OPTIONS: CustomizeColumnsOption[] = [ { id: "id", label: "ID", required: true }, { id: "prompt", label: "Prompt" }, + { id: "kind", label: "Kind" }, + { id: "taxonomy", label: "Dimension" }, { id: "gates", label: "Gates" }, { id: "dependencies", label: "Dependencies" }, { id: "actions", label: "Actions" }, @@ -87,12 +91,17 @@ export function CriteriaList() { const sortedCriteria = useMemo(() => { const selectedGates = state.getFilterList("gate") as GateId[]; + const selectedKinds = state.getFilterList("kind"); // A criterion matches if it is compatible with any selected gate (OR). // Criteria with no explicit gates are compatible with every gate. - const filtered = + const gateFiltered = selectedGates.length === 0 ? criteria - : criteria.filter((c) => selectedGates.some((g) => isCriterionCompatibleWithGate(c.gates, g))); + : criteria.filter((c) => (c.kind ?? "gate") === "gate" && selectedGates.some((g) => isCriterionCompatibleWithGate(c.gates, g))); + const filtered = + selectedKinds.length === 0 + ? gateFiltered + : gateFiltered.filter((c) => selectedKinds.includes(c.kind ?? "gate")); if (!state.sort) return filtered; const sorted = [...filtered]; sorted.sort((a, b) => { @@ -111,11 +120,19 @@ export function CriteriaList() { visibleGates.map((gate) => ({ value: gate, label: GATE_METADATA[gate].label, - count: criteria.filter((c) => isCriterionCompatibleWithGate(c.gates, gate)).length, + count: criteria.filter((c) => (c.kind ?? "gate") === "gate" && isCriterionCompatibleWithGate(c.gates, gate)).length, })), [criteria, visibleGates], ); + const kindFilterOptions = useMemo( + () => [ + { value: "gate", label: "Gate", count: criteria.filter((c) => (c.kind ?? "gate") === "gate").length }, + { value: "observation", label: "Observation", count: criteria.filter((c) => c.kind === "observation").length }, + ], + [criteria], + ); + const total = sortedCriteria.length; const pageStart = (state.page - 1) * state.pageSize; const pageItems = sortedCriteria.slice(pageStart, pageStart + state.pageSize); @@ -159,15 +176,41 @@ export function CriteriaList() { {truncate(c.prompt, 100)} ), }, + { + id: "kind", + header: "Kind", + width: "130px", + hidden: visibility.isHidden("kind"), + cell: (c) => , + }, + { + id: "taxonomy", + header: "Dimension", + width: "190px", + hidden: visibility.isHidden("taxonomy"), + cell: (c) => ( + + {c.kind === "observation" && c.taxonomyElementId + ? TAXONOMY_ELEMENT_METADATA[c.taxonomyElementId].label + : c.kind === "observation" + ? "Unclassified" + : "—"} + + ), + }, { id: "gates", header: "Gates", width: "160px", hidden: visibility.isHidden("gates"), cell: (c) => ( - - {formatGateList(c.gates)} - + c.kind === "observation" ? ( + Record-only + ) : ( + + {formatGateList(c.gates)} + + ) ), }, { @@ -297,6 +340,13 @@ export function CriteriaList() { onToggle={(value) => state.toggleFilterValue("gate", value)} /> + + state.toggleFilterValue("kind", value)} + /> + } secondaryPanel={ diff --git a/apps/portal/src/pages/CriterionDetail.tsx b/apps/portal/src/pages/CriterionDetail.tsx index 82c951105..51ec8b358 100644 --- a/apps/portal/src/pages/CriterionDetail.tsx +++ b/apps/portal/src/pages/CriterionDetail.tsx @@ -11,6 +11,7 @@ import { Badge } from "@/components/ui/badge"; import { Input } from "@/components/ui/input"; import { Textarea } from "@/components/ui/textarea"; import { Label } from "@/components/ui/label"; +import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"; import { Separator } from "@/components/ui/separator"; import { AlertDialog, AlertDialogAction, AlertDialogCancel, AlertDialogContent, @@ -19,7 +20,9 @@ import { import { Skeleton } from "@/components/ui/skeleton"; import { ArrowLeft, Save, Trash2, Loader2, Sparkles, Check, X, Plus, Download } from "lucide-react"; import { CriteriaPicker } from "@/components/CriteriaPicker"; +import { CriteriaKindBadge } from "@/components/CriteriaBadge"; import { GateCompatibilityPicker } from "@/components/GateCompatibilityPicker"; +import { ObservationTaxonomySelect } from "@/components/ObservationTaxonomySelect"; import { formatDate } from "@/lib/utils"; import { formatGateList, @@ -30,6 +33,7 @@ import { useCommandEnter } from "@/hooks/useCommandEnter"; import { KbdBadge } from "@/components/KbdBadge"; import { criteriaToExportYaml, downloadAsFile } from "@/lib/criteria-export"; import { toast } from "sonner"; +import { TAXONOMY_ELEMENT_METADATA, type CriterionKind, type TaxonomyElementId } from "@/types"; export function CriterionDetail() { const { id } = useParams<{ id: string }>(); @@ -49,6 +53,8 @@ export function CriterionDetail() { const [prompt, setPrompt] = useState(""); const [editDependsOn, setEditDependsOn] = useState([]); const [editGates, setEditGates] = useState(undefined); + const [editKind, setEditKind] = useState("gate"); + const [editTaxonomyElementId, setEditTaxonomyElementId] = useState(undefined); // AI Suggest state const [aiSuggestOpen, setAiSuggestOpen] = useState(false); @@ -64,7 +70,7 @@ export function CriterionDetail() { }); const updateMutation = useMutation({ - mutationFn: (body: { prompt?: string; dependsOn?: string[]; gates?: GateId[] }) => + mutationFn: (body: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }) => api.updateCriterion(id!, body), onSuccess: async () => { // Update accepted children to depend on this criterion @@ -125,6 +131,8 @@ export function CriterionDetail() { setPrompt(criterion.prompt); setEditDependsOn(criterion.dependsOn ?? []); setEditGates(criterion.gates && criterion.gates.length > 0 ? criterion.gates : ["select"]); + setEditKind(criterion.kind ?? "gate"); + setEditTaxonomyElementId(criterion.taxonomyElementId); setAiSuggestOpen(false); setBehaviorInput(""); setSuggestedPrompt(null); @@ -135,14 +143,16 @@ export function CriterionDetail() { }; const handleSave = () => { - if (!editGates || editGates.length === 0) return; + if (editKind === "gate" && (!editGates || editGates.length === 0)) return; updateMutation.mutate({ prompt: prompt.trim(), // Always send the array (even empty) so removing the last dependency // persists. The backend treats `undefined` as "no change", so collapsing // [] to undefined here silently dropped deletions. dependsOn: editDependsOn, - gates: editGates, + ...(editKind === "gate" ? { gates: editGates } : {}), + kind: editKind, + taxonomyElementId: editKind === "observation" ? editTaxonomyElementId : undefined, }); }; @@ -164,13 +174,13 @@ export function CriterionDetail() { ? [ ...editDependsOn.flatMap((depId) => { const parent = allCriteria.find((c) => c.id === depId); - return parent && !gatesSatisfyInvariant(parent.gates, editGates) + return editKind === "gate" && parent && !gatesSatisfyInvariant(parent.gates, editGates) ? [`Parent '${depId}' must include ${formatGateList(editGates)}.`] : []; }), ...(criterion?.dependents.flatMap((childId) => { const child = allCriteria.find((c) => c.id === childId); - return child && !gatesSatisfyInvariant(editGates, child.gates) + return editKind === "gate" && child && !gatesSatisfyInvariant(editGates, child.gates) ? [`Dependent '${childId}' requires ${formatGateList(child.gates)} compatibility.`] : []; }) ?? []), @@ -451,7 +461,54 @@ export function CriterionDetail() { + {/* Kind */} + + + Kind + + Whether this criterion gates iteration flow or records observation evidence. + + + + {editing ? ( + <> +
+ + +
+ {editKind === "observation" && ( + + )} + + ) : ( +
+ + {(criterion.kind ?? "gate") === "observation" && ( + + {criterion.taxonomyElementId + ? TAXONOMY_ELEMENT_METADATA[criterion.taxonomyElementId].label + : "Unclassified"} + + )} +
+ )} + + + {/* Gate compatibility */} + {(editing ? editKind === "gate" : (criterion.kind ?? "gate") === "gate") && ( Gate compatibility @@ -479,6 +536,7 @@ export function CriterionDetail() { )} + )} {/* Dependencies */} @@ -495,6 +553,8 @@ export function CriterionDetail() { c.id !== criterion.id && (editKind === "observation" || gatesSatisfyInvariant(c.gates, editGates))} /> ) : (criterion.dependsOn ?? []).length > 0 ? (
diff --git a/apps/portal/src/pages/RunDetail.tsx b/apps/portal/src/pages/RunDetail.tsx index eb2732011..12308f173 100644 --- a/apps/portal/src/pages/RunDetail.tsx +++ b/apps/portal/src/pages/RunDetail.tsx @@ -41,7 +41,14 @@ import { } from "@/lib/gates"; import { useState, useMemo, useEffect, type ReactNode } from "react"; import { toast } from "sonner"; -import type { RunState, LogEvent } from "@/types"; +import { + TAXONOMY_ELEMENT_METADATA, + type CriteriaDocument, + type CriterionResult, + type RunState, + type LogEvent, + type TaxonomyElementId, +} from "@/types"; import { useShiftModifier } from "@/hooks/useShiftModifier"; import { getRetryButtonState } from "@/components/RetryButton"; @@ -202,6 +209,91 @@ function CriteriaGateTabs({ ); } +type ObservationDimensionKey = TaxonomyElementId | "unclassified"; + +function observationDimensionLabel(key: ObservationDimensionKey): string { + return key === "unclassified" ? "Unclassified" : TAXONOMY_ELEMENT_METADATA[key].label; +} + +function ObservationResultsPanel({ + turns, + criteria, +}: { + turns: RunState["turns"]; + criteria: CriteriaDocument[]; +}) { + const criteriaById = useMemo(() => new Map(criteria.map((criterion) => [criterion.id, criterion])), [criteria]); + const groups = useMemo(() => { + const byDimension = new Map[number]; result: CriterionResult; criterion?: CriteriaDocument }>>(); + for (const turn of turns ?? []) { + for (const result of turn.observationResults ?? []) { + const criterion = criteriaById.get(result.criterionId); + const key: ObservationDimensionKey = criterion?.taxonomyElementId ?? "unclassified"; + const existing = byDimension.get(key) ?? []; + existing.push({ turn, result, criterion }); + byDimension.set(key, existing); + } + } + return Array.from(byDimension.entries()).sort(([a], [b]) => observationDimensionLabel(a).localeCompare(observationDimensionLabel(b))); + }, [turns, criteriaById]); + + if (groups.length === 0) { + return ( + + + No observation results have been recorded for this attempt. + + + ); + } + + return ( +
+ {groups.map(([dimension, items]) => ( + + + + + Observation + + {observationDimensionLabel(dimension)} + + + + {items.map(({ turn, result, criterion }) => ( +
+
+ Iteration {turn.iteration} + {turn.gate && {GATE_METADATA[turn.gate].label}} + {result.criterionId} + + {result.evaluated ? ( + result.passed ? : + ) : ( + + )} + {result.evaluated ? (result.passed ? "True" : "False") : "Not evaluated"} + +
+ {criterion?.prompt && ( +

{criterion.prompt}

+ )} +
+

Evidence

+

{result.feedback || "No evidence provided."}

+
+
+ ))} + + + ))} +
+ ); +} + export function RunDetail() { const { id, tab } = useParams<{ id: string; tab?: string }>(); const navigate = useNavigate(); @@ -300,6 +392,14 @@ export function RunDetail() { enabled: !!taskPromptId, }); + const hasObservationResults = (activeRun?.turns ?? []).some((turn) => (turn.observationResults?.length ?? 0) > 0); + const hasConfiguredObservations = (run?.observations?.length ?? 0) > 0; + const { data: observationCriteria = [] } = useQuery({ + queryKey: ["criteria", "observations", run?.observations], + queryFn: () => api.listCriteria(undefined, run?.observations?.length ? { ids: run.observations } : undefined), + enabled: hasConfiguredObservations || hasObservationResults, + }); + // Fetch reports for this run const { data: reports, refetch: refetchReports } = useQuery({ queryKey: ["run-reports", id], @@ -860,6 +960,11 @@ export function RunDetail() { {hasHarData && Network} {hasHarData && Tool Calls} {hasVideoData && } + {(hasConfiguredObservations || hasObservationResults) && ( + + Observations {hasObservationResults ? `(${activeRun?.turns?.reduce((count, turn) => count + (turn.observationResults?.length ?? 0), 0) ?? 0})` : ""} + + )} Logs Reports {reports && reports.length > 0 ? `(${reports.length})` : ""} @@ -955,6 +1060,12 @@ export function RunDetail() { )} + {(hasConfiguredObservations || hasObservationResults) && ( + + + + )} + {/* Logs tab */} {criteriaGates.length > 0 && ( @@ -1175,6 +1286,25 @@ export function RunDetail() {
)} + {(run.observations?.length ?? 0) > 0 && ( +
+

Observations ({run.observations!.length})

+
+ {run.observations!.map((criterionId) => ( + + ))} +
+
+ )} diff --git a/apps/portal/src/pages/SubmitRun.tsx b/apps/portal/src/pages/SubmitRun.tsx index bb40f1fca..e0c075802 100644 --- a/apps/portal/src/pages/SubmitRun.tsx +++ b/apps/portal/src/pages/SubmitRun.tsx @@ -197,6 +197,7 @@ export function SubmitRun() { // Form state const [task, setTask] = useState(""); const [pickedCriteria, setPickedCriteria] = useState([]); + const [pickedObservations, setPickedObservations] = useState([]); const [worker, setWorker] = useState("coder-acp-copilot"); const [model, setModel] = useState(""); const [reasoningEffort, setReasoningEffort] = useState(""); @@ -228,6 +229,7 @@ export function SubmitRun() { // Inline criteria creation dialog const [createCriterionOpen, setCreateCriterionOpen] = useState(false); + const [createObservationOpen, setCreateObservationOpen] = useState(false); const [gateCriterionDialog, setGateCriterionDialog] = useState | null>(null); const [createProfileOpen, setCreateProfileOpen] = useState(false); @@ -459,6 +461,7 @@ export function SubmitRun() { const applyRecentRun = (run: Run) => { if (run.scenario?.task) setTask(run.scenario.task); if (run.scenario?.criteria) setPickedCriteria(run.scenario.criteria); + setPickedObservations(run.observations ?? []); setWorker(run.workerType); if (run.model) setModel(run.model); if (run.agentVersion) setSelectedAgentVersion(run.agentVersion); @@ -615,6 +618,7 @@ export function SubmitRun() { submitMutation.mutate({ scenario: { task: task.trim(), criteria: pickedCriteria }, + ...(pickedObservations.length > 0 ? { observations: pickedObservations } : {}), ...(inVariationMode ? {} : { ...(worker ? { worker } : {}) }), ...(inVariationMode ? {} : { ...(model ? { model } : {}) }), ...(inVariationMode ? {} : { ...(reasoningEffort ? { reasoningEffort } : {}) }), @@ -713,6 +717,7 @@ export function SubmitRun() { const summaryChips: string[] = [ `${maxIterations} iteration${maxIterations === 1 ? "" : "s"}`, `${pickedCriteria.length} select criteri${pickedCriteria.length === 1 ? "on" : "a"}`, + pickedObservations.length > 0 ? `${pickedObservations.length} observation${pickedObservations.length === 1 ? "" : "s"}` : "", gatesEnabled ? `${gateConfigs.length} configured gates` : "single-pass Select", occurrences > 1 ? `×${occurrences} runs` : "", worker, @@ -939,6 +944,7 @@ export function SubmitRun() { onChange={setPickedCriteria} inputId="criteria" gate="select" + kind="gate" trailingAction={( + )} + /> + setPickedObservations((prev) => [...prev, id])} + /> +

+ + Observation results appear in the run detail observations tab grouped by dimension. +

+ +
@@ -1099,6 +1145,7 @@ export function SubmitRun() { updateGateDraft(gate, { criteria })} trailingAction={( diff --git a/apps/portal/src/types.ts b/apps/portal/src/types.ts index 74918b4bf..079ef7b2d 100644 --- a/apps/portal/src/types.ts +++ b/apps/portal/src/types.ts @@ -27,6 +27,40 @@ export interface ToolCall { timestamp?: string; } +export type CriterionKind = "gate" | "observation"; + +export const TAXONOMY_ELEMENT_IDS = [ + "dimension:idiomatic-use", + "dimension:dependency-currency", + "dimension:configuration-correctness", +] as const; + +export type TaxonomyElementId = (typeof TAXONOMY_ELEMENT_IDS)[number]; + +export interface TaxonomyElementMetadata { + id: TaxonomyElementId; + label: string; + description: string; +} + +export const TAXONOMY_ELEMENT_METADATA: Record = { + "dimension:idiomatic-use": { + id: "dimension:idiomatic-use", + label: "Idiomatic use", + description: "Did the agent follow recommended patterns and best practices?", + }, + "dimension:dependency-currency": { + id: "dimension:dependency-currency", + label: "Dependency currency", + description: "Did the agent use current versions, packages, and namespaces?", + }, + "dimension:configuration-correctness": { + id: "dimension:configuration-correctness", + label: "Configuration correctness", + description: "Are auth, connection, environment, and deployment settings configured properly?", + }, +}; + export interface ConversationTurn { iteration: number; gate?: GateId; @@ -36,6 +70,7 @@ export interface ConversationTurn { passed: boolean; timestamp: string; criteriaResults?: CriterionResult[]; + observationResults?: CriterionResult[]; harUrl?: string; videoUrls?: string[]; tokenUsage?: TokenUsage; @@ -144,6 +179,7 @@ export interface Run { profileId?: string; profileVersionId?: string; gates?: GateConfig[]; + observations?: string[]; gateSummaries?: GateRunSummary[]; } @@ -206,6 +242,8 @@ export interface CriteriaConfig { prompt: string; dependsOn?: string[]; gates?: GateId[]; + kind?: CriterionKind; + taxonomyElementId?: TaxonomyElementId; } export interface CriteriaDocument extends CriteriaConfig { @@ -215,7 +253,7 @@ export interface CriteriaDocument extends CriteriaConfig { } export interface CriteriaGraphData { - nodes: Array<{ id: string; prompt: string; dependsOn: string[]; gates?: GateId[] }>; + nodes: Array<{ id: string; prompt: string; dependsOn: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }>; edges: Array<{ source: string; target: string }>; } From 78899c20b1b1c7226f7e19efffe812c76206c795 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 11:39:49 -0700 Subject: [PATCH 03/17] test(judge): live ATIF-trajectory observation acceptance (issue #1156) Proves the judge consumes atifUrl: a trajectory-only observation (dependency_added_then_removed) resolves passed:true WITH the synthetic agent_trajectory_atif tool call and passed:false WITHOUT it. Excluded from default vitest; runs via vitest.integration.config.ts with a Copilot token. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...observation-trajectory.integration.test.ts | 165 ++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 apps/judge/src/observation-trajectory.integration.test.ts diff --git a/apps/judge/src/observation-trajectory.integration.test.ts b/apps/judge/src/observation-trajectory.integration.test.ts new file mode 100644 index 000000000..c8b31030b --- /dev/null +++ b/apps/judge/src/observation-trajectory.integration.test.ts @@ -0,0 +1,165 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/** + * ATIF-trajectory observation acceptance test (issue #1156, acceptance step 3). + * + * Proves the judge resolves a trajectory-only observation — one whose answer is + * NOT derivable from the final workspace snapshot — by consuming the agent's + * ATIF trajectory that pp-taxonomy forwards as a synthetic `agent_trajectory_atif` + * tool call (see apps/judge/src/index.ts). The canonical case is + * `dependency_added_then_removed`: a package is installed then uninstalled mid-run, + * so it is invisible in the end-state snapshot and only the trajectory reveals it. + * + * Two assertions form the acceptance: + * 1. WITH the trajectory tool call → criterion resolves passed:true (the judge + * actually consumed the ATIF). + * 2. WITHOUT the trajectory (snapshot only) → the same criterion flips to + * passed:false, proving the verdict comes from the trajectory, not the files. + * + * Exercises the REAL Copilot SDK, so it requires GITHUB_TOKEN (copilot scope) and + * is skipped automatically when the token is absent — mirroring + * judge-evaluation.integration.test.ts. + */ +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import { mkdtempSync, writeFileSync, rmSync, realpathSync } from "fs"; +import { tmpdir } from "os"; +import { join } from "path"; +import { resetCriteriaProvider } from "shared/criteria-provider-factory"; +import type { ToolCall } from "shared"; +import { evaluateWorkspace } from "./judge-agent.js"; + +const hasToken = !!process.env.GITHUB_TOKEN; +const JUDGE_MODEL = process.env.JUDGE_MODEL || "gpt-5.4-mini"; + +/** + * Synthetic ATIF trajectory mirroring what pp-taxonomy forwards: the agent + * installs `left-pad`, uses it, then uninstalls it before finishing. The final + * package.json (below) therefore does NOT list left-pad. + */ +const ATIF_TRAJECTORY = { + schema: "atif/1", + steps: [ + { type: "tool_call", name: "run_in_terminal", arguments: { command: "npm install left-pad" }, response: "+ left-pad@1.3.0\nadded 1 package" }, + { type: "tool_call", name: "edit_file", arguments: { path: "index.js" }, response: "require('left-pad')" }, + { type: "tool_call", name: "run_in_terminal", arguments: { command: "npm uninstall left-pad" }, response: "removed 1 package" }, + ], +}; + +const TRAJECTORY_TOOL_CALL: ToolCall = { + id: "atif-trajectory", + name: "agent_trajectory_atif", + arguments: {}, + response: JSON.stringify(ATIF_TRAJECTORY), +}; + +describe("judge ATIF-trajectory observation (integration)", () => { + let workspaceDir: string; + let criteriaDir: string; + const saved: Record = {}; + + function setEnv(key: string, value: string): void { + saved[key] = process.env[key]; + process.env[key] = value; + } + + beforeAll(() => { + workspaceDir = realpathSync(mkdtempSync(join(tmpdir(), "judge-atif-ws-"))); + criteriaDir = realpathSync(mkdtempSync(join(tmpdir(), "judge-atif-crit-"))); + + // End-state snapshot: package.json WITHOUT left-pad. The add-then-remove is + // invisible here; only the trajectory reveals it. + writeFileSync( + join(workspaceDir, "package.json"), + JSON.stringify({ name: "demo", version: "1.0.0", dependencies: {} }, null, 2) + "\n", + ); + writeFileSync(join(workspaceDir, "index.js"), "console.log('hello');\n"); + + // Observation criterion (matches config/criteria/dependency_added_then_removed.yaml). + writeFileSync( + join(criteriaDir, "dependency_added_then_removed.yaml"), + [ + "id: dependency_added_then_removed", + "kind: observation", + "taxonomy_element_id: dimension:dependency-currency", + "prompt: |", + " During the run the agent installed a package and later uninstalled or", + " removed it before finishing (e.g. added left-pad then removed it). This", + " is only derivable from the agent's trajectory (mid-run tool calls), not", + " from the final codebase snapshot, because the dependency is absent in the", + " end state. Inspect the agent_trajectory_atif tool output to detect a", + " transient add-then-remove.", + "", + ].join("\n"), + ); + + setEnv("CRITERIA_DIR", criteriaDir); + setEnv("JUDGE_STRATEGY", process.env.JUDGE_STRATEGY || "independent"); + setEnv("JUDGE_MODEL", JUDGE_MODEL); + setEnv("FEEDBACK_MODEL", process.env.FEEDBACK_MODEL || JUDGE_MODEL); + resetCriteriaProvider(); + }); + + afterAll(() => { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + resetCriteriaProvider(); + rmSync(workspaceDir, { recursive: true, force: true }); + rmSync(criteriaDir, { recursive: true, force: true }); + }); + + it.skipIf(!hasToken)( + "resolves a trajectory-only observation true WITH the ATIF, false WITHOUT it", + { timeout: 600_000 }, + async () => { + // (1) WITH the ATIF trajectory tool call → the agent's add-then-remove is + // visible, so the observation must resolve true. + const withAtif = await evaluateWorkspace({ + workspacePath: workspaceDir, + criteria: ["dependency_added_then_removed"], + conversationHistory: [], + toolCalls: [TRAJECTORY_TOOL_CALL], + }); + + const obsWith = withAtif.criteriaResults.find( + (r) => r.criterionId === "dependency_added_then_removed", + ); + expect(obsWith, "observation result missing (with ATIF)").toBeTruthy(); + expect(obsWith!.evaluated).toBe(true); + expect( + obsWith!.passed, + `expected trajectory-only observation to pass WITH ATIF. Feedback: ${obsWith!.feedback}`, + ).toBe(true); + + // (2) WITHOUT the ATIF (snapshot only) → the add-then-remove is invisible + // in the end state, so the same observation must flip to false. This proves + // the verdict was driven by the trajectory, not the workspace files. + const withoutAtif = await evaluateWorkspace({ + workspacePath: workspaceDir, + criteria: ["dependency_added_then_removed"], + conversationHistory: [], + }); + + const obsWithout = withoutAtif.criteriaResults.find( + (r) => r.criterionId === "dependency_added_then_removed", + ); + expect(obsWithout, "observation result missing (without ATIF)").toBeTruthy(); + expect(obsWithout!.evaluated).toBe(true); + expect( + obsWithout!.passed, + `expected trajectory-only observation to FAIL without ATIF (snapshot has no left-pad). Feedback: ${obsWithout!.feedback}`, + ).toBe(false); + }, + ); + + it("logs a skip note when GITHUB_TOKEN is absent (no real evaluation possible)", () => { + if (!hasToken) { + console.warn( + "[integration] GITHUB_TOKEN not set — ATIF-trajectory observation test skipped", + ); + } + expect(true).toBe(true); + }); +}); From c0a20936ddc1887bfa2545bb85a78c889501d0ed Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 12:01:47 -0700 Subject: [PATCH 04/17] fix(post-processor): copy handler.taxonomy.yaml into image so pp-taxonomy registers The Dockerfile copied only handler.yaml, so the register-handler-taxonomy job (HANDLER_YAML_PATH=apps/workers/post-processor/handler.taxonomy.yaml) failed with ENOENT in the container, exit 1, gating the post-processor (hard depends_on service_completed_successfully). Use a handler*.yaml glob in both builder and prod stages so every handler manifest is baked in. Verified in compose: register-handler-taxonomy now exits 0 ("Registered pp-taxonomy successfully") and the post-processor loads both the atif and taxonomy handlers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- apps/workers/post-processor/Dockerfile | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/apps/workers/post-processor/Dockerfile b/apps/workers/post-processor/Dockerfile index 6d6bf8e04..a70478630 100644 --- a/apps/workers/post-processor/Dockerfile +++ b/apps/workers/post-processor/Dockerfile @@ -15,7 +15,10 @@ COPY packages/shared/tsconfig.json ./packages/shared/ COPY packages/shared/src ./packages/shared/src/ COPY apps/workers/post-processor/tsconfig.json ./apps/workers/post-processor/ COPY apps/workers/post-processor/src ./apps/workers/post-processor/src/ -COPY apps/workers/post-processor/handler.yaml ./apps/workers/post-processor/ +# Copy every handler manifest (handler.yaml = pp-atif, handler.taxonomy.yaml = +# pp-taxonomy, plus any future handlers). The registration jobs read these by +# HANDLER_YAML_PATH, so all must be present in the image. +COPY apps/workers/post-processor/handler*.yaml ./apps/workers/post-processor/ RUN pnpm --filter shared build && pnpm --filter post-processor build @@ -44,7 +47,7 @@ RUN pnpm install --frozen-lockfile --prod COPY --from=builder /app/packages/shared/dist ./packages/shared/dist/ COPY --from=builder /app/apps/workers/post-processor/dist ./apps/workers/post-processor/dist/ -COPY --from=builder /app/apps/workers/post-processor/handler.yaml ./apps/workers/post-processor/handler.yaml +COPY --from=builder /app/apps/workers/post-processor/handler*.yaml ./apps/workers/post-processor/ WORKDIR /app/apps/workers/post-processor From d5f866badaa3350c859c781efa67a090e3456f0c Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 13:30:53 -0700 Subject: [PATCH 05/17] fix(compose): give post-processor JUDGE_SERVICE_URL so pp-taxonomy reaches judge The post-processor service used the *common-env anchor, which omits JUDGE_SERVICE_URL (only *worker-env defines it). pp-atif never called the judge so the gap was invisible, but pp-taxonomy (issue #1156) evaluates observation criteria via JudgeClient and fell back to localhost:3002 inside the container. Set JUDGE_SERVICE_URL=http://judge:80 explicitly. k8s is unaffected: post-processor.yaml already pulls JUDGE_SERVICE_URL from the worker-config ConfigMap via envFrom. Verified in compose: the containerized post-processor reached the judge container, which loaded the ATIF and wrote run.turns[].observationResults (dependency_added_then_removed -> passed:true). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docker-compose.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docker-compose.yml b/docker-compose.yml index aea2d76f7..aa1c06821 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -686,6 +686,10 @@ services: AZURE_STORAGE_QUEUE_POSTPROCESSOR: post-processor-queue SCOPE_MT_API_URL: http://api:80 SCHEDULER_URL: http://scheduler:8080 + # pp-taxonomy (issue #1156) calls the judge service to evaluate + # observation criteria. Without this the JudgeClient falls back to + # localhost:3002 and cannot reach the judge container. + JUDGE_SERVICE_URL: http://judge:80 depends_on: mongodb: condition: service_healthy From 7fde27f8200624a4693a56b362e144b57d43cd0c Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 13:32:02 -0700 Subject: [PATCH 06/17] docs(env): clarify post-processor JUDGE_SERVICE_URL wiring in compose vs k8s In k8s the post-processor inherits JUDGE_SERVICE_URL from the worker-config ConfigMap via envFrom; in docker-compose the service uses the common-env anchor (not worker-env), so JUDGE_SERVICE_URL is set explicitly on the post-processor service. Reflects the compose fix in 3f2e98cc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ENV_VARIABLES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ENV_VARIABLES.md b/ENV_VARIABLES.md index 3001c4b16..293c83db2 100644 --- a/ENV_VARIABLES.md +++ b/ENV_VARIABLES.md @@ -347,7 +347,7 @@ Azure Storage Queue name used by both the scheduler (to enqueue post-processing **Default:** falls back to `JUDGE_SERVICE_URL`, then `http://localhost:3002` **Type:** URL string -Base URL of the judge service used by the `pp-taxonomy` post-process handler to evaluate observation criteria per iteration (snapshot + ATIF trajectory → boolean + evidence). The handler reuses the shared `JudgeClient`, so `JUDGE_CLIENT_TIMEOUT` / `JUDGE_CLIENT_RETRIES` apply. In Docker/K8s the worker reads `JUDGE_SERVICE_URL` from `worker-config`; no extra var needed. See issue #1156 and [docs/architecture/post-processing.md](docs/architecture/post-processing.md). +Base URL of the judge service used by the `pp-taxonomy` post-process handler to evaluate observation criteria per iteration (snapshot + ATIF trajectory → boolean + evidence). The handler reuses the shared `JudgeClient`, so `JUDGE_CLIENT_TIMEOUT` / `JUDGE_CLIENT_RETRIES` apply. In K8s the post-processor reads `JUDGE_SERVICE_URL` from the `worker-config` ConfigMap (`envFrom`); in docker-compose the `post-processor` service sets `JUDGE_SERVICE_URL: http://judge:80` explicitly (it uses the `common-env` anchor, which — unlike `worker-env` — does not carry it). See issue #1156 and [docs/architecture/post-processing.md](docs/architecture/post-processing.md). ### SCHEDULER_URL **Default:** (not set) From 7a639ad33bef7f90cbea30fcead8082f038a17f7 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 14:17:48 -0700 Subject: [PATCH 07/17] fix(pp-taxonomy): read selected observations from request top-level, not run.observations The API submit path persists the selected observation-criteria ids at the request document top-level (RequestDocument.observations, types.ts), but the TaxonomyHandler read them from doc.run.observations. They never met, so pp-taxonomy no-op'd on every real submitted run and never wrote turn.observationResults. Read doc.observations (top-level) instead; turns still come from doc.run.turns. Rename the local RunDoc interface to RequestDoc and document the placement. Update the handler unit tests to put observations at the canonical top-level, and add a regression test asserting that legacy run.observations is ignored (no-op), which would have caught this. Verified end-to-end on a real submitted run through the containerized stack: turn.observationResults is written with passed:true and feedback citing the mid-run install/uninstall trajectory (the judge consumed atifUrl). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/handlers/taxonomy-handler.test.ts | 22 ++++++++++++++---- .../src/handlers/taxonomy-handler.ts | 23 +++++++++++-------- 2 files changed, 31 insertions(+), 14 deletions(-) diff --git a/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts b/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts index 18831592a..4b328b8e6 100644 --- a/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts +++ b/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts @@ -31,7 +31,21 @@ describe("TaxonomyHandler", () => { }); it("no-ops when no observations are selected", async () => { - const ctx = makeCtx({ _id: "req-1", run: { observations: [], turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }); + const ctx = makeCtx({ _id: "req-1", observations: [], run: { turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }); + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + expect(judge.evaluate).not.toHaveBeenCalled(); + expect(ctx.collection.updateOne).not.toHaveBeenCalled(); + }); + + it("reads observations from the request top-level, not run.observations (#1156 regression)", async () => { + // The API persists selected observation ids at the request top-level + // (RequestDocument.observations). A doc that only carries the legacy + // run.observations must be treated as "none selected" → no-op, proving the + // handler reads the canonical location the submit path actually writes. + const ctx = makeCtx({ + _id: "req-1", + run: { observations: ["dep_removed"], turns: [{ iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }] }, + }); await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); expect(judge.evaluate).not.toHaveBeenCalled(); expect(ctx.collection.updateOne).not.toHaveBeenCalled(); @@ -45,8 +59,8 @@ describe("TaxonomyHandler", () => { }); const ctx = makeCtx({ _id: "req-1", + observations: ["dep_removed"], run: { - observations: ["dep_removed"], turns: [{ iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }], }, }); @@ -67,14 +81,14 @@ describe("TaxonomyHandler", () => { it("skips iterations without a snapshotUrl", async () => { judge.evaluate.mockResolvedValue({ passed: true, feedback: "", criteriaResults: [] }); - const ctx = makeCtx({ _id: "req-1", run: { observations: ["c1"], turns: [{ iteration: 1 }] } }); + const ctx = makeCtx({ _id: "req-1", observations: ["c1"], run: { turns: [{ iteration: 1 }] } }); await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); expect(judge.evaluate).not.toHaveBeenCalled(); }); it("throws on judge infrastructure error so the handler is marked failed", async () => { judge.evaluate.mockRejectedValue(new JudgeInfrastructureError("down", { httpStatus: 503 })); - const ctx = makeCtx({ _id: "req-1", run: { observations: ["c1"], turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }); + const ctx = makeCtx({ _id: "req-1", observations: ["c1"], run: { turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }); await expect(new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx)).rejects.toThrow("down"); }); }); diff --git a/apps/workers/post-processor/src/handlers/taxonomy-handler.ts b/apps/workers/post-processor/src/handlers/taxonomy-handler.ts index 3701c4cd8..571320317 100644 --- a/apps/workers/post-processor/src/handlers/taxonomy-handler.ts +++ b/apps/workers/post-processor/src/handlers/taxonomy-handler.ts @@ -5,11 +5,14 @@ import { JudgeClient, isRetryableJudgeError } from "shared"; import type { ConversationTurn, CriterionResult } from "shared"; import type { PostProcessHandler, PostProcessorMessage, HandlerContext } from "../types.js"; -/** Minimal shape of the run document fields this handler reads/writes. */ -interface RunDoc { +/** Minimal shape of the request document fields this handler reads/writes. */ +interface RequestDoc { + /** Observation-criteria ids selected at submit. Stored at the request + * top-level by the API (see RequestDocument.observations / issue #1156), + * NOT under `run` — the run subdocument only carries per-turn results. */ + observations?: string[]; run?: { _id?: string; - observations?: string[]; turns?: ConversationTurn[]; }; } @@ -18,11 +21,11 @@ interface RunDoc { * Taxonomy Handler (pp-taxonomy): records per-iteration boolean observations. * * For each iteration turn with a `snapshotUrl`, reuses the judge service to - * evaluate the run's selected observation criteria (`run.observations`) against - * the codebase snapshot AND the agent trajectory (ATIF) — sending `atifUrl` in - * place of `toolCallsUrl`. Results are written to `turn.observationResults`, - * mirroring gate `criteriaResults`. Never gates, never steers the agent. - * Empty/absent `observations` → no-op. See issue #1156. + * evaluate the run's selected observation criteria (top-level + * `request.observations`) against the codebase snapshot AND the agent + * trajectory (ATIF) — sending `atifUrl` in place of `toolCallsUrl`. Results are + * written to `turn.observationResults`, mirroring gate `criteriaResults`. Never + * gates, never steers the agent. Empty/absent `observations` → no-op. #1156. */ export class TaxonomyHandler implements PostProcessHandler { readonly type = "taxonomy"; @@ -35,8 +38,8 @@ export class TaxonomyHandler implements PostProcessHandler { async process(message: PostProcessorMessage, ctx: HandlerContext): Promise { const { requestId, iteration } = message; - const doc = (await ctx.collection.findOne({ _id: requestId } as any)) as RunDoc | null; - const observations = doc?.run?.observations ?? []; + const doc = (await ctx.collection.findOne({ _id: requestId } as any)) as RequestDoc | null; + const observations = doc?.observations ?? []; const turns = doc?.run?.turns ?? []; if (observations.length === 0) { From f689b013948c7ded948812dc2a63ebdc9336f004 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 14:34:20 -0700 Subject: [PATCH 08/17] feat(portal): rename run-detail Observations tab to Taxonomy with Observations subsection The run-detail tab is now labeled "Taxonomy" and its content nests an "Observations" sub-section (heading + description) around the per-dimension ObservationResultsPanel, so the tab can host the broader taxonomy view with observations as one section within it. The internal tab value stays "observations" to preserve routing/state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- apps/portal/src/pages/RunDetail.tsx | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/apps/portal/src/pages/RunDetail.tsx b/apps/portal/src/pages/RunDetail.tsx index 12308f173..74a1e9623 100644 --- a/apps/portal/src/pages/RunDetail.tsx +++ b/apps/portal/src/pages/RunDetail.tsx @@ -962,7 +962,7 @@ export function RunDetail() { {hasVideoData && } {(hasConfiguredObservations || hasObservationResults) && ( - Observations {hasObservationResults ? `(${activeRun?.turns?.reduce((count, turn) => count + (turn.observationResults?.length ?? 0), 0) ?? 0})` : ""} + Taxonomy {hasObservationResults ? `(${activeRun?.turns?.reduce((count, turn) => count + (turn.observationResults?.length ?? 0), 0) ?? 0})` : ""} )} Logs @@ -1061,8 +1061,16 @@ export function RunDetail() { )} {(hasConfiguredObservations || hasObservationResults) && ( - - + +
+
+

Observations

+

+ Per-iteration boolean observations recorded over the codebase and agent trajectory, grouped by taxonomy dimension. Non-gating — they never steer the agent. +

+
+ +
)} From 85eb446ea525af152305e935ff85563be55589a5 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 14:53:11 -0700 Subject: [PATCH 09/17] fix(portal): aggregate Enrichment badge over handler DAG + rename observations tab URL to taxonomy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Enrichment badge read the legacy single `run.postProcessorStatus` scalar, which every post-processor handler overwrites on its own success. It therefore flipped to "Enriched" as soon as pp-atif finished, while pp-taxonomy was still computing observations. Derive an aggregate enrichment status from `run.handlerStatus` (the #1147 per-handler DAG state): stay "Enriching…" until every handler is terminal, treating pp-taxonomy as pending whenever observations are configured (which also covers the gap before the scheduler dispatches it). Use the derived status for both badge placements and the detail poller so taxonomy results auto-appear. Falls back to the legacy scalar for pre-DAG runs. Also rename the run-detail tab slug "observations" -> "taxonomy" so the URL reads /runs/:id/taxonomy, with an in-place redirect for legacy /observations deep-links. Proven live on the containerized stack: a real run held "Enriching…" from pp-atif done (legacy scalar already "done") through pp-taxonomy queued, and flipped to "Enriched" only once pp-taxonomy completed ~32s later. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- apps/portal/src/lib/enrichment.test.ts | 92 ++++++++++++++++++++++++++ apps/portal/src/lib/enrichment.ts | 48 ++++++++++++++ apps/portal/src/pages/RunDetail.tsx | 31 +++++++-- apps/portal/src/types.ts | 19 ++++++ 4 files changed, 183 insertions(+), 7 deletions(-) create mode 100644 apps/portal/src/lib/enrichment.test.ts create mode 100644 apps/portal/src/lib/enrichment.ts diff --git a/apps/portal/src/lib/enrichment.test.ts b/apps/portal/src/lib/enrichment.test.ts new file mode 100644 index 000000000..879bc5b11 --- /dev/null +++ b/apps/portal/src/lib/enrichment.test.ts @@ -0,0 +1,92 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import { describe, expect, it } from "vitest"; +import { deriveEnrichmentStatus } from "./enrichment"; +import type { RunState } from "@/types"; + +function makeRun(partial: Partial): RunState { + return { _id: "r1", attemptNumber: 1, status: "done", ...partial } as RunState; +} + +describe("deriveEnrichmentStatus", () => { + it("returns undefined when there is no run", () => { + expect(deriveEnrichmentStatus(undefined, false)).toBeUndefined(); + }); + + it("falls back to the legacy scalar for runs without a handler DAG", () => { + expect(deriveEnrichmentStatus(makeRun({ postProcessorStatus: "done" }), false)).toBe("done"); + expect(deriveEnrichmentStatus(makeRun({ postProcessorStatus: "processing" }), true)).toBe("processing"); + expect(deriveEnrichmentStatus(makeRun({ handlerStatus: {} }), false)).toBeUndefined(); + }); + + it("stays processing while pp-atif is still running", () => { + const run = makeRun({ handlerStatus: { "pp-atif": { status: "processing" } } }); + expect(deriveEnrichmentStatus(run, false)).toBe("processing"); + }); + + it("reports done once the only handler (pp-atif) is done and no observations are configured", () => { + const run = makeRun({ + postProcessorStatus: "done", + handlerStatus: { "pp-atif": { status: "done" } }, + }); + expect(deriveEnrichmentStatus(run, false)).toBe("done"); + }); + + it("does NOT flip to done after pp-atif while pp-taxonomy is still pending (the bug)", () => { + // Observations configured, pp-atif done, pp-taxonomy not yet dispatched. + const beforeDispatch = makeRun({ + postProcessorStatus: "done", // legacy scalar already flipped — must be ignored + handlerStatus: { "pp-atif": { status: "done" } }, + }); + expect(deriveEnrichmentStatus(beforeDispatch, true)).toBe("processing"); + + // pp-taxonomy queued. + const queued = makeRun({ + postProcessorStatus: "done", + handlerStatus: { "pp-atif": { status: "done" }, "pp-taxonomy": { status: "queued" } }, + }); + expect(deriveEnrichmentStatus(queued, true)).toBe("processing"); + + // pp-taxonomy processing. + const processing = makeRun({ + postProcessorStatus: "done", + handlerStatus: { "pp-atif": { status: "done" }, "pp-taxonomy": { status: "processing" } }, + }); + expect(deriveEnrichmentStatus(processing, true)).toBe("processing"); + }); + + it("flips to done only once both pp-atif and pp-taxonomy are done", () => { + const run = makeRun({ + handlerStatus: { "pp-atif": { status: "done" }, "pp-taxonomy": { status: "done" } }, + }); + expect(deriveEnrichmentStatus(run, true)).toBe("done"); + }); + + it("reports failed when a handler failed and nothing is still in flight", () => { + const run = makeRun({ + handlerStatus: { "pp-atif": { status: "done" }, "pp-taxonomy": { status: "failed" } }, + }); + expect(deriveEnrichmentStatus(run, true)).toBe("failed"); + }); + + it("prefers processing over failed while another handler is still running", () => { + const run = makeRun({ + handlerStatus: { "pp-atif": { status: "processing" }, "pp-taxonomy": { status: "failed" } }, + }); + expect(deriveEnrichmentStatus(run, true)).toBe("processing"); + }); + + it("treats a no-op pp-taxonomy (no observations) as part of the DAG", () => { + // No observations configured, but pp-taxonomy still runs as a no-op handler. + const stillRunning = makeRun({ + handlerStatus: { "pp-atif": { status: "done" }, "pp-taxonomy": { status: "processing" } }, + }); + expect(deriveEnrichmentStatus(stillRunning, false)).toBe("processing"); + + const allDone = makeRun({ + handlerStatus: { "pp-atif": { status: "done" }, "pp-taxonomy": { status: "done" } }, + }); + expect(deriveEnrichmentStatus(allDone, false)).toBe("done"); + }); +}); diff --git a/apps/portal/src/lib/enrichment.ts b/apps/portal/src/lib/enrichment.ts new file mode 100644 index 000000000..99f5bfebc --- /dev/null +++ b/apps/portal/src/lib/enrichment.ts @@ -0,0 +1,48 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import type { RunState } from "@/types"; + +export type PostProcessorStatus = "queued" | "processing" | "done" | "failed"; + +/** + * Derive the aggregate enrichment status across the whole post-processing + * handler DAG (issue #1147), rather than trusting the legacy single + * `postProcessorStatus` scalar. + * + * Each post-processor handler overwrites that scalar on its own completion, so + * it flips to "done" as soon as the FIRST handler (pp-atif) finishes — even + * while pp-taxonomy is still computing observations. The "Enriched" badge (and + * the run-detail poller) must instead stay "Enriching…" until every handler in + * the DAG is terminal. + * + * Known topology: pp-atif always runs; pp-taxonomy is expected whenever the run + * has configured observations. We treat taxonomy as still-pending until + * pp-taxonomy reaches a terminal state, which also covers the brief window + * after pp-atif finishes but before the scheduler has dispatched pp-taxonomy + * (so it isn't yet present in `handlerStatus`). + * + * Falls back to the legacy `postProcessorStatus` scalar for runs that predate + * the handler DAG (no `handlerStatus` map). + */ +export function deriveEnrichmentStatus( + run: RunState | undefined, + hasConfiguredObservations: boolean, +): PostProcessorStatus | undefined { + if (!run) return undefined; + const handlerStatus = run.handlerStatus; + // Legacy runs predating the handler DAG only carry the scalar field. + if (!handlerStatus || Object.keys(handlerStatus).length === 0) { + return run.postProcessorStatus; + } + const statuses = Object.values(handlerStatus).map((h) => h?.status); + const taxonomyStatus = handlerStatus["pp-taxonomy"]?.status; + const taxonomyPending = + hasConfiguredObservations && taxonomyStatus !== "done" && taxonomyStatus !== "failed"; + if (taxonomyPending || statuses.some((s) => s === "queued" || s === "processing")) { + return "processing"; + } + if (statuses.some((s) => s === "failed")) return "failed"; + if (statuses.length > 0 && statuses.every((s) => s === "done")) return "done"; + return run.postProcessorStatus; +} diff --git a/apps/portal/src/pages/RunDetail.tsx b/apps/portal/src/pages/RunDetail.tsx index 74a1e9623..2b058d107 100644 --- a/apps/portal/src/pages/RunDetail.tsx +++ b/apps/portal/src/pages/RunDetail.tsx @@ -51,6 +51,7 @@ import { } from "@/types"; import { useShiftModifier } from "@/hooks/useShiftModifier"; import { getRetryButtonState } from "@/components/RetryButton"; +import { deriveEnrichmentStatus } from "@/lib/enrichment"; /** A compact labeled stat: a micro uppercase label above its value. */ function MetaItem({ label, value, title }: { label: string; value: ReactNode; title?: string }) { @@ -314,10 +315,10 @@ export function RunDetail() { refetchInterval: (query) => { const data = query.state.data; const status = data?.run?.status; - const ppStatus = data?.run?.postProcessorStatus; // Keep polling while run is in progress OR post-processing is pending/in-progress if (status !== "done") return 5_000; - if (ppStatus && ppStatus !== "done" && ppStatus !== "failed") return 5_000; + const enrichment = deriveEnrichmentStatus(data?.run, (data?.observations?.length ?? 0) > 0); + if (enrichment && enrichment !== "done" && enrichment !== "failed") return 5_000; return false; }, }); @@ -354,6 +355,16 @@ export function RunDetail() { }, [selectedRunId, run?.run, attempts]); const isViewingHistorical = !!selectedRunId && selectedRunId !== run?.run?._id; + // The "observations" tab was renamed to "taxonomy". Map the legacy slug to the + // new one for tab selection, and rewrite stale `/observations` URLs in place + // so deep-links and bookmarks keep working. + const normalizedTab = tab === "observations" ? "taxonomy" : tab; + useEffect(() => { + if (tab === "observations") { + navigate(`/runs/${id}/taxonomy${selectedRunId ? `?runId=${selectedRunId}` : ""}`, { replace: true }); + } + }, [tab, id, selectedRunId, navigate]); + const isActive = activeRun?.status === "pending" || activeRun?.status === "processing"; // Note: "done" is terminal — not active, no log streaming needed @@ -394,6 +405,12 @@ export function RunDetail() { const hasObservationResults = (activeRun?.turns ?? []).some((turn) => (turn.observationResults?.length ?? 0) > 0); const hasConfiguredObservations = (run?.observations?.length ?? 0) > 0; + // Aggregate enrichment status across the whole post-processing DAG so the + // "Enriched" badge waits for pp-taxonomy instead of flipping after pp-atif. + const enrichmentStatus = useMemo( + () => deriveEnrichmentStatus(activeRun, hasConfiguredObservations), + [activeRun, hasConfiguredObservations], + ); const { data: observationCriteria = [] } = useQuery({ queryKey: ["criteria", "observations", run?.observations], queryFn: () => api.listCriteria(undefined, run?.observations?.length ? { ids: run.observations } : undefined), @@ -673,7 +690,7 @@ export function RunDetail() { /> {activeRun?.status === "done" && } {activeRun?.status === "done" && ( - + )}
@@ -947,7 +964,7 @@ export function RunDetail() { {/* Tabs */} 0 ? "turns" : "logs")} + value={normalizedTab || (activeRun?.status === "done" && activeRun?.turns && activeRun.turns.length > 0 ? "turns" : "logs")} onValueChange={(value) => navigate(`/runs/${id}/${value}${selectedRunId ? `?runId=${selectedRunId}` : ""}`)} > @@ -961,7 +978,7 @@ export function RunDetail() { {hasHarData && Tool Calls} {hasVideoData && } {(hasConfiguredObservations || hasObservationResults) && ( - + Taxonomy {hasObservationResults ? `(${activeRun?.turns?.reduce((count, turn) => count + (turn.observationResults?.length ?? 0), 0) ?? 0})` : ""} )} @@ -1061,7 +1078,7 @@ export function RunDetail() { )} {(hasConfiguredObservations || hasObservationResults) && ( - +

Observations

@@ -1447,7 +1464,7 @@ export function RunDetail() {
Status:{" "} - +
{activeRun.postProcessorVersion !== undefined && (
diff --git a/apps/portal/src/types.ts b/apps/portal/src/types.ts index 079ef7b2d..a7a6b2ec2 100644 --- a/apps/portal/src/types.ts +++ b/apps/portal/src/types.ts @@ -114,6 +114,17 @@ export interface OsInfo { arch: string; } +/** + * Per-handler post-processing status in the DAG (issue #1147). Keys in + * `RunState.handlerStatus` are handler IDs (e.g. "pp-atif", "pp-taxonomy"). + */ +export interface HandlerRunStatus { + status: "queued" | "processing" | "done" | "failed"; + version?: number; + updatedAt?: string; + error?: string; +} + /** * Per-attempt mutable state (nested under `Run.run` in API responses since * migration 014). Fields here change as a single attempt progresses; fields @@ -148,6 +159,14 @@ export interface RunState { resumedAt?: string; postProcessorVersion?: number; postProcessorStatus?: "queued" | "processing" | "done" | "failed"; + /** + * Per-handler post-processing status keyed by handler ID (e.g. "pp-atif", + * "pp-taxonomy"). Drives the aggregate "Enriched" badge so it only flips to + * done once every handler in the DAG is terminal — not after the first + * handler (pp-atif) completes. The legacy `postProcessorStatus` scalar only + * tracks pp-atif and is kept for backward compatibility. + */ + handlerStatus?: Record; } export interface Run { From 84f6e7b5ba8da87f4caafa034ff603a5ccfcc268 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 17:31:27 -0700 Subject: [PATCH 10/17] fix(judge): unify trajectory navigation into one source-agnostic tool set (#1156 P10) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes a non-deterministic false negative on trajectory observations (e.g. `dependency_added_then_removed`): pp-taxonomy fed the judge the entire ~86 KB ATIF as ONE opaque synthetic tool call, so `read_tool_outputs`' 2000-char preview only ever showed the ~26 KB system prompt and the mid-run npm calls were buried tens of KB deep — the LLM found them only by luck. Replace the opaque blob + the two legacy tools (`read_tool_outputs`/ `get_tool_output`) with ONE normalized `AgentTrajectory` model and a unified, source-agnostic tool set used by BOTH gates (HAR `ToolCall[]`) and observations (parsed ATIF): - new `agent-trajectory.ts`: `fromToolCalls`/`fromAtif` normalizers + five `defineTool`s — two symmetric pairs (definitions: `list_agent_tools`/ `get_agent_tools`; calls: `list_agent_tool_calls`/`get_agent_tool_calls`) plus `get_trajectory_overview`. Outputs larger than INLINE_LIMIT (16 KB) spill to a per-evaluation temp dir created OUTSIDE the workspace and are returned as an absolute `responseFile`/`descriptionFile` the judge reads with `read_file`. - `index.ts`: build the trajectory at the `/evaluate` boundary (ATIF wins as superset; tool-calls otherwise); non-fatal on load failure. - `judge-strategies.ts`: thread `trajectory` through both strategies; extend `createFileTools` with `extraReadRoots` (read_file/file_exists may read the out-of-workspace spill dir; list_directory/search_files stay workspace-only); create+cleanup the spill dir in `runCopilotSession`; swap TOOL_OUTPUTS_GUIDANCE → TRAJECTORY_GUIDANCE. Gate-parity guard preserved (`trajectory.calls.length > 0`). - `judge-agent.ts`: `EvaluationInput.toolCalls` → `trajectory?`. Tests: new `agent-trajectory.test.ts` (normalizers, 5 tools, out-of-workspace spill, strict-superset-over-HAR contract); `judge-strategies.test.ts` (isWithinAnyRoot + extraReadRoots + new guidance/tool names); `observation-trajectory.integration.test.ts` rewritten to pass `fromAtif(...)`; `judge-evaluation.integration.test.ts` extended with a gate clear-PASS/clear-FAIL regression guard built from `fromToolCalls(...)`. Verified live on the containerized stack: the same observation that flipped non-deterministically now resolves passed:true 5/5 WITH the ATIF and false 3/3 WITHOUT it, citing `npm install left-pad` / `npm uninstall left-pad`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- apps/judge/src/agent-trajectory.test.ts | 446 ++++++++++++++ apps/judge/src/agent-trajectory.ts | 544 ++++++++++++++++++ apps/judge/src/index.ts | 53 +- apps/judge/src/judge-agent.ts | 10 +- .../src/judge-evaluation.integration.test.ts | 93 +++ apps/judge/src/judge-strategies.test.ts | 204 ++++--- apps/judge/src/judge-strategies.ts | 256 ++++----- ...observation-trajectory.integration.test.ts | 120 +++- 8 files changed, 1461 insertions(+), 265 deletions(-) create mode 100644 apps/judge/src/agent-trajectory.test.ts create mode 100644 apps/judge/src/agent-trajectory.ts diff --git a/apps/judge/src/agent-trajectory.test.ts b/apps/judge/src/agent-trajectory.test.ts new file mode 100644 index 000000000..78252d88f --- /dev/null +++ b/apps/judge/src/agent-trajectory.test.ts @@ -0,0 +1,446 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/** + * Unit tests for the source-agnostic agent-trajectory model (issue #1156, P10). + * + * Covers both normalizers (`fromAtif` for the observation/ATIF path, `fromToolCalls` + * for the gate/HAR path), the five unified navigation tools, the out-of-workspace + * spill behaviour for oversized outputs, and the strict-superset contract that the + * new tools expose everything the removed `read_tool_outputs`/`get_tool_output` + * pair did. + */ +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import { mkdtempSync, mkdirSync, rmSync, existsSync, readFileSync } from "fs"; +import { tmpdir } from "os"; +import { join, sep } from "path"; +import type { ToolCall } from "shared"; +import { + fromAtif, + fromToolCalls, + createTrajectoryTools, + summarizeArgs, + safeId, + previewOf, + INLINE_LIMIT, + type AgentTrajectory, +} from "./agent-trajectory.js"; + +const stubInvocation = { + sessionId: "test-session", + toolCallId: "test-call", + toolName: "test-tool", + arguments: {}, +}; + +/** Invoke a trajectory tool's handler by name. */ +async function call( + tools: ReturnType, + name: string, + args: Record = {} +): Promise { + const tool = tools.find((t) => t.name === name); + if (!tool?.handler) throw new Error(`tool ${name} not found`); + return (tool.handler as (a: unknown, b: unknown) => unknown)(args, stubInvocation); +} + +/** + * Realistic ATIF blob: system + user step, then agent steps that install, + * use, and uninstall `left-pad`. Mirrors the real ATIF shape pp-taxonomy + * downloads (steps carry their own tool_calls AND the matching + * observation.results, paired by source_call_id === tool_call_id). + */ +const REALISTIC_ATIF = { + schema_version: "atif/1", + session_id: "fixture", + agent: { + name: "copilot", + version: "1.0.0", + model_name: "claude-opus-4.6", + tool_definitions: [ + { + type: "function", + function: { + name: "bash", + description: "Run a shell command.", + parameters: { type: "object", properties: { command: { type: "string" } } }, + }, + }, + { + type: "function", + function: { name: "view", description: "View a file.", parameters: { type: "object" } }, + }, + { + type: "function", + function: { name: "edit", description: "Edit a file." }, + }, + // Nameless definition — must be dropped. + { type: "function", function: { description: "no name" } }, + ], + }, + steps: [ + { step_id: 1, source: "system" }, + { step_id: 2, source: "user" }, + { + step_id: 3, + source: "agent", + model_name: "claude-opus-4.6", + reasoning_content: "Add left-pad.", + tool_calls: [{ tool_call_id: "call-1", function_name: "bash", arguments: { command: "npm install left-pad" } }], + observation: { results: [{ source_call_id: "call-1", content: "added 1 package\n+ left-pad@1.3.0" }] }, + }, + { + step_id: 4, + source: "agent", + tool_calls: [{ tool_call_id: "call-2", function_name: "edit", arguments: { path: "index.js" } }], + // ContentPart[] form of observation content. + observation: { + results: [ + { source_call_id: "call-2", content: [{ type: "text", text: "edited " }, { type: "text", text: "index.js" }] }, + ], + }, + }, + { + step_id: 5, + source: "agent", + reasoning_content: "Remove left-pad.", + tool_calls: [{ tool_call_id: "call-3", function_name: "bash", arguments: { command: "npm uninstall left-pad" } }], + observation: { results: [{ source_call_id: "call-3", content: "removed 1 package" }] }, + }, + ], + final_metrics: { steps: 5 }, +}; + +describe("fromAtif", () => { + const traj = fromAtif(REALISTIC_ATIF); + + it("flattens only agent tool calls, keyed by tool_call_id", () => { + expect(traj.source).toBe("atif"); + expect(traj.calls.map((c) => c.toolCallId)).toEqual(["call-1", "call-2", "call-3"]); + expect(traj.byId.get("call-1")!.name).toBe("bash"); + expect(traj.byId.get("call-1")!.summary).toBe("npm install left-pad"); + }); + + it("excludes system/user steps from calls but records them in steps overview", () => { + expect(traj.steps?.map((s) => s.source)).toEqual(["system", "user", "agent", "agent", "agent"]); + // No call carries the system prompt text. + expect(traj.calls.every((c) => c.step && c.step >= 3)).toBe(true); + }); + + it("matches each call's response from the same-step observation result", () => { + expect(traj.byId.get("call-1")!.response).toContain("left-pad@1.3.0"); + expect(traj.byId.get("call-3")!.response).toBe("removed 1 package"); + }); + + it("coerces ContentPart[] observation content into joined text", () => { + expect(traj.byId.get("call-2")!.response).toBe("edited index.js"); + }); + + it("captures per-step reasoning on the calls", () => { + expect(traj.byId.get("call-1")!.reasoning).toBe("Add left-pad."); + expect(traj.byId.get("call-3")!.reasoning).toBe("Remove left-pad."); + }); + + it("reads agent + model metadata", () => { + expect(traj.agent).toBe("copilot"); + expect(traj.model).toBe("claude-opus-4.6"); + expect(traj.finalMetrics).toEqual({ steps: 5 }); + }); + + it("populates tool definitions from agent.tool_definitions, dropping nameless ones", () => { + expect(traj.toolDefinitions.map((d) => d.name)).toEqual(["bash", "view", "edit"]); + expect(traj.toolDefsByName.get("bash")!.description).toBe("Run a shell command."); + }); + + it("parses a JSON string input", () => { + const fromString = fromAtif(JSON.stringify(REALISTIC_ATIF)); + expect(fromString.calls.map((c) => c.toolCallId)).toEqual(["call-1", "call-2", "call-3"]); + }); + + it("unwraps a { trajectory: … } wrapper", () => { + const wrapped = fromAtif({ trajectory: REALISTIC_ATIF }); + expect(wrapped.calls).toHaveLength(3); + }); + + it("returns an empty trajectory for garbage / empty input", () => { + expect(fromAtif("not json").calls).toEqual([]); + expect(fromAtif(null).calls).toEqual([]); + expect(fromAtif({}).calls).toEqual([]); + expect(fromAtif(42).calls).toEqual([]); + }); + + it("synthesizes an id for a tool call missing tool_call_id", () => { + const traj2 = fromAtif({ + steps: [ + { + step_id: 7, + source: "agent", + tool_calls: [{ function_name: "bash", arguments: { command: "ls" } }], + }, + ], + }); + expect(traj2.calls).toHaveLength(1); + expect(traj2.calls[0].toolCallId).toBe("atif-7-0"); + }); +}); + +describe("fromToolCalls (HAR / gate path)", () => { + const harCalls: ToolCall[] = [ + { id: "h1", name: "bash", arguments: { command: "npm run build" }, response: "build ok", timestamp: "" }, + { id: "h2", name: "view", arguments: { path: "server.js" }, response: "file contents", timestamp: "" }, + ]; + const traj = fromToolCalls(harCalls); + + it("maps each HAR ToolCall to an AgentToolCall keyed by id", () => { + expect(traj.source).toBe("tool-calls"); + expect(traj.calls.map((c) => c.toolCallId)).toEqual(["h1", "h2"]); + expect(traj.byId.get("h1")!.response).toBe("build ok"); + expect(traj.byId.get("h1")!.summary).toBe("npm run build"); + }); + + it("has no tool catalog (HAR carries no definitions)", () => { + expect(traj.toolDefinitions).toEqual([]); + expect(traj.steps).toBeUndefined(); + }); + + it("synthesizes positional ids when a HAR call has none", () => { + const t = fromToolCalls([{ id: "", name: "bash", arguments: {}, response: "x", timestamp: "" }]); + expect(t.calls[0].toolCallId).toBe("call-0"); + }); + + it("handles undefined input", () => { + expect(fromToolCalls(undefined).calls).toEqual([]); + }); +}); + +describe("createTrajectoryTools — calls pair (list/get)", () => { + let spillRoot: string; + const traj = fromAtif(REALISTIC_ATIF); + + beforeAll(() => { + spillRoot = mkdtempSync(join(tmpdir(), "traj-spill-")); + }); + afterAll(() => rmSync(spillRoot, { recursive: true, force: true })); + + it("list_agent_tool_calls returns a triage index of every call", async () => { + const tools = createTrajectoryTools(traj, spillRoot); + const res = await call(tools, "list_agent_tool_calls"); + expect(res.count).toBe(3); + expect(res.calls.map((c: any) => c.toolCallId)).toEqual(["call-1", "call-2", "call-3"]); + expect(res.calls[0]).toMatchObject({ name: "bash", step: 3, summary: "npm install left-pad" }); + expect(res.calls[0].responseLength).toBeGreaterThan(0); + }); + + it("get_agent_tool_calls returns full details for requested ids and reports notFound", async () => { + const tools = createTrajectoryTools(traj, spillRoot); + const res = await call(tools, "get_agent_tool_calls", { toolCallIds: ["call-1", "nope"] }); + expect(res.notFound).toEqual(["nope"]); + expect(res.calls).toHaveLength(1); + expect(res.calls[0]).toMatchObject({ + toolCallId: "call-1", + name: "bash", + arguments: { command: "npm install left-pad" }, + reasoning: "Add left-pad.", + }); + expect(res.calls[0].response).toContain("left-pad@1.3.0"); + }); + + it("empty trajectory → list returns count 0 with a message", async () => { + const tools = createTrajectoryTools(fromAtif({}), spillRoot); + const res = await call(tools, "list_agent_tool_calls"); + expect(res).toMatchObject({ count: 0, calls: [] }); + expect(res.message).toBeTruthy(); + }); +}); + +describe("createTrajectoryTools — oversized output spills OUTSIDE the workspace", () => { + let parent: string; + let workspace: string; + let spillDir: string; + const bigOutput = "X".repeat(INLINE_LIMIT + 5000); + + // A trajectory whose single call has an output larger than INLINE_LIMIT. + const bigTraj: AgentTrajectory = fromAtif({ + steps: [ + { + step_id: 3, + source: "agent", + tool_calls: [{ tool_call_id: "big-1", function_name: "bash", arguments: { command: "cat huge.log" } }], + observation: { results: [{ source_call_id: "big-1", content: bigOutput }] }, + }, + ], + }); + + beforeAll(() => { + parent = mkdtempSync(join(tmpdir(), "traj-ws-parent-")); + workspace = join(parent, "workspace"); + spillDir = join(parent, "spill"); + mkdirSync(workspace, { recursive: true }); + }); + afterAll(() => rmSync(parent, { recursive: true, force: true })); + + it("writes the full output to a spill file outside the workspace and omits the inline response", async () => { + const tools = createTrajectoryTools(bigTraj, spillDir); + const res = await call(tools, "get_agent_tool_calls", { toolCallIds: ["big-1"] }); + const one = res.calls[0]; + + // No inline blob — only a preview + a file pointer. + expect(one.response).toBeUndefined(); + expect(one.responsePreview).toBeTruthy(); + expect(one.responseLength).toBe(bigOutput.length); + + // The spill file is real, holds the FULL output, and lives OUTSIDE the workspace. + expect(typeof one.responseFile).toBe("string"); + expect(existsSync(one.responseFile)).toBe(true); + expect(readFileSync(one.responseFile, "utf-8")).toBe(bigOutput); + expect(one.responseFile.startsWith(workspace + sep)).toBe(false); + expect(one.responseFile.startsWith(spillDir)).toBe(true); + }); + + it("keeps a small output inline (no spill file)", async () => { + const smallTraj = fromAtif({ + steps: [ + { + step_id: 3, + source: "agent", + tool_calls: [{ tool_call_id: "s1", function_name: "bash", arguments: { command: "echo hi" } }], + observation: { results: [{ source_call_id: "s1", content: "hi" }] }, + }, + ], + }); + const tools = createTrajectoryTools(smallTraj, spillDir); + const res = await call(tools, "get_agent_tool_calls", { toolCallIds: ["s1"] }); + expect(res.calls[0].response).toBe("hi"); + expect(res.calls[0].responseFile).toBeUndefined(); + }); +}); + +describe("createTrajectoryTools — definitions pair (list/get)", () => { + let spillDir: string; + const traj = fromAtif(REALISTIC_ATIF); + + beforeAll(() => { + spillDir = mkdtempSync(join(tmpdir(), "traj-defs-")); + }); + afterAll(() => rmSync(spillDir, { recursive: true, force: true })); + + it("list_agent_tools returns the agent's tool catalog", async () => { + const tools = createTrajectoryTools(traj, spillDir); + const res = await call(tools, "list_agent_tools"); + expect(res.count).toBe(3); + expect(res.tools.map((t: any) => t.name)).toEqual(["bash", "view", "edit"]); + const bash = res.tools.find((t: any) => t.name === "bash"); + expect(bash).toMatchObject({ type: "function", hasParameters: true }); + expect(bash.descriptionLength).toBeGreaterThan(0); + }); + + it("get_agent_tools returns full definition + parameters and reports notFound", async () => { + const tools = createTrajectoryTools(traj, spillDir); + const res = await call(tools, "get_agent_tools", { toolNames: ["bash", "ghost"] }); + expect(res.notFound).toEqual(["ghost"]); + expect(res.tools).toHaveLength(1); + expect(res.tools[0]).toMatchObject({ + name: "bash", + type: "function", + description: "Run a shell command.", + }); + expect(res.tools[0].parameters).toMatchObject({ type: "object" }); + }); + + it("on the HAR/gate path the catalog is empty (count 0)", async () => { + const tools = createTrajectoryTools(fromToolCalls([]), spillDir); + const res = await call(tools, "list_agent_tools"); + expect(res).toMatchObject({ count: 0, tools: [] }); + }); + + it("spills a very long tool description to a descriptionFile outside the workspace", async () => { + const bigDesc = "D".repeat(INLINE_LIMIT + 1000); + const t = fromAtif({ + agent: { tool_definitions: [{ type: "function", function: { name: "huge", description: bigDesc } }] }, + steps: [], + }); + const tools = createTrajectoryTools(t, spillDir); + const res = await call(tools, "get_agent_tools", { toolNames: ["huge"] }); + const def = res.tools[0]; + expect(def.description).toBeUndefined(); + expect(def.descriptionPreview).toBeTruthy(); + expect(existsSync(def.descriptionFile)).toBe(true); + expect(readFileSync(def.descriptionFile, "utf-8")).toBe(bigDesc); + }); +}); + +describe("get_trajectory_overview", () => { + it("summarizes the run without the system-prompt blob", async () => { + const spillDir = mkdtempSync(join(tmpdir(), "traj-ov-")); + try { + const tools = createTrajectoryTools(fromAtif(REALISTIC_ATIF), spillDir); + const res = await call(tools, "get_trajectory_overview"); + expect(res).toMatchObject({ + source: "atif", + agent: "copilot", + model: "claude-opus-4.6", + totalTools: 3, + totalCalls: 3, + }); + expect(res.steps).toHaveLength(5); + } finally { + rmSync(spillDir, { recursive: true, force: true }); + } + }); +}); + +describe("strict-superset contract over the HAR path (replaces read_tool_outputs/get_tool_output)", () => { + // The new tools must expose everything the removed pair did: list every call, + // and return each call's FULL response — for large outputs via a spill file + // whose content equals the full response. + let spillDir: string; + const bigResponse = "Z".repeat(INLINE_LIMIT + 2000); + const harCalls: ToolCall[] = [ + { id: "g1", name: "bash", arguments: { command: "npm run build" }, response: "build succeeded", timestamp: "" }, + { id: "g2", name: "bash", arguments: { command: "npm test" }, response: bigResponse, timestamp: "" }, + ]; + + beforeAll(() => { + spillDir = mkdtempSync(join(tmpdir(), "traj-super-")); + }); + afterAll(() => rmSync(spillDir, { recursive: true, force: true })); + + it("list_agent_tool_calls lists every HAR call (same count + names as read_tool_outputs)", async () => { + const tools = createTrajectoryTools(fromToolCalls(harCalls), spillDir); + const res = await call(tools, "list_agent_tool_calls"); + expect(res.count).toBe(2); + expect(res.calls.map((c: any) => `${c.toolCallId}:${c.name}`)).toEqual(["g1:bash", "g2:bash"]); + }); + + it("get_agent_tool_calls returns the same full response get_tool_output would (small inline)", async () => { + const tools = createTrajectoryTools(fromToolCalls(harCalls), spillDir); + const res = await call(tools, "get_agent_tool_calls", { toolCallIds: ["g1"] }); + expect(res.calls[0].response).toBe("build succeeded"); + }); + + it("a large HAR response is spilled and the file content equals the full response", async () => { + const tools = createTrajectoryTools(fromToolCalls(harCalls), spillDir); + const res = await call(tools, "get_agent_tool_calls", { toolCallIds: ["g2"] }); + expect(res.calls[0].response).toBeUndefined(); + expect(readFileSync(res.calls[0].responseFile, "utf-8")).toBe(bigResponse); + }); +}); + +describe("helpers", () => { + it("summarizeArgs prefers command, then path/file, then pattern/url/summary, else JSON", () => { + expect(summarizeArgs("bash", { command: "ls -la" })).toBe("ls -la"); + expect(summarizeArgs("view", { path: "a/b.ts" })).toBe("a/b.ts"); + expect(summarizeArgs("search", { pattern: "TODO" })).toBe("pattern: TODO"); + expect(summarizeArgs("x", { foo: 1 })).toContain("foo"); + }); + + it("safeId slugifies the id and appends the index", () => { + expect(safeId("call/with:weird*chars", 3)).toBe("call_with_weird_chars-3"); + expect(safeId("", 0)).toBe("call-0"); + }); + + it("previewOf truncates with an ellipsis marker", () => { + expect(previewOf("abcdef", 3)).toBe("abc…"); + expect(previewOf("abc", 10)).toBe("abc"); + }); +}); diff --git a/apps/judge/src/agent-trajectory.ts b/apps/judge/src/agent-trajectory.ts new file mode 100644 index 000000000..29b20a469 --- /dev/null +++ b/apps/judge/src/agent-trajectory.ts @@ -0,0 +1,544 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/** + * Source-agnostic model of the coding agent's captured trajectory, plus the + * read-only judge tools that expose it. Issue #1156 / scope #1125. + * + * The judge is fed the agent's work through one of two channels that are really + * the SAME thing captured at different fidelities: + * + * - GATES send the HAR-extracted `ToolCall[]` (build/test/run output) via + * `toolCallsUrl` → normalized here by {@link fromToolCalls}. + * - OBSERVATIONS (#1156) send the full ATIF trajectory via `atifUrl` → the + * normalized superset, parsed here by {@link fromAtif}. The ATIF additionally + * carries the agent's **tool definitions** (the catalog of tools it had + * available) and per-step reasoning. + * + * Both normalize into ONE in-memory {@link AgentTrajectory}, over which we expose + * ONE unified tool set ({@link createTrajectoryTools}) — two symmetric pairs: + * - DEFINITIONS — `list_agent_tools` / `get_agent_tools` (what tools existed). + * - CALLS — `list_agent_tool_calls` / `get_agent_tool_calls` (what the agent did). + * + * This REPLACES the previous opaque "stuff the 86 KB ATIF into one synthetic tool + * call" hack (apps/judge/src/index.ts) whose 2 000-char preview buried the actual + * tool calls under the ~26 KB system prompt, causing non-deterministic false + * negatives on trajectory-only observations (e.g. `dependency_added_then_removed`). + * + * Large outputs are spilled to a caller-provided directory OUTSIDE the workspace + * so they never pollute the codebase other criteria inspect; the judge reads them + * back with its existing `read_file` (which is granted that dir as an extra root). + */ +import { defineTool } from "@github/copilot-sdk"; +import { mkdirSync, writeFileSync } from "fs"; +import { join } from "path"; +import type { ToolCall } from "shared"; + +/** Max chars of a tool-call response (or tool-definition description) returned + * inline by `get_agent_tool_calls` / `get_agent_tools`. Above this the full text + * is written to the spill dir and returned as a file path instead. */ +export const INLINE_LIMIT = 16_000; + +/** Chars of the response/description retained as an at-a-glance preview. */ +export const PREVIEW_LIMIT = 500; + +/** Chars of the per-call preview surfaced by the list/triage tools. */ +export const LIST_PREVIEW_LIMIT = 200; + +/** One captured invocation the agent made (normalized across HAR and ATIF). */ +export interface AgentToolCall { + /** Stable id — the ATIF `tool_call_id` or HAR `ToolCall.id` (synthesized if absent). */ + toolCallId: string; + /** 1-based ATIF step the call was made in (undefined for the HAR/gate path). */ + step?: number; + /** Tool/function name (e.g. `bash`, `create`, `edit`). */ + name: string; + /** One-line digest of the arguments for triage. */ + summary: string; + /** Raw arguments the agent passed. */ + arguments: Record; + /** Captured output/result text. */ + response: string; + /** First {@link PREVIEW_LIMIT} chars of {@link response}. */ + responsePreview: string; + /** Full length of {@link response} in chars. */ + responseLength: number; + /** Step-level reasoning that preceded the call (ATIF only). */ + reasoning?: string; +} + +/** One entry from the agent's tool catalog (ATIF `agent.tool_definitions`). */ +export interface AgentToolDefinition { + /** Tool name — the identifier callers pass to `get_agent_tools`. */ + name: string; + /** Tool type (usually `"function"`). */ + type: string; + /** Human-readable description the agent was given. */ + description?: string; + /** JSON-schema of the tool's parameters. */ + parameters?: Record; +} + +/** Lightweight overview of one ATIF step (never the raw system-prompt text). */ +export interface StepOverview { + step: number; + source: string; + toolCallCount: number; + reasoningSummary?: string; +} + +/** Normalized, source-agnostic view of the agent's captured trajectory. */ +export interface AgentTrajectory { + source: "tool-calls" | "atif"; + agent?: string; + model?: string; + calls: AgentToolCall[]; + /** Index of {@link calls} by {@link AgentToolCall.toolCallId}. */ + byId: Map; + /** The agent's tool catalog (ATIF only; empty for the HAR/gate path). */ + toolDefinitions: AgentToolDefinition[]; + /** Index of {@link toolDefinitions} by {@link AgentToolDefinition.name}. */ + toolDefsByName: Map; + /** Per-step overview (ATIF only). */ + steps?: StepOverview[]; + /** ATIF `final_metrics`, passed through opaquely. */ + finalMetrics?: unknown; +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +/** First `limit` chars of `text`, with an ellipsis marker when truncated. */ +export function previewOf(text: string, limit: number = PREVIEW_LIMIT): string { + if (text.length <= limit) return text; + return text.slice(0, limit) + "…"; +} + +/** Coerce any value to a readable string (ATIF content can be string|object). */ +function asText(value: unknown): string { + if (value == null) return ""; + if (typeof value === "string") return value; + try { + return JSON.stringify(value); + } catch { + return String(value); + } +} + +/** + * Coerce ATIF observation/message content (string | ContentPart[]) into text. + * ContentPart is `{ type: "text"|"image", text?, source? }`; we keep the text + * parts and note image parts. + */ +function coerceContent(content: unknown): string { + if (content == null) return ""; + if (typeof content === "string") return content; + if (Array.isArray(content)) { + return content + .map((part) => { + if (part && typeof part === "object") { + const p = part as { type?: string; text?: string }; + if (typeof p.text === "string") return p.text; + if (p.type === "image") return "[image]"; + } + return asText(part); + }) + .join(""); + } + return asText(content); +} + +/** One-line digest of a call's arguments for triage in the list tools. */ +export function summarizeArgs(name: string, args: Record): string { + if (!args || typeof args !== "object") return ""; + const a = args as Record; + if (typeof a.command === "string") return a.command.slice(0, 200); + if (typeof a.path === "string") return a.path; + if (typeof a.file === "string") return a.file; + if (typeof a.pattern === "string") return `pattern: ${a.pattern}`; + if (typeof a.url === "string") return a.url; + if (typeof a.summary === "string") return a.summary.slice(0, 200); + const json = asText(a); + return json.length > 120 ? json.slice(0, 120) + "…" : json; +} + +/** Filesystem-safe spill filename derived from an id + its position. */ +export function safeId(id: string, index: number): string { + const slug = (id || "").replace(/[^A-Za-z0-9_-]/g, "_").slice(0, 80); + return `${slug || "call"}-${index}`; +} + +function buildById(calls: AgentToolCall[]): Map { + const byId = new Map(); + for (const call of calls) { + // First write wins; later collisions are unreachable in practice because + // ids are unique (synthesized ids embed the position). + if (!byId.has(call.toolCallId)) byId.set(call.toolCallId, call); + } + return byId; +} + +// --------------------------------------------------------------------------- +// Normalizers +// --------------------------------------------------------------------------- + +/** + * Normalize the HAR-extracted tool calls (the gate path) into an + * {@link AgentTrajectory}. HAR carries no tool catalog or reasoning, so + * `toolDefinitions`/`steps` are empty. + */ +export function fromToolCalls(toolCalls: ToolCall[] | undefined): AgentTrajectory { + const list = Array.isArray(toolCalls) ? toolCalls : []; + const calls: AgentToolCall[] = list.map((tc, i) => { + const response = asText(tc.response ?? ""); + const args = (tc.arguments ?? {}) as Record; + const name = tc.name ?? "unknown"; + return { + toolCallId: tc.id || `call-${i}`, + name, + summary: summarizeArgs(name, args), + arguments: args, + response, + responsePreview: previewOf(response), + responseLength: response.length, + }; + }); + return { + source: "tool-calls", + calls, + byId: buildById(calls), + toolDefinitions: [], + toolDefsByName: new Map(), + }; +} + +interface AtifContentPart { + type?: string; + text?: string; +} +interface AtifToolCall { + tool_call_id?: string; + function_name?: string; + arguments?: Record; +} +interface AtifObservationResult { + source_call_id?: string; + content?: string | AtifContentPart[]; +} +interface AtifStep { + step_id?: number; + source?: string; + model_name?: string; + reasoning_content?: string; + tool_calls?: AtifToolCall[]; + observation?: { results?: AtifObservationResult[] }; +} +interface AtifToolDefinition { + type?: string; + function?: { name?: string; description?: string; parameters?: Record }; +} +interface AtifTrajectory { + agent?: { + name?: string; + version?: string; + model_name?: string; + tool_definitions?: AtifToolDefinition[]; + }; + steps?: AtifStep[]; + final_metrics?: unknown; +} + +/** Parse the raw ATIF blob defensively into an object (handles string input and + * an optional `{ trajectory: … }` wrapper). Returns `{}` on any failure. */ +function parseAtif(input: unknown): AtifTrajectory { + let obj: unknown = input; + if (typeof input === "string") { + try { + obj = JSON.parse(input); + } catch { + return {}; + } + } + if (!obj || typeof obj !== "object") return {}; + const maybeWrapped = obj as { trajectory?: unknown }; + if (maybeWrapped.trajectory && typeof maybeWrapped.trajectory === "object") { + return maybeWrapped.trajectory as AtifTrajectory; + } + return obj as AtifTrajectory; +} + +/** + * Normalize a parsed ATIF trajectory (the observation path) into an + * {@link AgentTrajectory}. Walks the steps: agent steps contribute their + * `tool_calls[]`, each paired with the matching same-step + * `observation.results[]` by `source_call_id === tool_call_id`. System and user + * steps are summarized in `steps` but never contribute calls (and their raw + * message text — including the large system prompt — is never surfaced). + */ +export function fromAtif(input: unknown): AgentTrajectory { + const traj = parseAtif(input); + const agent = traj.agent ?? {}; + const stepsArr = Array.isArray(traj.steps) ? traj.steps : []; + + const calls: AgentToolCall[] = []; + const steps: StepOverview[] = []; + let synth = 0; + let firstAgentModel: string | undefined; + + for (let i = 0; i < stepsArr.length; i++) { + const step = stepsArr[i] ?? {}; + const stepNum = typeof step.step_id === "number" ? step.step_id : i + 1; + const source = typeof step.source === "string" ? step.source : "unknown"; + const reasoning = + typeof step.reasoning_content === "string" && step.reasoning_content.length > 0 + ? step.reasoning_content + : undefined; + const stepCalls = Array.isArray(step.tool_calls) ? step.tool_calls : []; + + if (source === "agent" && !firstAgentModel && typeof step.model_name === "string") { + firstAgentModel = step.model_name; + } + + // Index this step's observation results by the call they answer. + const obsByCallId = new Map(); + const results = step.observation?.results; + if (Array.isArray(results)) { + for (const r of results) { + if (r && typeof r.source_call_id === "string") { + obsByCallId.set(r.source_call_id, coerceContent(r.content)); + } + } + } + + for (const tc of stepCalls) { + const name = typeof tc.function_name === "string" ? tc.function_name : "unknown"; + const args = (tc.arguments ?? {}) as Record; + const toolCallId = + typeof tc.tool_call_id === "string" && tc.tool_call_id.length > 0 + ? tc.tool_call_id + : `atif-${stepNum}-${synth++}`; + const response = obsByCallId.get(toolCallId) ?? ""; + calls.push({ + toolCallId, + step: stepNum, + name, + summary: summarizeArgs(name, args), + arguments: args, + response, + responsePreview: previewOf(response), + responseLength: response.length, + reasoning, + }); + } + + steps.push({ + step: stepNum, + source, + toolCallCount: stepCalls.length, + reasoningSummary: reasoning ? previewOf(reasoning, LIST_PREVIEW_LIMIT) : undefined, + }); + } + + const defs = Array.isArray(agent.tool_definitions) ? agent.tool_definitions : []; + const toolDefinitions: AgentToolDefinition[] = defs + .map((d) => ({ + name: d?.function?.name ?? "", + type: typeof d?.type === "string" ? d.type : "function", + description: d?.function?.description, + parameters: d?.function?.parameters, + })) + .filter((d) => d.name.length > 0); + const toolDefsByName = new Map(toolDefinitions.map((d) => [d.name, d])); + + return { + source: "atif", + agent: typeof agent.name === "string" ? agent.name : undefined, + model: agent.model_name ?? firstAgentModel, + calls, + byId: buildById(calls), + toolDefinitions, + toolDefsByName, + steps, + finalMetrics: traj.final_metrics, + }; +} + +// --------------------------------------------------------------------------- +// Tools +// --------------------------------------------------------------------------- + +function writeSpill(spillDir: string, filename: string, text: string): string { + mkdirSync(spillDir, { recursive: true }); + const file = join(spillDir, `${filename}.txt`); + writeFileSync(file, text, "utf-8"); + return file; +} + +/** + * Build the four read-only trajectory tools over a normalized + * {@link AgentTrajectory}. All are `skipPermission: true` because the judge runs + * headless (no TUI to answer permission prompts; see scope #1125). + * + * @param trajectory normalized agent trajectory + * @param spillDir directory OUTSIDE the workspace where oversized outputs are + * written so they never pollute the codebase under evaluation. The judge reads + * them back via `read_file` (which is granted this dir as an extra read root). + */ +export function createTrajectoryTools(trajectory: AgentTrajectory, spillDir: string) { + const listAgentTools = defineTool("list_agent_tools", { + description: + "List the tools the coding agent had available during this run (its tool catalog / definitions). Returns each tool's name, type, a short description preview and whether it takes parameters. Use get_agent_tools(toolNames) to fetch a tool's full description and parameter schema. Empty when no tool catalog was captured.", + skipPermission: true, + parameters: { type: "object", properties: {}, required: [] }, + handler: async () => { + const tools = trajectory.toolDefinitions.map((d) => { + const description = d.description ?? ""; + return { + name: d.name, + type: d.type, + descriptionPreview: previewOf(description, LIST_PREVIEW_LIMIT), + descriptionLength: description.length, + hasParameters: !!d.parameters && Object.keys(d.parameters).length > 0, + }; + }); + return { count: tools.length, tools }; + }, + }); + + const getAgentTools = defineTool("get_agent_tools", { + description: + "Return the full definition (type, description and parameter JSON-schema) of the named agent tools, as listed by list_agent_tools. Pass a list of tool names. A very long description is written to a file (descriptionFile) you can open with read_file; the parameter schema is always returned inline. Unknown names are returned in notFound.", + skipPermission: true, + parameters: { + type: "object", + properties: { + toolNames: { + type: "array", + items: { type: "string" }, + description: "Tool names to fetch (from list_agent_tools).", + }, + }, + required: ["toolNames"], + }, + handler: async (args: { toolNames?: string[] }) => { + const names = Array.isArray(args?.toolNames) ? args.toolNames : []; + const tools: Record[] = []; + const notFound: string[] = []; + for (let i = 0; i < names.length; i++) { + const name = names[i]; + const def = trajectory.toolDefsByName.get(name); + if (!def) { + notFound.push(name); + continue; + } + const description = def.description ?? ""; + const base: Record = { + name: def.name, + type: def.type, + parameters: def.parameters, + descriptionLength: description.length, + }; + if (description.length <= INLINE_LIMIT) { + base.description = description; + } else { + base.descriptionPreview = previewOf(description); + base.descriptionFile = writeSpill(spillDir, safeId(`tool-${def.name}`, i), description); + } + tools.push(base); + } + return { tools, notFound }; + }, + }); + + const listAgentToolCalls = defineTool("list_agent_tool_calls", { + description: + "List the tool calls the coding agent made during this run (its trajectory of invocations). Returns each call's toolCallId, step, name, a one-line argument summary, a short response preview and the full response length. Use get_agent_tool_calls(toolCallIds) to fetch full arguments and output for the calls you care about. Consult these to determine what the agent actually did and whether a command succeeded.", + skipPermission: true, + parameters: { type: "object", properties: {}, required: [] }, + handler: async () => { + if (trajectory.calls.length === 0) { + return { count: 0, calls: [], message: "No tool calls were captured for this run." }; + } + const calls = trajectory.calls.map((c) => ({ + toolCallId: c.toolCallId, + step: c.step, + name: c.name, + summary: c.summary, + responsePreview: c.responsePreview.slice(0, LIST_PREVIEW_LIMIT), + responseLength: c.responseLength, + })); + return { count: calls.length, calls }; + }, + }); + + const getAgentToolCalls = defineTool("get_agent_tool_calls", { + description: + "Return the full arguments and captured output for a list of tool calls, identified by their toolCallId (as listed by list_agent_tool_calls). A very long output is written to a file (responseFile) you can open with read_file; shorter outputs are returned inline. Unknown ids are returned in notFound.", + skipPermission: true, + parameters: { + type: "object", + properties: { + toolCallIds: { + type: "array", + items: { type: "string" }, + description: "Tool call ids to fetch (from list_agent_tool_calls).", + }, + }, + required: ["toolCallIds"], + }, + handler: async (args: { toolCallIds?: string[] }) => { + const ids = Array.isArray(args?.toolCallIds) ? args.toolCallIds : []; + const calls: Record[] = []; + const notFound: string[] = []; + for (const id of ids) { + const c = trajectory.byId.get(id); + if (!c) { + notFound.push(id); + continue; + } + const base: Record = { + toolCallId: c.toolCallId, + step: c.step, + name: c.name, + arguments: c.arguments, + reasoning: c.reasoning, + responseLength: c.responseLength, + }; + if (c.responseLength <= INLINE_LIMIT) { + base.response = c.response; + } else { + const pos = trajectory.calls.indexOf(c); + base.responsePreview = c.responsePreview; + base.responseFile = writeSpill(spillDir, safeId(c.toolCallId, pos), c.response); + } + calls.push(base); + } + return { calls, notFound }; + }, + }); + + const getTrajectoryOverview = defineTool("get_trajectory_overview", { + description: + "Return a high-level overview of the agent's run: source, agent, model, the number of available tools and tool calls, a per-step summary (without the raw system prompt) and final metrics. Use this to orient before drilling into specific tools or calls.", + skipPermission: true, + parameters: { type: "object", properties: {}, required: [] }, + handler: async () => ({ + source: trajectory.source, + agent: trajectory.agent, + model: trajectory.model, + totalTools: trajectory.toolDefinitions.length, + totalCalls: trajectory.calls.length, + steps: trajectory.steps, + finalMetrics: trajectory.finalMetrics, + }), + }); + + return [ + listAgentTools, + getAgentTools, + listAgentToolCalls, + getAgentToolCalls, + getTrajectoryOverview, + ]; +} diff --git a/apps/judge/src/index.ts b/apps/judge/src/index.ts index 0dea452cd..fd5a42e38 100644 --- a/apps/judge/src/index.ts +++ b/apps/judge/src/index.ts @@ -5,6 +5,7 @@ import express, { Request, Response, NextFunction } from "express"; import dotenv from "dotenv"; import { evaluateWorkspace } from "./judge-agent.js"; import { verifyCopilotProtocol } from "./protocol-check.js"; +import { type AgentTrajectory, fromAtif, fromToolCalls } from "./agent-trajectory.js"; import { BlobStorage, RedisLogPublisher } from "shared"; import { mkdtempSync, rmSync } from "fs"; import { tmpdir } from "os"; @@ -89,44 +90,38 @@ app.post( console.log(`[judge] Snapshot extracted to ${workDir}`); - // Download this iteration's captured tool calls/outputs (build/test/run - // output), if provided. Failures are non-fatal — the judge can still - // evaluate the workspace files. - let toolCalls: import("shared").ToolCall[] = []; - if (toolCallsUrl && typeof toolCallsUrl === "string") { + // Build the agent's captured trajectory from whichever channel was + // provided, normalized into ONE source-agnostic model the judge navigates + // with the unified trajectory tools (list/get_agent_tool_calls, + // list/get_agent_tools, get_trajectory_overview). Gates send the HAR + // ToolCall[] via toolCallsUrl; observations (#1156) send the full ATIF via + // atifUrl — the normalized superset that also carries the agent's tool + // catalog and per-step reasoning. ATIF wins if both ever arrive (superset). + // Failures are non-fatal — the judge can still evaluate the workspace + // files. See apps/judge/src/agent-trajectory.ts. + let trajectory: AgentTrajectory | undefined; + if (atifUrl && typeof atifUrl === "string") { try { - toolCalls = await blobStorage.getToolCalls(toolCallsUrl); + const atif = await blobStorage.downloadJson(atifUrl); + trajectory = fromAtif(atif); console.log( - `[judge] Loaded ${toolCalls.length} tool call(s) for gate '${gate ?? "select"}'`, + `[judge] Loaded ATIF trajectory from ${atifUrl}: ${trajectory.calls.length} call(s), ${trajectory.toolDefinitions.length} tool definition(s)`, ); } catch (err) { console.warn( - `[judge] Failed to load tool calls from ${toolCallsUrl}: ${err instanceof Error ? err.message : String(err)}`, + `[judge] Failed to load ATIF from ${atifUrl}: ${err instanceof Error ? err.message : String(err)}`, ); } - } - - // Load the ATIF trajectory (normalized superset of tool calls) when - // provided — used by pp-taxonomy to evaluate observations against the - // full agent trajectory, not just the end-state snapshot. Exposed to the - // judge as a synthetic tool output so trajectory-only observations (e.g. - // a dependency added then removed mid-run) are detectable. Issue #1156. - if (atifUrl && typeof atifUrl === "string") { + } else if (toolCallsUrl && typeof toolCallsUrl === "string") { try { - const atif = await blobStorage.downloadJson(atifUrl); - toolCalls = [ - ...toolCalls, - { - id: "atif-trajectory", - name: "agent_trajectory_atif", - arguments: {}, - response: typeof atif === "string" ? atif : JSON.stringify(atif), - }, - ]; - console.log(`[judge] Loaded ATIF trajectory from ${atifUrl}`); + const toolCalls = await blobStorage.getToolCalls(toolCallsUrl); + trajectory = fromToolCalls(toolCalls); + console.log( + `[judge] Loaded ${trajectory.calls.length} tool call(s) for gate '${gate ?? "select"}'`, + ); } catch (err) { console.warn( - `[judge] Failed to load ATIF from ${atifUrl}: ${err instanceof Error ? err.message : String(err)}`, + `[judge] Failed to load tool calls from ${toolCallsUrl}: ${err instanceof Error ? err.message : String(err)}`, ); } } @@ -153,7 +148,7 @@ app.post( personaInstructions, onProgress, gate, - toolCalls, + trajectory, }); const elapsed = Date.now() - startTime; diff --git a/apps/judge/src/judge-agent.ts b/apps/judge/src/judge-agent.ts index a6c3672c0..19d74abbc 100644 --- a/apps/judge/src/judge-agent.ts +++ b/apps/judge/src/judge-agent.ts @@ -8,10 +8,10 @@ import { DetailedEvaluationResult, DependencyGraph, GateId, - ToolCall, } from "shared"; import { getCriteriaProvider } from "shared/criteria-provider-factory"; import { createJudgeStrategy } from "./judge-strategies.js"; +import type { AgentTrajectory } from "./agent-trajectory.js"; import { FeedbackGenerator } from "./feedback-generator.js"; export interface EvaluationInput { @@ -23,8 +23,10 @@ export interface EvaluationInput { onProgress?: (result: CriterionResult) => void; /** Which gate is being evaluated. Defaults to select. */ gate?: GateId; - /** This iteration's captured tool calls/outputs, exposed to the judge via read_tool_outputs. */ - toolCalls?: ToolCall[]; + /** The coding agent's captured trajectory (normalized from the HAR tool calls + * for gates, or the ATIF for observations), exposed to the judge via the + * unified trajectory tools. */ + trajectory?: AgentTrajectory; } export interface EvaluationResult { @@ -102,7 +104,7 @@ export async function evaluateWorkspace( personaInstructions: input.personaInstructions, onProgress: input.onProgress, gate: input.gate, - toolCalls: input.toolCalls, + trajectory: input.trajectory, }); console.log( diff --git a/apps/judge/src/judge-evaluation.integration.test.ts b/apps/judge/src/judge-evaluation.integration.test.ts index af96b1e42..8b5f7237f 100644 --- a/apps/judge/src/judge-evaluation.integration.test.ts +++ b/apps/judge/src/judge-evaluation.integration.test.ts @@ -24,7 +24,9 @@ import { mkdtempSync, writeFileSync, rmSync, realpathSync } from "fs"; import { tmpdir } from "os"; import { join } from "path"; import { resetCriteriaProvider } from "shared/criteria-provider-factory"; +import type { ToolCall } from "shared"; import { evaluateWorkspace } from "./judge-agent.js"; +import { fromToolCalls } from "./agent-trajectory.js"; const hasToken = !!process.env.GITHUB_TOKEN; @@ -64,6 +66,26 @@ describe("judge end-to-end evaluation (integration)", () => { ].join("\n") ); + // Gate-regression criterion (issue #1156 / P10): its verdict is derivable + // ONLY from the agent's captured tool output, NOT from the workspace files, + // so the judge must navigate the trajectory (list_agent_tool_calls / + // get_agent_tool_calls over a `fromToolCalls`-derived trajectory — the gate + // path) to decide. Guards that normalizing the HAR ToolCall[] and exposing it + // through the new tools yields the same boolean verdict the legacy + // read_tool_outputs/get_tool_output pair would. + writeFileSync( + join(criteriaDir, "build_succeeded.yaml"), + [ + "id: build_succeeded", + "prompt: |", + " The agent ran the project's build command and it completed", + " successfully (exit code 0, no compiler errors). This is only derivable", + " from the agent's captured tool calls — inspect them to confirm the", + " build command's output reports success.", + "", + ].join("\n") + ); + setEnv("CRITERIA_DIR", criteriaDir); setEnv("JUDGE_STRATEGY", process.env.JUDGE_STRATEGY || "independent"); setEnv("JUDGE_MODEL", JUDGE_MODEL); @@ -120,6 +142,77 @@ describe("judge end-to-end evaluation (integration)", () => { } ); + it.skipIf(!hasToken)( + "gate regression: a clear-PASS build gate built from a tool-call trajectory passes", + { timeout: 300_000 }, + async () => { + // The gate path: normalize the HAR ToolCall[] via fromToolCalls and let the + // judge navigate it with the unified trajectory tools. The build output says + // success, so the gate must pass. Verdict comes from the trajectory, not the + // workspace (which has no build log). + const passTrajectory = fromToolCalls([ + { + id: "tc-build-pass", + name: "bash", + arguments: { command: "npm run build" }, + response: + "> demo@1.0.0 build\n> tsc\n\nBuild succeeded. 0 errors, 0 warnings.\nExit code: 0", + timestamp: "", + } as ToolCall, + ]); + + const result = await evaluateWorkspace({ + workspacePath: workspaceDir, + criteria: ["build_succeeded"], + conversationHistory: [], + trajectory: passTrajectory, + }); + + const gate = result.criteriaResults.find((r) => r.criterionId === "build_succeeded"); + expect(gate, "build_succeeded result missing").toBeTruthy(); + expect(gate!.evaluated).toBe(true); + expect( + gate!.passed, + `expected clear-PASS build gate to pass from tool output. Feedback: ${gate!.feedback}` + ).toBe(true); + } + ); + + it.skipIf(!hasToken)( + "gate regression: a clear-FAIL build gate built from a tool-call trajectory fails", + { timeout: 300_000 }, + async () => { + // Same gate, opposite evidence: the build output reports a compiler error and + // a non-zero exit, so the gate must fail. Proves the trajectory drives the + // boolean verdict in both directions (no false positives). + const failTrajectory = fromToolCalls([ + { + id: "tc-build-fail", + name: "bash", + arguments: { command: "npm run build" }, + response: + "> demo@1.0.0 build\n> tsc\n\nsrc/index.ts(3,7): error TS2304: Cannot find name 'foo'.\n\nBuild failed. 1 error.\nExit code: 1", + timestamp: "", + } as ToolCall, + ]); + + const result = await evaluateWorkspace({ + workspacePath: workspaceDir, + criteria: ["build_succeeded"], + conversationHistory: [], + trajectory: failTrajectory, + }); + + const gate = result.criteriaResults.find((r) => r.criterionId === "build_succeeded"); + expect(gate, "build_succeeded result missing").toBeTruthy(); + expect(gate!.evaluated).toBe(true); + expect( + gate!.passed, + `expected clear-FAIL build gate to fail from tool output. Feedback: ${gate!.feedback}` + ).toBe(false); + } + ); + it("logs a skip note when GITHUB_TOKEN is absent (no real evaluation possible)", () => { if (!hasToken) { console.warn( diff --git a/apps/judge/src/judge-strategies.test.ts b/apps/judge/src/judge-strategies.test.ts index 5cf0dfcdc..4162fa6ef 100644 --- a/apps/judge/src/judge-strategies.test.ts +++ b/apps/judge/src/judge-strategies.test.ts @@ -8,8 +8,9 @@ import { join } from "path"; import { IndependentStrategy, BundledStrategy, - TOOL_OUTPUTS_GUIDANCE, + TRAJECTORY_GUIDANCE, isWithinWorkspace, + isWithinAnyRoot, JUDGE_AVAILABLE_TOOLS, JUDGE_EXCLUDED_TOOLS, } from "./judge-strategies.js"; @@ -20,17 +21,17 @@ import { * Copilot session. */ class TestableStrategy extends IndependentStrategy { - publicCreateFileTools(workspacePath: string) { - return this.createFileTools(workspacePath); - } - publicCreateToolOutputTools(toolCalls: any[]) { - return this.createToolOutputTools(toolCalls); + publicCreateFileTools( + workspacePath: string, + opts?: { extraReadRoots?: string[] } + ) { + return this.createFileTools(workspacePath, opts); } publicBuildSessionConfig(tools: any[], systemPrompt: string) { return this.buildSessionConfig(tools, systemPrompt); } - publicBuildSystemPrompt(hasToolOutputs: boolean) { - return (this as any).buildSystemPrompt(undefined, hasToolOutputs); + publicBuildSystemPrompt(hasTrajectory: boolean) { + return (this as any).buildSystemPrompt(undefined, hasTrajectory); } publicBuildUserPrompt( criterion: { id: string; prompt: string }, @@ -40,9 +41,12 @@ class TestableStrategy extends IndependentStrategy { } } -function toolMap(workspacePath: string) { +function toolMap( + workspacePath: string, + opts?: { extraReadRoots?: string[] } +) { const strategy = new TestableStrategy("test-model"); - const tools = strategy.publicCreateFileTools(workspacePath); + const tools = strategy.publicCreateFileTools(workspacePath, opts); return new Map(tools.map((t) => [t.name, t])); } @@ -56,9 +60,10 @@ const stubInvocation = { async function callTool( workspacePath: string, name: string, - args: Record + args: Record, + opts?: { extraReadRoots?: string[] } ): Promise { - const tool = toolMap(workspacePath).get(name); + const tool = toolMap(workspacePath, opts).get(name); if (!tool?.handler) throw new Error(`tool ${name} has no handler`); return (tool.handler as (a: unknown, b: unknown) => unknown)(args, stubInvocation); } @@ -86,37 +91,6 @@ describe("judge file tools", () => { }); }); -describe("judge tool-output tools (issue #1125)", () => { - const strategy = new TestableStrategy("test-model"); - const toolCalls = [ - { - id: "1", - name: "bash", - arguments: { command: "npm run build" }, - response: "build ok", - timestamp: "", - }, - ]; - const tools = strategy.publicCreateToolOutputTools(toolCalls); - - it("exposes read_tool_outputs and get_tool_output", () => { - const names = tools.map((t) => t.name).sort(); - expect(names).toEqual(["get_tool_output", "read_tool_outputs"]); - }); - - // Regression guard for scope #1125: headless, any tool without skipPermission - // is denied at execution time ("could not request permission from user"). When - // this affected read_tool_outputs/get_tool_output, the judge could never see - // the coding agent's captured build/test output and wrongly demanded on-disk - // proof files, forcing the build gate to loop for many iterations. - it("marks every tool-output tool to skip the permission prompt", () => { - expect(tools.length).toBeGreaterThan(0); - for (const tool of tools) { - expect(tool.skipPermission, `${tool.name} must skip the permission prompt`).toBe(true); - } - }); -}); - describe("isWithinWorkspace", () => { it("accepts the root itself and nested paths", () => { expect(isWithinWorkspace("/tmp/ws", "/tmp/ws")).toBe(true); @@ -137,6 +111,21 @@ describe("isWithinWorkspace", () => { }); }); +describe("isWithinAnyRoot", () => { + it("accepts a path inside any of the roots", () => { + expect(isWithinAnyRoot(["/tmp/ws", "/tmp/spill"], "/tmp/ws/src/a.ts")).toBe(true); + expect(isWithinAnyRoot(["/tmp/ws", "/tmp/spill"], "/tmp/spill/out.txt")).toBe(true); + expect(isWithinAnyRoot(["/tmp/ws", "/tmp/spill"], "/tmp/spill")).toBe(true); + }); + + it("rejects a path outside every root (guard preserved per-root)", () => { + expect(isWithinAnyRoot(["/tmp/ws", "/tmp/spill"], "/tmp/ws2/secret")).toBe(false); + expect(isWithinAnyRoot(["/tmp/ws", "/tmp/spill"], "/etc/passwd")).toBe(false); + // sibling-prefix of an allowed root must not slip through + expect(isWithinAnyRoot(["/tmp/ws", "/tmp/spill"], "/tmp/spill2/x")).toBe(false); + }); +}); + describe("judge file tool handlers (workspace scoping)", () => { let parent: string; let workspace: string; @@ -201,6 +190,85 @@ describe("judge file tool handlers (workspace scoping)", () => { }); }); +describe("judge file tools — extraReadRoots (P10 trajectory spill dir)", () => { + // The trajectory tools spill oversized outputs to a dir OUTSIDE the workspace + // and return its absolute path; read_file/file_exists must be able to open it + // (via extraReadRoots) WITHOUT widening list_directory/search_files, so the + // codebase view other criteria inspect is never polluted by spill files. + let parent: string; + let workspace: string; + let spill: string; + + beforeAll(() => { + parent = mkdtempSync(join(tmpdir(), "judge-spill-")); + workspace = join(parent, "ws"); + spill = join(parent, "spill"); + mkdirSync(workspace, { recursive: true }); + mkdirSync(spill, { recursive: true }); + writeFileSync(join(workspace, "inside.txt"), "in-workspace"); + writeFileSync(join(spill, "big-output.txt"), "SPILLED OUTPUT"); + }); + + afterAll(() => { + rmSync(parent, { recursive: true, force: true }); + }); + + it("read_file opens an absolute path under an extra read root", async () => { + const result = (await callTool( + workspace, + "read_file", + { path: join(spill, "big-output.txt") }, + { extraReadRoots: [spill] } + )) as { content?: string; error?: string }; + expect(result.error).toBeUndefined(); + expect(result.content).toBe("SPILLED OUTPUT"); + }); + + it("file_exists confirms a file under an extra read root", async () => { + const result = (await callTool( + workspace, + "file_exists", + { path: join(spill, "big-output.txt") }, + { extraReadRoots: [spill] } + )) as { exists?: boolean; error?: string }; + expect(result.error).toBeUndefined(); + expect(result.exists).toBe(true); + }); + + it("read_file still rejects an absolute path outside the workspace AND the extra roots", async () => { + const outside = join(parent, "other", "secret.txt"); + mkdirSync(join(parent, "other"), { recursive: true }); + writeFileSync(outside, "SECRET"); + const result = (await callTool( + workspace, + "read_file", + { path: outside }, + { extraReadRoots: [spill] } + )) as { content?: string; error?: string }; + expect(result.error).toBe("Path traversal not allowed"); + expect(result.content).toBeUndefined(); + }); + + it("read_file without extraReadRoots rejects the spill path (default behaviour unchanged)", async () => { + const result = (await callTool(workspace, "read_file", { + path: join(spill, "big-output.txt"), + })) as { content?: string; error?: string }; + expect(result.error).toBe("Path traversal not allowed"); + }); + + it("list_directory stays workspace-only — it cannot reveal an extra read root's contents", async () => { + const result = (await callTool( + workspace, + "list_directory", + { path: spill }, + { extraReadRoots: [spill] } + )) as { entries?: { name: string }[]; error?: string }; + // Whether it errors or resolves the absolute path back under the workspace, + // it must never surface the spilled file — list_directory ignores extraReadRoots. + expect(result.entries?.some((e) => e.name === "big-output.txt") ?? false).toBe(false); + }); +}); + describe("judge session tool restriction (scope #1117)", () => { const strategy = new TestableStrategy("test-model"); const config = strategy.publicBuildSessionConfig([], "system prompt"); @@ -243,8 +311,8 @@ describe("judge session tool restriction (scope #1117)", () => { * without running a real Copilot session. */ class TestableBundledStrategy extends BundledStrategy { - publicBuildSystemPrompt(hasToolOutputs: boolean) { - return (this as any).buildSystemPrompt(undefined, hasToolOutputs); + publicBuildSystemPrompt(hasTrajectory: boolean) { + return (this as any).buildSystemPrompt(undefined, hasTrajectory); } publicBuildUserPrompt( criteria: { id: string; prompt: string }[] = [ @@ -256,51 +324,59 @@ class TestableBundledStrategy extends BundledStrategy { } } -describe("judge tool-outputs guidance (issue #1125)", () => { +describe("judge trajectory guidance (issue #1125 / #1156)", () => { // The headless judge cannot run commands; it must decide from the codebase - // plus the coding agent's captured tool outputs. The guidance must be generic + // plus the coding agent's captured trajectory. The guidance must be generic // (not build/test specific) and must tell the judge to treat captured output // as authoritative instead of demanding the agent redo or re-prove the work. it("is generic, not tied to any one command type", () => { - const g = TOOL_OUTPUTS_GUIDANCE.toLowerCase(); + const g = TRAJECTORY_GUIDANCE.toLowerCase(); expect(g).not.toMatch(/build\.log|build_proof|\bnpm run build\b/); }); it("tells the judge it cannot run commands itself", () => { - expect(TOOL_OUTPUTS_GUIDANCE.toLowerCase()).toContain("cannot run any commands"); + expect(TRAJECTORY_GUIDANCE.toLowerCase()).toContain("cannot run any commands"); + }); + + it("names the judge's own read-only tools and the unified trajectory tools", () => { + expect(TRAJECTORY_GUIDANCE).toContain("## Your Tools"); + expect(TRAJECTORY_GUIDANCE).toContain("read_file"); + expect(TRAJECTORY_GUIDANCE).toContain("list_agent_tool_calls"); + expect(TRAJECTORY_GUIDANCE).toContain("get_agent_tool_calls"); + expect(TRAJECTORY_GUIDANCE).toContain("list_agent_tools"); + expect(TRAJECTORY_GUIDANCE).toContain("get_agent_tools"); + expect(TRAJECTORY_GUIDANCE.toLowerCase()).toContain("codebase"); }); - it("names the judge's own read-only tools and disambiguates them from the agent's", () => { - expect(TOOL_OUTPUTS_GUIDANCE).toContain("## Your Tools"); - expect(TOOL_OUTPUTS_GUIDANCE).toContain("read_file"); - expect(TOOL_OUTPUTS_GUIDANCE).toContain("read_tool_outputs"); - expect(TOOL_OUTPUTS_GUIDANCE).toContain("get_tool_output"); - expect(TOOL_OUTPUTS_GUIDANCE.toLowerCase()).toContain("codebase"); + it("tells the judge to open spilled outputs (responseFile/descriptionFile) with read_file", () => { + expect(TRAJECTORY_GUIDANCE).toContain("responseFile"); + expect(TRAJECTORY_GUIDANCE).toContain("descriptionFile"); + expect(TRAJECTORY_GUIDANCE.toLowerCase()).toMatch(/open .*with read_file|read_file/); }); - it("frames the codebase and captured outputs as equally authoritative and to be examined together", () => { - expect(TOOL_OUTPUTS_GUIDANCE).toContain("## How to Judge"); - const g = TOOL_OUTPUTS_GUIDANCE.toLowerCase(); + it("frames the codebase and captured trajectory as equally authoritative and to be examined together", () => { + expect(TRAJECTORY_GUIDANCE).toContain("## How to Judge"); + const g = TRAJECTORY_GUIDANCE.toLowerCase(); expect(g).toContain("equally authoritative"); expect(g).toContain("examine both"); }); it("treats captured output as the record of what happened and forbids redundant re-proving", () => { - const g = TOOL_OUTPUTS_GUIDANCE.toLowerCase(); + const g = TRAJECTORY_GUIDANCE.toLowerCase(); expect(g).toContain("record of what happened"); expect(g).toMatch(/redo or re-prove/); }); it("overrides criteria wording that asks the judge to run commands", () => { - const g = TOOL_OUTPUTS_GUIDANCE.toLowerCase(); + const g = TRAJECTORY_GUIDANCE.toLowerCase(); expect(g).toContain("criterion"); expect(g).toMatch(/run, execute, or re-run/); expect(g).toContain("ignore that instruction"); }); - it("includes the guidance in the system prompt when tool outputs are present", () => { + it("includes the guidance in the system prompt when a trajectory is present", () => { const prompt = new TestableBundledStrategy("test-model").publicBuildSystemPrompt(true); - expect(prompt).toContain(TOOL_OUTPUTS_GUIDANCE); + expect(prompt).toContain(TRAJECTORY_GUIDANCE); }); it("frames What to Evaluate around the agent's work and points at the user message", () => { @@ -310,9 +386,9 @@ describe("judge tool-outputs guidance (issue #1125)", () => { expect(prompt).toMatch(/each criterion provided in the user message/); }); - it("omits the guidance when no tool outputs were captured", () => { + it("omits the guidance when no trajectory was captured", () => { const prompt = new TestableBundledStrategy("test-model").publicBuildSystemPrompt(false); - expect(prompt).not.toContain(TOOL_OUTPUTS_GUIDANCE); + expect(prompt).not.toContain(TRAJECTORY_GUIDANCE); }); }); diff --git a/apps/judge/src/judge-strategies.ts b/apps/judge/src/judge-strategies.ts index 19705be5e..dfa6c2ef2 100644 --- a/apps/judge/src/judge-strategies.ts +++ b/apps/judge/src/judge-strategies.ts @@ -2,8 +2,9 @@ // Licensed under the MIT License. import { CopilotClient, defineTool, SessionEvent } from "@github/copilot-sdk"; -import { readFileSync, readdirSync, statSync, existsSync } from "fs"; -import { join, resolve, sep } from "path"; +import { readFileSync, readdirSync, statSync, existsSync, mkdtempSync, rmSync } from "fs"; +import { tmpdir } from "os"; +import { isAbsolute, join, resolve, sep } from "path"; import { execFileSync } from "child_process"; import { CriteriaConfig, @@ -12,10 +13,13 @@ import { ConversationTurn, DependencyGraph, GateId, - ToolCall, TokenManagerClient, withRetry, } from "shared"; +import { + type AgentTrajectory, + createTrajectoryTools, +} from "./agent-trajectory.js"; export interface JudgeStrategyContext { workspacePath: string; @@ -28,8 +32,9 @@ export interface JudgeStrategyContext { onProgress?: (result: CriterionResult) => void; /** Which gate is being evaluated. Defaults to select. */ gate?: GateId; - /** This iteration's captured tool calls/outputs (build/test/run output). */ - toolCalls?: ToolCall[]; + /** The coding agent's captured trajectory (normalized from the HAR tool calls + * for gates, or the ATIF for observations). */ + trajectory?: AgentTrajectory; } /** @@ -54,11 +59,16 @@ const DEFAULT_JUDGE_RETRIES = 3; * its decision on actual evidence instead of demanding the agent re-prove work * it has already done. It is intentionally generic across all criteria. */ -export const TOOL_OUTPUTS_GUIDANCE = `## Your Tools -You have read-only tools to gather evidence: read_file, list_directory, search_files and file_exists to inspect the workspace, and read_tool_outputs / get_tool_output to review the tool calls the coding agent ran while doing the task. You yourself cannot run any commands or coding-agent tools — you can only read what the agent already did. +export const TRAJECTORY_GUIDANCE = `## Your Tools +You have read-only tools to gather evidence — you cannot run any commands or coding-agent tools yourself, you can only read what the agent already did: +- Inspect the resulting codebase: read_file, list_directory, search_files, file_exists. +- See what tools the agent HAD available (its catalog): list_agent_tools, then get_agent_tools(toolNames) for a tool's full description and parameter schema. +- See what the agent DID: list_agent_tool_calls to triage its invocations (id, name, a short summary and a preview of each output), then get_agent_tool_calls(toolCallIds) for the full arguments and captured output of the calls you care about. +- get_trajectory_overview gives a high-level orientation (agent, model, steps) without the raw system prompt. +When get_agent_tool_calls or get_agent_tools returns a responseFile or descriptionFile (large outputs are spilled to a file instead of inlined), open that path with read_file. ## How to Judge -The codebase and the agent's captured tool outputs are two complementary, equally authoritative sources of evidence — examine both. The files show the resulting state of the code; the captured outputs (logs, results, exit status) show what actually happened when the agent ran a command, which the files alone may not reveal. When a criterion concerns something the agent did or ran, take the agent's captured output and exit status as the record of what happened, rather than asking the agent to redo or re-prove work the evidence already shows. If a criterion's wording tells you to run, execute, or re-run a command, ignore that instruction and judge the outcome from the captured outputs together with the codebase.`; +The resulting codebase and the agent's captured trajectory are two complementary, equally authoritative sources of evidence — examine both. The files show the resulting state of the code; the captured tool calls and their outputs show what actually happened while the agent worked, including intermediate actions the final files no longer reveal (for example a dependency added and then later removed within the run). Start with list_agent_tool_calls to see what the agent did, then drill into the relevant calls with get_agent_tool_calls. When a criterion concerns something the agent did or ran, take the agent's captured output and exit status as the record of what happened, rather than asking the agent to redo or re-prove work the evidence already shows. If a criterion's wording tells you to run, execute, or re-run a command, ignore that instruction and judge the outcome from the captured trajectory together with the codebase.`; /** * Tool filter applied to every judge session. @@ -66,11 +76,11 @@ The codebase and the agent's captured tool outputs are two complementary, equall * The Copilot SDK's `CopilotClient` defaults to `mode: "copilot-cli"`, which * injects the full set of built-in CLI tools (bash, edit, view, ...) into the * session alongside the read-only `custom:*` tools we register in - * `createFileTools`/`createToolOutputTools`. Those built-ins are NOT + * `createFileTools`/`createTrajectoryTools`. Those built-ins are NOT * `skipPermission`, and the judge runs headless (no TUI to answer prompts). * * The failure mode this guards against: instead of reading the coder's captured - * output via `read_tool_outputs`/`get_tool_output`, the judge model decides to + * output via `list_agent_tool_calls`/`get_agent_tool_calls`, the judge model decides to * "verify" a build/test by running the command itself through the built-in * `bash` tool. Headless, that call is denied with "could not request permission * from user". The judge then mis-reports this as the coder's result ("execution @@ -103,6 +113,17 @@ export function isWithinWorkspace(root: string, target: string): boolean { ); } +/** + * Returns true if `target` resolves inside (or equal to) ANY of `roots`. Used so + * read_file/file_exists may read both the workspace and a judge-created spill dir + * (created OUTSIDE the workspace for oversized trajectory tool outputs) without + * weakening the per-root traversal guard — each root is still checked with the + * exact-or-separator-boundary rule of `isWithinWorkspace`. + */ +export function isWithinAnyRoot(roots: string[], target: string): boolean { + return roots.some((root) => isWithinWorkspace(root, target)); +} + export abstract class JudgeStrategy { protected model: string; protected timeout: number; @@ -126,11 +147,21 @@ export abstract class JudgeStrategy { /** * Create filesystem inspection tools scoped to the workspace */ - protected createFileTools(workspacePath: string) { + protected createFileTools( + workspacePath: string, + opts?: { extraReadRoots?: string[] }, + ) { const workspaceRoot = resolve(workspacePath); + // Additional roots (OUTSIDE the workspace) that read_file/file_exists may + // read by ABSOLUTE path — used for the trajectory spill dir so the judge can + // open oversized tool outputs returned as responseFile/descriptionFile. + // list_directory/search_files stay workspace-only so the codebase view the + // judge inspects is never polluted by spill files. + const extraReadRoots = (opts?.extraReadRoots ?? []).map((r) => resolve(r)); + const readRoots = [workspaceRoot, ...extraReadRoots]; const readFile = defineTool("read_file", { description: - "Read the contents of a file in the workspace. Returns the full text content. Use relative paths from the workspace root.", + "Read the contents of a file. Returns the full text content. Use a relative path from the workspace root, or an absolute path returned as responseFile/descriptionFile by the agent-trajectory tools.", // Read-only, workspace-scoped, traversal-guarded tools must run without a // permission prompt: the judge is headless (no TUI), so the v3 runtime // would otherwise deny every call with "could not request permission from @@ -147,8 +178,10 @@ export abstract class JudgeStrategy { required: ["path"], }, handler: async (args: { path: string }) => { - const fullPath = join(workspaceRoot, args.path); - if (!isWithinWorkspace(workspaceRoot, fullPath)) { + const fullPath = isAbsolute(args.path) + ? resolve(args.path) + : join(workspaceRoot, args.path); + if (!isWithinAnyRoot(readRoots, fullPath)) { return { error: "Path traversal not allowed" }; } if (!existsSync(fullPath)) { @@ -318,8 +351,10 @@ export abstract class JudgeStrategy { required: ["path"], }, handler: async (args: { path: string }) => { - const fullPath = join(workspaceRoot, args.path); - if (!isWithinWorkspace(workspaceRoot, fullPath)) { + const fullPath = isAbsolute(args.path) + ? resolve(args.path) + : join(workspaceRoot, args.path); + if (!isWithinAnyRoot(readRoots, fullPath)) { return { error: "Path traversal not allowed" }; } const exists = existsSync(fullPath); @@ -335,129 +370,68 @@ export abstract class JudgeStrategy { return [readFile, listDirectory, searchFiles, fileExists]; } - /** - * Create tools that expose the tool calls/outputs captured during the gate - * iteration being judged (e.g. a build/test command's stdout/stderr/exit - * code). `read_tool_outputs` lists calls with truncated previews; - * `get_tool_output` returns the full output for a given index on demand. - */ - protected createToolOutputTools(toolCalls: ToolCall[]) { - const PREVIEW_LIMIT = 2_000; - const FULL_LIMIT = 100_000; - - const readToolOutputs = defineTool("read_tool_outputs", { - description: - "List the tool calls the coding agent made during this iteration (e.g. shell/bash commands and their output). Returns each call's index, name, arguments and a truncated response preview. Use get_tool_output(index) to fetch the full output of a specific call. Consult these to decide whether a command (build, test, run) actually succeeded.", - // Read-only in-memory inspection of already-captured tool calls. Like the - // file tools above, this MUST run without a permission prompt: the judge is - // headless (no TUI), so the v3 runtime would otherwise deny every call with - // "could not request permission from user" — which silently blocks the judge - // from ever seeing the coding agent's build/test output and forces it to - // demand on-disk proof files instead. See scope #1125. - skipPermission: true, - parameters: { - type: "object", - properties: {}, - required: [], - }, - handler: async () => { - if (toolCalls.length === 0) { - return { count: 0, calls: [], message: "No tool calls were captured for this iteration." }; - } - const calls = toolCalls.map((tc, index) => { - const response = tc.response ?? ""; - return { - index, - name: tc.name, - arguments: tc.arguments, - responsePreview: - response.length > PREVIEW_LIMIT - ? response.substring(0, PREVIEW_LIMIT) + "\n…(truncated, use get_tool_output)" - : response, - responseTruncated: response.length > PREVIEW_LIMIT, - responseLength: response.length, - }; - }); - return { count: calls.length, calls }; - }, - }); - - const getToolOutput = defineTool("get_tool_output", { - description: - "Return the full captured output (response) of a single tool call by its index, as listed by read_tool_outputs.", - // Read-only; same headless permission rationale as read_tool_outputs. See scope #1125. - skipPermission: true, - parameters: { - type: "object", - properties: { - index: { - type: "number", - description: "Zero-based index of the tool call from read_tool_outputs", - }, - }, - required: ["index"], - }, - handler: async (args: { index: number }) => { - const tc = toolCalls[args.index]; - if (!tc) { - return { error: `No tool call at index ${args.index} (have ${toolCalls.length})` }; - } - const response = tc.response ?? ""; - if (response.length > FULL_LIMIT) { - return { - index: args.index, - name: tc.name, - arguments: tc.arguments, - response: response.substring(0, FULL_LIMIT), - truncated: true, - totalLength: response.length, - }; - } - return { index: args.index, name: tc.name, arguments: tc.arguments, response }; - }, - }); - - return [readToolOutputs, getToolOutput]; - } - /** * Run a Copilot session with given prompt and tools, retrying on timeout. + * + * When a trajectory with at least one captured call is supplied, the unified + * trajectory navigation tools are registered (mirroring the previous + * `toolCalls.length > 0` gate) and a per-evaluation spill dir is created + * OUTSIDE the workspace for oversized tool outputs; read_file is granted that + * dir as an extra read root so the judge can open spilled files. The spill dir + * is best-effort removed when the session finishes. */ protected async runCopilotSession( workspacePath: string, systemPrompt: string, userPrompt: string, - toolCalls?: ToolCall[] + trajectory?: AgentTrajectory ): Promise { + const hasTrajectory = !!trajectory && trajectory.calls.length > 0; + const spillDir = hasTrajectory + ? mkdtempSync(join(tmpdir(), "judge-trajectory-")) + : undefined; const tools = [ - ...this.createFileTools(workspacePath), - ...(toolCalls && toolCalls.length > 0 ? this.createToolOutputTools(toolCalls) : []), + ...this.createFileTools( + workspacePath, + spillDir ? { extraReadRoots: [spillDir] } : undefined, + ), + ...(hasTrajectory ? createTrajectoryTools(trajectory!, spillDir!) : []), ]; - return withRetry( - () => this.doRunCopilotSession(tools, systemPrompt, userPrompt), - { - maxRetries: this.maxRetries, - baseDelayMs: 10_000, - maxDelayMs: 30_000, - isRetryable: (error) => { - const msg = error instanceof Error ? error.message : String(error); - return ( - msg.includes("timeout") || - msg.includes("Timeout") || - msg.includes("aborted") || - msg.includes("ECONNRESET") || - msg.includes("socket hang up") - ); - }, - onRetry: (error, attempt) => { - const msg = error instanceof Error ? error.message : String(error); - console.warn( - `[judge-strategy] sendAndWait attempt ${attempt} failed (retrying in ≤30s): ${msg.substring(0, 200)}` - ); - }, + try { + return await withRetry( + () => this.doRunCopilotSession(tools, systemPrompt, userPrompt), + { + maxRetries: this.maxRetries, + baseDelayMs: 10_000, + maxDelayMs: 30_000, + isRetryable: (error) => { + const msg = error instanceof Error ? error.message : String(error); + return ( + msg.includes("timeout") || + msg.includes("Timeout") || + msg.includes("aborted") || + msg.includes("ECONNRESET") || + msg.includes("socket hang up") + ); + }, + onRetry: (error, attempt) => { + const msg = error instanceof Error ? error.message : String(error); + console.warn( + `[judge-strategy] sendAndWait attempt ${attempt} failed (retrying in ≤30s): ${msg.substring(0, 200)}` + ); + }, + } + ); + } finally { + if (spillDir) { + try { + rmSync(spillDir, { recursive: true, force: true }); + } catch { + // best-effort cleanup of the out-of-workspace spill dir + } } - ); + } } /** @@ -526,12 +500,12 @@ export class BundledStrategy extends JudgeStrategy { criteria, conversationHistory, personaInstructions, - toolCalls, + trajectory, } = context; const systemPrompt = this.buildSystemPrompt( personaInstructions, - toolCalls && toolCalls.length > 0 + !!trajectory && trajectory.calls.length > 0 ); const userPrompt = this.buildUserPrompt(criteria, conversationHistory); @@ -540,7 +514,7 @@ export class BundledStrategy extends JudgeStrategy { workspacePath, systemPrompt, userPrompt, - toolCalls + trajectory ); return this.parseJsonResponse(response, criteria, context.onProgress); @@ -555,14 +529,14 @@ export class BundledStrategy extends JudgeStrategy { */ private buildSystemPrompt( personaInstructions?: string, - hasToolOutputs?: boolean + hasTrajectory?: boolean ): string { const personaSection = personaInstructions ? `\n## Persona\n${personaInstructions}\n` : ""; - const toolOutputsSection = hasToolOutputs - ? `\n${TOOL_OUTPUTS_GUIDANCE}\n` + const toolOutputsSection = hasTrajectory + ? `\n${TRAJECTORY_GUIDANCE}\n` : ""; return `You are an expert code reviewer evaluating the tool calls, logs and generated code produced by a coding agent. @@ -729,7 +703,7 @@ export class IndependentStrategy extends JudgeStrategy { personaInstructions, onProgress, gate, - toolCalls, + trajectory, } = context; // Get topological order @@ -795,7 +769,7 @@ export class IndependentStrategy extends JudgeStrategy { personaInstructions, evalIndex++, gate, - toolCalls + trajectory ) ); @@ -836,11 +810,11 @@ export class IndependentStrategy extends JudgeStrategy { personaInstructions: string | undefined, index: number, gate?: GateId, - toolCalls?: ToolCall[] + trajectory?: AgentTrajectory ): Promise { const systemPrompt = this.buildSystemPrompt( personaInstructions, - toolCalls && toolCalls.length > 0 + !!trajectory && trajectory.calls.length > 0 ); const userPrompt = this.buildUserPrompt(criterion, conversationHistory); @@ -850,7 +824,7 @@ export class IndependentStrategy extends JudgeStrategy { workspacePath, systemPrompt, userPrompt, - toolCalls + trajectory ); const passed = this.detectPassFail(response); @@ -884,14 +858,14 @@ export class IndependentStrategy extends JudgeStrategy { */ protected buildSystemPrompt( personaInstructions?: string, - hasToolOutputs?: boolean + hasTrajectory?: boolean ): string { const personaSection = personaInstructions ? `\n## Persona\n${personaInstructions}\n` : ""; - const toolOutputsSection = hasToolOutputs - ? `\n${TOOL_OUTPUTS_GUIDANCE}\n` + const toolOutputsSection = hasTrajectory + ? `\n${TRAJECTORY_GUIDANCE}\n` : ""; return `You are an expert code reviewer evaluating the tool calls, logs and generated code produced by a coding agent against ONE specific criterion. diff --git a/apps/judge/src/observation-trajectory.integration.test.ts b/apps/judge/src/observation-trajectory.integration.test.ts index c8b31030b..185a6b54b 100644 --- a/apps/judge/src/observation-trajectory.integration.test.ts +++ b/apps/judge/src/observation-trajectory.integration.test.ts @@ -6,14 +6,15 @@ * * Proves the judge resolves a trajectory-only observation — one whose answer is * NOT derivable from the final workspace snapshot — by consuming the agent's - * ATIF trajectory that pp-taxonomy forwards as a synthetic `agent_trajectory_atif` - * tool call (see apps/judge/src/index.ts). The canonical case is - * `dependency_added_then_removed`: a package is installed then uninstalled mid-run, - * so it is invisible in the end-state snapshot and only the trajectory reveals it. + * ATIF trajectory normalized into an {@link AgentTrajectory} and navigated via + * the unified trajectory tools (list_agent_tool_calls / get_agent_tool_calls). + * The canonical case is `dependency_added_then_removed`: a package is installed + * then uninstalled mid-run, so it is invisible in the end-state snapshot and + * only the trajectory reveals it. * * Two assertions form the acceptance: - * 1. WITH the trajectory tool call → criterion resolves passed:true (the judge - * actually consumed the ATIF). + * 1. WITH the trajectory → criterion resolves passed:true (the judge actually + * consumed the ATIF via the navigation tools). * 2. WITHOUT the trajectory (snapshot only) → the same criterion flips to * passed:false, proving the verdict comes from the trajectory, not the files. * @@ -26,33 +27,97 @@ import { mkdtempSync, writeFileSync, rmSync, realpathSync } from "fs"; import { tmpdir } from "os"; import { join } from "path"; import { resetCriteriaProvider } from "shared/criteria-provider-factory"; -import type { ToolCall } from "shared"; +import { fromAtif } from "./agent-trajectory.js"; import { evaluateWorkspace } from "./judge-agent.js"; const hasToken = !!process.env.GITHUB_TOKEN; const JUDGE_MODEL = process.env.JUDGE_MODEL || "gpt-5.4-mini"; /** - * Synthetic ATIF trajectory mirroring what pp-taxonomy forwards: the agent - * installs `left-pad`, uses it, then uninstalls it before finishing. The final - * package.json (below) therefore does NOT list left-pad. + * Realistic ATIF trajectory (the exact shape pp-taxonomy downloads and forwards + * via `fromAtif`): a system + user step, then agent steps that install + * `left-pad`, use it, and uninstall it before finishing. Each agent step pairs + * its `tool_calls[]` with the matching same-step `observation.results[]` by + * `source_call_id === tool_call_id`. The final package.json (below) therefore + * does NOT list left-pad — the add-then-remove is only in the trajectory. */ -const ATIF_TRAJECTORY = { - schema: "atif/1", +const REALISTIC_ATIF = { + schema_version: "atif/1", + session_id: "atif-trajectory-test", + agent: { + name: "copilot", + version: "1.0.0", + model_name: "claude-opus-4.6", + tool_definitions: [ + { + type: "function", + function: { + name: "bash", + description: "Run a shell command in the workspace.", + parameters: { + type: "object", + properties: { command: { type: "string" } }, + required: ["command"], + }, + }, + }, + { + type: "function", + function: { + name: "edit", + description: "Edit a file in the workspace.", + parameters: { + type: "object", + properties: { path: { type: "string" } }, + required: ["path"], + }, + }, + }, + ], + }, steps: [ - { type: "tool_call", name: "run_in_terminal", arguments: { command: "npm install left-pad" }, response: "+ left-pad@1.3.0\nadded 1 package" }, - { type: "tool_call", name: "edit_file", arguments: { path: "index.js" }, response: "require('left-pad')" }, - { type: "tool_call", name: "run_in_terminal", arguments: { command: "npm uninstall left-pad" }, response: "removed 1 package" }, + { step_id: 1, source: "system" }, + { step_id: 2, source: "user" }, + { + step_id: 3, + source: "agent", + model_name: "claude-opus-4.6", + reasoning_content: "I'll add left-pad to pad the output.", + tool_calls: [ + { tool_call_id: "call-1", function_name: "bash", arguments: { command: "npm install left-pad" } }, + ], + observation: { + results: [ + { source_call_id: "call-1", content: "added 1 package\n+ left-pad@1.3.0\nfound 0 vulnerabilities" }, + ], + }, + }, + { + step_id: 4, + source: "agent", + model_name: "claude-opus-4.6", + tool_calls: [ + { tool_call_id: "call-2", function_name: "edit", arguments: { path: "index.js" } }, + ], + observation: { + results: [{ source_call_id: "call-2", content: "File index.js edited: now requires('left-pad')" }], + }, + }, + { + step_id: 5, + source: "agent", + model_name: "claude-opus-4.6", + reasoning_content: "On reflection I don't need left-pad; removing it.", + tool_calls: [ + { tool_call_id: "call-3", function_name: "bash", arguments: { command: "npm uninstall left-pad" } }, + ], + observation: { + results: [{ source_call_id: "call-3", content: "removed 1 package\nfound 0 vulnerabilities" }], + }, + }, ], }; -const TRAJECTORY_TOOL_CALL: ToolCall = { - id: "atif-trajectory", - name: "agent_trajectory_atif", - arguments: {}, - response: JSON.stringify(ATIF_TRAJECTORY), -}; - describe("judge ATIF-trajectory observation (integration)", () => { let workspaceDir: string; let criteriaDir: string; @@ -87,8 +152,9 @@ describe("judge ATIF-trajectory observation (integration)", () => { " removed it before finishing (e.g. added left-pad then removed it). This", " is only derivable from the agent's trajectory (mid-run tool calls), not", " from the final codebase snapshot, because the dependency is absent in the", - " end state. Inspect the agent_trajectory_atif tool output to detect a", - " transient add-then-remove.", + " end state. Use list_agent_tool_calls to scan the agent's tool calls and", + " get_agent_tool_calls to inspect the relevant ones, detecting a transient", + " add-then-remove of a dependency.", "", ].join("\n"), ); @@ -114,13 +180,13 @@ describe("judge ATIF-trajectory observation (integration)", () => { "resolves a trajectory-only observation true WITH the ATIF, false WITHOUT it", { timeout: 600_000 }, async () => { - // (1) WITH the ATIF trajectory tool call → the agent's add-then-remove is - // visible, so the observation must resolve true. + // (1) WITH the ATIF trajectory → the agent's add-then-remove is visible via + // the navigation tools, so the observation must resolve true. const withAtif = await evaluateWorkspace({ workspacePath: workspaceDir, criteria: ["dependency_added_then_removed"], conversationHistory: [], - toolCalls: [TRAJECTORY_TOOL_CALL], + trajectory: fromAtif(REALISTIC_ATIF), }); const obsWith = withAtif.criteriaResults.find( From 4c1785fd14707959847dc94ba08179e689728329 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 19:17:14 -0700 Subject: [PATCH 11/17] feat(shared): add criterion subject (run|iteration) + run-level observationResults (#1156) Introduce a kind-dependent `subject` discriminator on criteria so observation criteria can be evaluated either per-iteration (today) or once over the whole run. Gate criteria are always `iteration`. Adds: - CriterionSubject type + CRITERION_SUBJECTS + isCriterionSubject guard - subject on CriteriaConfig; run-level observationResults on RunState - zod schemas (optional, no flat default); kind-dependent default applied in the store (observation -> run, gate/legacy -> iteration) - subject threaded through criteria-store create/update/find + both providers - atifUrls[] on JudgeEvaluateRequest for merged whole-run trajectory - migration 022 backfills subject kind-aware (gate/legacy -> iteration, observation -> run) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../022-backfill-criteria-subject.ts | 76 +++++++++++++++++++ .../db-migrations/src/required-migrations.ts | 1 + .../src/criteria/criteria-provider-api.ts | 2 + .../src/criteria/criteria-provider-fs.ts | 8 ++ .../src/criteria/criteria-store.test.ts | 62 +++++++++++++++ .../shared/src/criteria/criteria-store.ts | 34 ++++++++- packages/shared/src/judge/judge-client.ts | 5 ++ packages/shared/src/schemas/criteria.ts | 7 +- packages/shared/src/schemas/request.ts | 4 + packages/shared/src/types/types.ts | 33 ++++++++ 10 files changed, 228 insertions(+), 4 deletions(-) create mode 100644 packages/db-migrations/src/migrations/022-backfill-criteria-subject.ts diff --git a/packages/db-migrations/src/migrations/022-backfill-criteria-subject.ts b/packages/db-migrations/src/migrations/022-backfill-criteria-subject.ts new file mode 100644 index 000000000..b3efd41c6 --- /dev/null +++ b/packages/db-migrations/src/migrations/022-backfill-criteria-subject.ts @@ -0,0 +1,76 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/** + * Migration: Backfill criteria subject. + * + * The criterion `subject` field (issue #1156) controls the granularity at which + * an observation criterion is evaluated: `"iteration"` (per-turn) or `"run"` + * (once over the whole run). Gate criteria are always `"iteration"`. + * + * The backfill is kind-aware to match the new store defaults: + * - gate / legacy (kind absent) → "iteration" + * - observation → "run" + * + * Backfilling existing observations to "run" also corrects the + * `dependency_added_then_removed` false-negative that arose when whole-run + * observations were forced through per-iteration evaluation. Blast radius is + * negligible — #1156 is unmerged, so there are no production observation + * criteria yet. Mirrors 021-backfill-criteria-kind.ts. + * + * down() unsets `subject` on rows whose value is "run" or "iteration", + * restoring the pre-migration state. + */ + +import type { Db } from "mongodb"; +import type { MigrationInterface } from "mongo-migrate-ts"; +import { batchUpdate } from "../batch-update.js"; + +export class BackfillCriteriaSubject implements MigrationInterface { + async up(db: Db): Promise { + const criteria = db.collection("criteria"); + + // Gate + legacy (kind absent/null) → "iteration". + const gateCount = await batchUpdate( + criteria, + { + $and: [ + { $or: [{ subject: { $exists: false } }, { subject: null }] }, + { $or: [{ kind: "gate" }, { kind: { $exists: false } }, { kind: null }] }, + ], + }, + { $set: { subject: "iteration" } }, + "[022] backfill criteria.subject=iteration (gate/legacy)", + ); + + // Observation → "run". + const obsCount = await batchUpdate( + criteria, + { + $and: [ + { $or: [{ subject: { $exists: false } }, { subject: null }] }, + { kind: "observation" }, + ], + }, + { $set: { subject: "run" } }, + "[022] backfill criteria.subject=run (observation)", + ); + + console.log( + `[022] Backfilled subject: iteration on ${gateCount} gate/legacy criteria, run on ${obsCount} observation criteria`, + ); + } + + async down(db: Db): Promise { + const criteria = db.collection("criteria"); + + const count = await batchUpdate( + criteria, + { subject: { $in: ["run", "iteration"] } }, + { $unset: { subject: "" } }, + "[022-down] unset criteria.subject", + ); + + console.log(`[022-down] Unset subject on ${count} criteria`); + } +} diff --git a/packages/db-migrations/src/required-migrations.ts b/packages/db-migrations/src/required-migrations.ts index f5742e603..cf21cf82d 100644 --- a/packages/db-migrations/src/required-migrations.ts +++ b/packages/db-migrations/src/required-migrations.ts @@ -37,4 +37,5 @@ export const REQUIRED_MIGRATIONS: readonly string[] = [ "020-backfill-handler-status.ts", "020-create-codebase-indexes.ts", "021-backfill-criteria-kind.ts", + "022-backfill-criteria-subject.ts", ]; diff --git a/packages/shared/src/criteria/criteria-provider-api.ts b/packages/shared/src/criteria/criteria-provider-api.ts index 080dd1c9e..5ff916a6e 100644 --- a/packages/shared/src/criteria/criteria-provider-api.ts +++ b/packages/shared/src/criteria/criteria-provider-api.ts @@ -205,6 +205,7 @@ function mapToCriteriaConfig(data: Record): CriteriaConfig { const gates = data.gates as CriteriaConfig["gates"] | undefined; const kind = data.kind as CriteriaConfig["kind"] | undefined; const taxonomyElementId = data.taxonomyElementId as CriteriaConfig["taxonomyElementId"] | undefined; + const subject = data.subject as CriteriaConfig["subject"] | undefined; return { id: String(data.id), prompt: String(data.prompt), @@ -212,5 +213,6 @@ function mapToCriteriaConfig(data: Record): CriteriaConfig { ...(gates && gates.length > 0 ? { gates } : {}), ...(kind ? { kind } : {}), ...(taxonomyElementId ? { taxonomyElementId } : {}), + ...(subject ? { subject } : {}), }; } diff --git a/packages/shared/src/criteria/criteria-provider-fs.ts b/packages/shared/src/criteria/criteria-provider-fs.ts index 9c5042700..56fe94657 100644 --- a/packages/shared/src/criteria/criteria-provider-fs.ts +++ b/packages/shared/src/criteria/criteria-provider-fs.ts @@ -81,6 +81,14 @@ export class FileSystemCriteriaProvider implements CriteriaProvider { criteria.taxonomyElementId = String(taxonomyElementId).trim() as CriteriaConfig["taxonomyElementId"]; } + const subject = data.subject; + if (subject !== undefined) { + if (subject !== "run" && subject !== "iteration") { + throw new Error(`'subject' must be 'run' or 'iteration' in ${file}`); + } + criteria.subject = subject; + } + if (this.registry.has(criteria.id)) { throw new Error(`Duplicate criteria id '${criteria.id}' found in ${file}`); } diff --git a/packages/shared/src/criteria/criteria-store.test.ts b/packages/shared/src/criteria/criteria-store.test.ts index 07908886e..36342b74b 100644 --- a/packages/shared/src/criteria/criteria-store.test.ts +++ b/packages/shared/src/criteria/criteria-store.test.ts @@ -370,3 +370,65 @@ describe("CriteriaStore kind + taxonomy (issue #1156)", () => { await expect(store.resolveObservations(["gate"])).rejects.toThrow(CriteriaValidationError); }); }); + +describe("CriteriaStore subject (run | iteration)", () => { + it("defaults an observation's subject to 'run' when absent", async () => { + const store = new CriteriaStore(fakeCollection()); + const created = await store.create({ id: "obs_run", prompt: "p", kind: "observation" }); + expect(created.subject).toBe("run"); + }); + + it("defaults a gate's subject to 'iteration' when absent", async () => { + const store = new CriteriaStore(fakeCollection()); + const created = await store.create({ id: "g_iter", prompt: "p", kind: "gate" }); + expect(created.subject).toBe("iteration"); + }); + + it("defaults a legacy (kind-absent) criterion's subject to 'iteration'", async () => { + const store = new CriteriaStore(fakeCollection()); + const created = await store.create({ id: "legacy", prompt: "p" }); + expect(created.subject).toBe("iteration"); + }); + + it("persists an explicit subject:'iteration' override on an observation", async () => { + const store = new CriteriaStore(fakeCollection()); + const created = await store.create({ + id: "obs_iter", + prompt: "p", + kind: "observation", + subject: "iteration", + }); + expect(created.subject).toBe("iteration"); + }); + + it("rejects subject:'run' on a gate criterion", async () => { + const store = new CriteriaStore(fakeCollection()); + await expect( + store.create({ id: "g_run", prompt: "p", kind: "gate", subject: "run" }), + ).rejects.toThrow(CriteriaValidationError); + }); + + it("rejects subject:'run' on a legacy (kind-absent) criterion", async () => { + const store = new CriteriaStore(fakeCollection()); + await expect( + store.create({ id: "legacy_run", prompt: "p", subject: "run" }), + ).rejects.toThrow(CriteriaValidationError); + }); + + it("rejects flipping an observation to a gate while keeping subject:'run' on update", async () => { + const col = fakeCollection([ + { id: "obs_x", prompt: "p", dependsOn: [], kind: "observation", subject: "run", createdAt: new Date() }, + ]); + const store = new CriteriaStore(col); + await expect(store.update("obs_x", { kind: "gate" })).rejects.toThrow(CriteriaValidationError); + }); + + it("allows updating an observation's subject to 'iteration'", async () => { + const col = fakeCollection([ + { id: "obs_y", prompt: "p", dependsOn: [], kind: "observation", subject: "run", createdAt: new Date() }, + ]); + const store = new CriteriaStore(col); + const updated = await store.update("obs_y", { subject: "iteration" }); + expect(updated.subject).toBe("iteration"); + }); +}); diff --git a/packages/shared/src/criteria/criteria-store.ts b/packages/shared/src/criteria/criteria-store.ts index de51748df..2fd5dba8b 100644 --- a/packages/shared/src/criteria/criteria-store.ts +++ b/packages/shared/src/criteria/criteria-store.ts @@ -2,7 +2,7 @@ // Licensed under the MIT License. import { Collection } from 'mongodb'; -import { CriteriaConfig, CriteriaDocument, CriterionKind, GateId, TaxonomyElementId } from '../types/types.js'; +import { CriteriaConfig, CriteriaDocument, CriterionKind, CriterionSubject, GateId, TaxonomyElementId } from '../types/types.js'; import { DependencyGraph } from '../graph/dependency-graph.js'; import { gatesSatisfyInvariant } from '../gates/gates.js'; import { @@ -42,9 +42,13 @@ export class CriteriaStore { gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId; + subject?: CriterionSubject; }): Promise { const { id, prompt, dependsOn = [], gates, taxonomyElementId } = input; const kind: CriterionKind = input.kind ?? 'gate'; + // Kind-dependent default: observation ⇒ "run" (whole-run), gate/legacy ⇒ + // "iteration". Applied here (where kind is known) — not a flat schema default. + const subject: CriterionSubject = input.subject ?? (kind === 'observation' ? 'run' : 'iteration'); // Validate ID format if (!/^[a-z][a-z0-9_]*$/.test(id)) { @@ -60,6 +64,14 @@ export class CriteriaStore { ); } + // subject "run" is only valid for observation criteria — gate criteria are + // always evaluated per iteration (their pass/fail feeds the per-turn loop). + if (subject !== 'iteration' && kind !== 'observation') { + throw new CriteriaValidationError( + `subject '${subject}' is not allowed for ${kind} criteria; gate criteria are always evaluated per iteration` + ); + } + // Check for duplicates among *active* criteria. const existing = await this.collection.findOne({ id, deletedAt: { $exists: false } }); if (existing) { @@ -97,6 +109,7 @@ export class CriteriaStore { ...(gates !== undefined && { gates }), kind, ...(taxonomyElementId !== undefined && { taxonomyElementId }), + subject, createdAt: new Date(), }; @@ -138,7 +151,7 @@ export class CriteriaStore { /** Update a criterion's prompt, dependencies and/or gate compatibility */ async update( id: string, - patch: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId } + patch: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId; subject?: CriterionSubject } ): Promise { const existing = await this.get(id); if (!existing) { @@ -180,12 +193,23 @@ export class CriteriaStore { ); } + // subject "run" is only valid for observation criteria. Resolve the effective + // subject (kind-dependent default when neither patch nor existing set it). + const effectiveSubject: CriterionSubject = + patch.subject ?? existing.subject ?? (effectiveKind === 'observation' ? 'run' : 'iteration'); + if (effectiveSubject !== 'iteration' && effectiveKind !== 'observation') { + throw new CriteriaValidationError( + `subject '${effectiveSubject}' is not allowed for ${effectiveKind} criteria; gate criteria are always evaluated per iteration` + ); + } + const update: Record = { updatedAt: new Date() }; if (patch.prompt !== undefined) update.prompt = patch.prompt.trim(); if (patch.dependsOn !== undefined) update.dependsOn = patch.dependsOn; if (patch.gates !== undefined) update.gates = patch.gates; if (patch.kind !== undefined) update.kind = patch.kind; if (patch.taxonomyElementId !== undefined) update.taxonomyElementId = patch.taxonomyElementId; + if (patch.subject !== undefined) update.subject = patch.subject; await this.collection.updateOne( { id, deletedAt: { $exists: false } }, @@ -255,6 +279,7 @@ export class CriteriaStore { ...(doc.gates !== undefined && { gates: doc.gates }), ...(doc.kind !== undefined && { kind: doc.kind }), ...(doc.taxonomyElementId !== undefined && { taxonomyElementId: doc.taxonomyElementId }), + ...(doc.subject !== undefined && { subject: doc.subject }), }); if (doc.dependsOn) { @@ -284,6 +309,7 @@ export class CriteriaStore { ...(c.gates !== undefined && { gates: c.gates }), ...(c.kind !== undefined && { kind: c.kind }), ...(c.taxonomyElementId !== undefined && { taxonomyElementId: c.taxonomyElementId }), + ...(c.subject !== undefined && { subject: c.subject }), })); const edges: { from: string; to: string }[] = []; @@ -307,13 +333,15 @@ export class CriteriaStore { for (const config of configs) { const existing = await this.collection.findOne({ id: config.id }); if (!existing) { + const seedKind: CriterionKind = config.kind ?? 'gate'; await this.collection.insertOne({ id: config.id, prompt: config.prompt, dependsOn: config.dependsOn || [], ...(config.gates !== undefined && { gates: config.gates }), - kind: config.kind ?? 'gate', + kind: seedKind, ...(config.taxonomyElementId !== undefined && { taxonomyElementId: config.taxonomyElementId }), + subject: config.subject ?? (seedKind === 'observation' ? 'run' : 'iteration'), createdAt: new Date(), } as any); inserted++; diff --git a/packages/shared/src/judge/judge-client.ts b/packages/shared/src/judge/judge-client.ts index c07cfa82f..336404f66 100644 --- a/packages/shared/src/judge/judge-client.ts +++ b/packages/shared/src/judge/judge-client.ts @@ -21,6 +21,11 @@ export interface JudgeEvaluateRequest { * tool calls; sent by pp-taxonomy in place of toolCallsUrl so observations * can be evaluated against the full agent trajectory. See issue #1156. */ atifUrl?: string; + /** Blob URLs of EVERY iteration's ATIF trajectory JSON, in iteration order. + * Sent by pp-taxonomy for `subject:"run"` observations so the judge merges + * them into one whole-run trajectory (cross-iteration behavior is visible in a + * single evaluation). Takes precedence over `atifUrl`. See issue #1156. */ + atifUrls?: string[]; } /** diff --git a/packages/shared/src/schemas/criteria.ts b/packages/shared/src/schemas/criteria.ts index efe0761c1..71715eefd 100644 --- a/packages/shared/src/schemas/criteria.ts +++ b/packages/shared/src/schemas/criteria.ts @@ -3,12 +3,13 @@ import { z } from "zod"; import { extendZodWithOpenApi } from "@asteasolutions/zod-to-openapi"; -import { GATES, CRITERION_KINDS, TAXONOMY_ELEMENT_IDS } from "../types/types.js"; +import { GATES, CRITERION_KINDS, CRITERION_SUBJECTS, TAXONOMY_ELEMENT_IDS } from "../types/types.js"; extendZodWithOpenApi(z); export const GateIdSchema = z.enum(GATES); export const CriterionKindSchema = z.enum(CRITERION_KINDS); +export const CriterionSubjectSchema = z.enum(CRITERION_SUBJECTS); export const TaxonomyElementIdSchema = z.enum(TAXONOMY_ELEMENT_IDS); export const CreateCriteriaInputSchema = z @@ -19,6 +20,7 @@ export const CreateCriteriaInputSchema = z gates: z.array(GateIdSchema).optional(), kind: CriterionKindSchema.default("gate"), taxonomyElementId: TaxonomyElementIdSchema.optional(), + subject: CriterionSubjectSchema.optional(), }) .openapi("CreateCriteriaInput"); @@ -29,6 +31,7 @@ export const UpdateCriteriaInputSchema = z gates: z.array(GateIdSchema).optional(), kind: CriterionKindSchema.optional(), taxonomyElementId: TaxonomyElementIdSchema.optional(), + subject: CriterionSubjectSchema.optional(), }) .openapi("UpdateCriteriaInput"); @@ -40,6 +43,7 @@ export const CriteriaResponseSchema = z gates: z.array(GateIdSchema).optional(), kind: CriterionKindSchema.optional(), taxonomyElementId: TaxonomyElementIdSchema.optional(), + subject: CriterionSubjectSchema.optional(), createdAt: z.coerce.date(), updatedAt: z.coerce.date().optional(), deletedAt: z.coerce.date().optional(), @@ -54,6 +58,7 @@ export const CriteriaGraphNodeSchema = z gates: z.array(GateIdSchema).optional(), kind: CriterionKindSchema.optional(), taxonomyElementId: TaxonomyElementIdSchema.optional(), + subject: CriterionSubjectSchema.optional(), }) .openapi("CriteriaGraphNode"); diff --git a/packages/shared/src/schemas/request.ts b/packages/shared/src/schemas/request.ts index 1abc04062..48f77e9eb 100644 --- a/packages/shared/src/schemas/request.ts +++ b/packages/shared/src/schemas/request.ts @@ -216,6 +216,10 @@ export const RunStateSchema = z rawChatFormat: z.string().optional(), pausedAt: z.coerce.date().optional(), resumedAt: z.coerce.date().optional(), + /** Whole-run observation results (one per selected subject:"run" observation + * criterion), evaluated once against the merged trajectory. Per-iteration + * observation results live on each turn's `observationResults`. #1156. */ + observationResults: z.array(CriterionResultSchema).optional(), }) .openapi("RunState"); diff --git a/packages/shared/src/types/types.ts b/packages/shared/src/types/types.ts index 34e708e16..5e5bd320b 100644 --- a/packages/shared/src/types/types.ts +++ b/packages/shared/src/types/types.ts @@ -78,6 +78,27 @@ export function isCriterionKind(value: unknown): value is CriterionKind { return typeof value === "string" && (CRITERION_KINDS as readonly string[]).includes(value); } +/** + * Granularity at which an observation criterion is evaluated. `"iteration"` + * evaluates each `run.turns[]` in isolation (result → `turn.observationResults`); + * `"run"` evaluates **once** over the whole run — the final snapshot + the merged + * trajectory across all iterations (result → run-level `run.observationResults`). + * + * Honored **only** by the observation/taxonomy pipeline. Gate criteria are always + * `"iteration"` (the API rejects `"run"` for gates). The kind-dependent default + * (observation ⇒ `"run"`, gate/legacy ⇒ `"iteration"`) is applied in the criteria + * store, not as a flat schema default. See issue #1156. + */ +export type CriterionSubject = "run" | "iteration"; + +/** The hard-coded set of criterion subjects. */ +export const CRITERION_SUBJECTS = ["run", "iteration"] as const; + +/** True when `value` is a valid criterion subject. */ +export function isCriterionSubject(value: unknown): value is CriterionSubject { + return typeof value === "string" && (CRITERION_SUBJECTS as readonly string[]).includes(value); +} + /** * Hard-coded taxonomy elements for observation criteria — the three quality * dimensions from the R&A Readout design. Gates (select/build/test/run/deploy) @@ -445,6 +466,12 @@ export interface RunState { rawChatFormat?: string; /** Per-handler post-processing status. Keys are handler IDs (e.g. "pp-atif"). */ handlerStatus?: Record; + /** Whole-run observation results — one entry per selected `subject:"run"` + * observation criterion, evaluated once against the final snapshot + the + * merged trajectory across all iterations by pp-taxonomy. Per-iteration + * (`subject:"iteration"`) observations live on `turn.observationResults` + * instead. Empty/absent when no run-subject observations were selected. #1156. */ + observationResults?: CriterionResult[]; /** Set once the scheduler has triggered report generation for this run after * the handler DAG drained (every handler reached a terminal state). Acts as * an exactly-once guard so the scheduler POSTs /api/v1/reports/trigger at @@ -651,6 +678,12 @@ export interface CriteriaConfig { /** Design-time classification under a hard-coded taxonomy element. Only set * on observation criteria. Static — never stored per run. See issue #1156. */ taxonomyElementId?: TaxonomyElementId; + /** Evaluation granularity for observation criteria. `"iteration"` (per-turn, + * → `turn.observationResults`) or `"run"` (once over the whole run, → + * `run.observationResults`). Honored only for observations; gates are always + * `"iteration"`. Kind-dependent default applied in the store (observation ⇒ + * "run", gate/legacy ⇒ "iteration"). See issue #1156. */ + subject?: CriterionSubject; } // Per-criterion result from judge evaluation From 01ec0a71653d93f2f0396ccf7ae5609b62bda24a Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 19:17:54 -0700 Subject: [PATCH 12/17] feat(api): validate criterion subject + AI auto-detect (#1156) - routes/criteria: reject subject:run for gate criteria (400); allow run| iteration for observations on create + update - llm: authoring assistant returns suggestedSubject; heuristic maps cross- iteration phrasing -> run, end-state/per-step -> iteration; defaults run for observations when the model is unavailable - endpoints test: subject create + run-subject-gate rejection - regenerated OpenAPI snapshot (subject enum + run-level observationResults) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../openapi-snapshot.test.ts.snap | 34 +++++++++++++++++++ apps/api/src/endpoints.test.ts | 32 +++++++++++++++++ apps/api/src/llm.ts | 23 +++++++++++-- apps/api/src/routes/criteria.ts | 8 ++--- 4 files changed, 90 insertions(+), 7 deletions(-) diff --git a/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap b/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap index 3d5d038c9..7c09fa8b5 100644 --- a/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap +++ b/apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap @@ -670,6 +670,13 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "prompt": { "type": "string", }, + "subject": { + "enum": [ + "run", + "iteration", + ], + "type": "string", + }, "taxonomyElementId": { "enum": [ "dimension:idiomatic-use", @@ -1058,6 +1065,13 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "prompt": { "type": "string", }, + "subject": { + "enum": [ + "run", + "iteration", + ], + "type": "string", + }, "taxonomyElementId": { "enum": [ "dimension:idiomatic-use", @@ -1121,6 +1135,13 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "prompt": { "type": "string", }, + "subject": { + "enum": [ + "run", + "iteration", + ], + "type": "string", + }, "taxonomyElementId": { "enum": [ "dimension:idiomatic-use", @@ -2626,6 +2647,12 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "logsUrl": { "type": "string", }, + "observationResults": { + "items": { + "$ref": "#/components/schemas/CriterionResult", + }, + "type": "array", + }, "os": { "properties": { "arch": { @@ -3161,6 +3188,13 @@ exports[`OpenAPI spec snapshot > matches the committed snapshot 1`] = ` "prompt": { "type": "string", }, + "subject": { + "enum": [ + "run", + "iteration", + ], + "type": "string", + }, "taxonomyElementId": { "enum": [ "dimension:idiomatic-use", diff --git a/apps/api/src/endpoints.test.ts b/apps/api/src/endpoints.test.ts index 34a89df39..bfca5e19c 100644 --- a/apps/api/src/endpoints.test.ts +++ b/apps/api/src/endpoints.test.ts @@ -220,6 +220,38 @@ describe("API Endpoints", () => { expect(res.status).toBe(400); }); + + it("defaults an observation's subject to 'run'", async () => { + (mocks.criteriaCollection.findOne as any).mockResolvedValue(null); + + const res = await request(app) + .post("/api/v1/criteria") + .send({ id: "obs_subj", prompt: "Did the agent add then remove a dep?", dependsOn: [], kind: "observation" }); + + expect(res.status).toBe(201); + expect(res.body).toHaveProperty("subject", "run"); + }); + + it("creates an observation with an explicit subject:'iteration' override", async () => { + (mocks.criteriaCollection.findOne as any).mockResolvedValue(null); + + const res = await request(app) + .post("/api/v1/criteria") + .send({ id: "obs_iter", prompt: "x", dependsOn: [], kind: "observation", subject: "iteration" }); + + expect(res.status).toBe(201); + expect(res.body).toHaveProperty("subject", "iteration"); + }); + + it("rejects subject:'run' on a gate criterion", async () => { + (mocks.criteriaCollection.findOne as any).mockResolvedValue(null); + + const res = await request(app) + .post("/api/v1/criteria") + .send({ id: "gate_run", prompt: "x", dependsOn: [], kind: "gate", subject: "run" }); + + expect(res.status).toBe(400); + }); }); describe("POST /api/v1/criteria/seed", () => { diff --git a/apps/api/src/llm.ts b/apps/api/src/llm.ts index 7c4cd9328..bda126d6b 100644 --- a/apps/api/src/llm.ts +++ b/apps/api/src/llm.ts @@ -2,7 +2,7 @@ // Licensed under the MIT License. import { isUnexpected } from "@azure-rest/ai-inference"; -import { gatesSatisfyInvariant, type GateId } from "shared"; +import { gatesSatisfyInvariant, type GateId, type CriterionSubject } from "shared"; import { acquireInferenceClient, isLlmAvailable as inferenceAvailable } from "./llm-token.js"; export type SuggestDirection = "parents" | "children"; @@ -23,13 +23,18 @@ Given a natural-language description of a behavior or pattern to detect in a cod - Contain only lowercase letters, digits, and underscores - Be concise but descriptive (e.g., has_unit_tests, uses_typescript, has_docker_config) +3. Classify the evaluation SUBJECT — whether the behavior must be judged across the WHOLE run or within a SINGLE iteration in isolation: + - "run": the behavior spans multiple steps/iterations or describes a transition over the course of the session — phrasing like "at any point", "ever", "then later", "added then removed", "during the run", or anything only confirmable by viewing the entire run together. + - "iteration": the behavior is a single end-state fact or a per-step property judgeable from one iteration alone (e.g., "uses TypeScript", "has a Dockerfile", "ran the test suite this turn"). + When unsure, prefer "run". + Here are examples of good criteria prompts: - "The project uses Azure Bicep for infrastructure as code. Look for *.bicep files, bicepconfig.json, or main.bicep entry points." - "The project uses React framework. Look for react dependency in package.json, .jsx or .tsx files with React components." - "The project uses Node.js as its runtime environment. Look for package.json file or Node.js-specific configuration files." Respond with ONLY a JSON object in this exact format (no markdown, no code fences): -{"prompt": "your evaluation prompt here", "suggestedId": "your_suggested_id"}`; +{"prompt": "your evaluation prompt here", "suggestedId": "your_suggested_id", "suggestedSubject": "run"}`; interface SuggestDirectionCopy { /** One-line definition of the relationship being asked for. */ @@ -103,6 +108,10 @@ export interface GenerateResult { suggestedId: string; suggestedParents: string[]; suggestedChildren: string[]; + /** Suggested evaluation granularity for observation criteria. Defaults to + * "run"; "iteration" only when the model clearly detects an end-state / + * per-step property. Ignored by gate criteria (always per-iteration). #1156. */ + suggestedSubject: CriterionSubject; } export function isLlmAvailable(): boolean { @@ -122,6 +131,12 @@ function sanitizeId(suggestedId: unknown): string { return sanitized || "new_criterion"; } +/** Coerce a model-suggested subject. Defaults to "run" — "iteration" is only + * honored when the model returns it exactly (matching "prefer run when unsure"). */ +function sanitizeSubject(value: unknown): CriterionSubject { + return value === "iteration" ? "iteration" : "run"; +} + function parseJson(content: string): any { const cleaned = content.replace(/```json\s*|```\s*/g, "").trim(); try { @@ -169,7 +184,7 @@ async function author( llm: ChatClient, model: string, behavior: string, -): Promise<{ prompt: string; suggestedId: string }> { +): Promise<{ prompt: string; suggestedId: string; suggestedSubject: CriterionSubject }> { const content = await chat(llm, model, SYSTEM_PROMPT_AUTHOR, `NEW CRITERION TO CREATE:\n${behavior}`); const parsed = parseJson(content); if (!parsed.prompt) { @@ -178,6 +193,7 @@ async function author( return { prompt: String(parsed.prompt).trim(), suggestedId: sanitizeId(parsed.suggestedId), + suggestedSubject: sanitizeSubject(parsed.suggestedSubject), }; } @@ -274,5 +290,6 @@ export async function generateCriteriaPrompt( suggestedId: authored.suggestedId, suggestedParents, suggestedChildren, + suggestedSubject: authored.suggestedSubject, }; } diff --git a/apps/api/src/routes/criteria.ts b/apps/api/src/routes/criteria.ts index 40aee5d0c..7be113e11 100644 --- a/apps/api/src/routes/criteria.ts +++ b/apps/api/src/routes/criteria.ts @@ -427,9 +427,9 @@ apiRoute(ctx.app, ctx.registry, { 409: { description: "Criterion already exists" }, }, handler: async (req, res) => { - const { id, prompt, dependsOn = [], gates, kind, taxonomyElementId } = req.body; + const { id, prompt, dependsOn = [], gates, kind, taxonomyElementId, subject } = req.body; try { - const doc = await getCriteriaStore().create({ id, prompt, dependsOn, gates, kind, taxonomyElementId }); + const doc = await getCriteriaStore().create({ id, prompt, dependsOn, gates, kind, taxonomyElementId, subject }); res.status(201).json(doc); } catch (err) { if (!sendStoreError(res, err)) throw err; @@ -452,9 +452,9 @@ apiRoute(ctx.app, ctx.registry, { }, handler: async (req, res) => { const { id } = req.params; - const { prompt, dependsOn, gates, kind, taxonomyElementId } = req.body; + const { prompt, dependsOn, gates, kind, taxonomyElementId, subject } = req.body; try { - const updated = await getCriteriaStore().update(id, { prompt, dependsOn, gates, kind, taxonomyElementId }); + const updated = await getCriteriaStore().update(id, { prompt, dependsOn, gates, kind, taxonomyElementId, subject }); res.json(updated); } catch (err) { if (!sendStoreError(res, err)) throw err; From aa92dbe43484c8085dfc15740017d6b1098f4996 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 19:18:49 -0700 Subject: [PATCH 13/17] feat(judge): merge whole-run trajectory from atifUrls[] (#1156) - agent-trajectory: mergeTrajectories(list) concatenates calls in iteration order and unions tool definitions by id; fromAtifList helper - index: accept atifUrls[]; download all + merge into the single AgentTrajectory (single atifUrl path unchanged) - integration test: run-subject dependency_added_then_removed is false per iteration but true on the merged trajectory (GITHUB_TOKEN-gated) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- apps/judge/src/agent-trajectory.test.ts | 82 +++++++++++++++++ apps/judge/src/agent-trajectory.ts | 83 +++++++++++++++++ apps/judge/src/index.ts | 22 ++++- ...observation-trajectory.integration.test.ts | 92 ++++++++++++++++++- 4 files changed, 275 insertions(+), 4 deletions(-) diff --git a/apps/judge/src/agent-trajectory.test.ts b/apps/judge/src/agent-trajectory.test.ts index 78252d88f..8bb870184 100644 --- a/apps/judge/src/agent-trajectory.test.ts +++ b/apps/judge/src/agent-trajectory.test.ts @@ -17,7 +17,9 @@ import { join, sep } from "path"; import type { ToolCall } from "shared"; import { fromAtif, + fromAtifList, fromToolCalls, + mergeTrajectories, createTrajectoryTools, summarizeArgs, safeId, @@ -444,3 +446,83 @@ describe("helpers", () => { expect(previewOf("abc", 10)).toBe("abc"); }); }); + +describe("mergeTrajectories / fromAtifList (run-subject whole-run path #1156)", () => { + it("returns an empty trajectory for an empty list", () => { + const merged = mergeTrajectories([]); + expect(merged.calls).toEqual([]); + expect(merged.toolDefinitions).toEqual([]); + expect(merged.byId.size).toBe(0); + }); + + it("returns the single trajectory unchanged for a length-1 list", () => { + const only = fromAtif(REALISTIC_ATIF); + expect(mergeTrajectories([only])).toBe(only); + }); + + it("concatenates calls in iteration order and preserves both iterations' work", () => { + const it1 = fromToolCalls([ + { id: "a", name: "bash", arguments: { command: "npm install left-pad" }, response: "added 1 package", timestamp: "" }, + ]); + const it2 = fromToolCalls([ + { id: "b", name: "bash", arguments: { command: "npm uninstall left-pad" }, response: "removed 1 package", timestamp: "" }, + ]); + const merged = mergeTrajectories([it1, it2]); + expect(merged.calls.map((c) => c.toolCallId)).toEqual(["a", "b"]); + expect(merged.calls[0].response).toContain("added"); + expect(merged.calls[1].response).toContain("removed"); + // Both addressable by id. + expect(merged.byId.get("a")?.summary).toBe("npm install left-pad"); + expect(merged.byId.get("b")?.summary).toBe("npm uninstall left-pad"); + }); + + it("re-keys colliding tool-call ids so every call stays addressable", () => { + const it1 = fromToolCalls([{ id: "dup", name: "bash", arguments: { command: "first" }, response: "r1", timestamp: "" }]); + const it2 = fromToolCalls([{ id: "dup", name: "bash", arguments: { command: "second" }, response: "r2", timestamp: "" }]); + const merged = mergeTrajectories([it1, it2]); + expect(merged.calls).toHaveLength(2); + const ids = merged.calls.map((c) => c.toolCallId); + expect(ids[0]).toBe("dup"); + expect(ids[1]).toBe("it2:dup"); + expect(new Set(ids).size).toBe(2); + expect(merged.byId.get("dup")?.summary).toBe("first"); + expect(merged.byId.get("it2:dup")?.summary).toBe("second"); + }); + + it("unions tool definitions by name (first wins)", () => { + const a = fromAtif(REALISTIC_ATIF); // has a populated catalog + const b = fromAtif(REALISTIC_ATIF); + const merged = mergeTrajectories([a, b]); + // Deduplicated, not doubled. + expect(merged.toolDefinitions.length).toBe(a.toolDefinitions.length); + expect(merged.toolDefsByName.size).toBe(a.toolDefsByName.size); + }); + + it("tags merged step overviews with their source iteration", () => { + const merged = mergeTrajectories([fromAtif(REALISTIC_ATIF), fromAtif(REALISTIC_ATIF)]); + expect(merged.steps?.some((s) => s.source.startsWith("it1:"))).toBe(true); + expect(merged.steps?.some((s) => s.source.startsWith("it2:"))).toBe(true); + }); + + it("fromAtifList normalizes + merges raw ATIF payloads", () => { + const merged = fromAtifList([REALISTIC_ATIF, REALISTIC_ATIF]); + expect(merged.source).toBe("atif"); + // Twice the agent calls of a single ATIF. + const single = fromAtif(REALISTIC_ATIF); + expect(merged.calls.length).toBe(single.calls.length * 2); + }); + + it("merged trajectory is navigable via createTrajectoryTools across iterations", async () => { + const it1 = fromToolCalls([{ id: "x", name: "bash", arguments: { command: "npm install left-pad" }, response: "added", timestamp: "" }]); + const it2 = fromToolCalls([{ id: "y", name: "bash", arguments: { command: "npm uninstall left-pad" }, response: "removed", timestamp: "" }]); + const spillDir = mkdtempSync(join(tmpdir(), "judge-merge-spill-")); + try { + const tools = createTrajectoryTools(mergeTrajectories([it1, it2]), spillDir); + const res = await call(tools, "list_agent_tool_calls"); + expect(res.count).toBe(2); + expect(res.calls.map((c: any) => c.toolCallId)).toEqual(["x", "y"]); + } finally { + rmSync(spillDir, { recursive: true, force: true }); + } + }); +}); diff --git a/apps/judge/src/agent-trajectory.ts b/apps/judge/src/agent-trajectory.ts index 29b20a469..fe1ae85cb 100644 --- a/apps/judge/src/agent-trajectory.ts +++ b/apps/judge/src/agent-trajectory.ts @@ -364,6 +364,89 @@ export function fromAtif(input: unknown): AgentTrajectory { }; } +/** + * Merge several {@link AgentTrajectory}s (e.g. one per iteration) into a single + * whole-run trajectory for `subject:"run"` observation evaluation (#1156). + * + * - `calls` are concatenated in iteration order. Colliding `toolCallId`s (the + * ATIF synthesizes step-local ids like `atif-3-0` that restart each iteration) + * are re-keyed `it:` so every call stays addressable via + * `get_agent_tool_calls`. + * - `toolDefinitions` are unioned by name (first wins). + * - `steps` are concatenated, each tagged with its source iteration. + * + * A list of length 0 yields an empty trajectory; length 1 is returned as-is. + */ +export function mergeTrajectories(list: AgentTrajectory[]): AgentTrajectory { + const items = (list ?? []).filter((t): t is AgentTrajectory => Boolean(t)); + if (items.length === 0) { + return { + source: "atif", + calls: [], + byId: new Map(), + toolDefinitions: [], + toolDefsByName: new Map(), + }; + } + if (items.length === 1) return items[0]; + + const calls: AgentToolCall[] = []; + const seen = new Set(); + items.forEach((traj, t) => { + for (const call of traj.calls) { + let uid = call.toolCallId; + if (seen.has(uid)) { + uid = `it${t + 1}:${call.toolCallId}`; + let k = 1; + while (seen.has(uid)) uid = `it${t + 1}:${call.toolCallId}-${k++}`; + } + seen.add(uid); + calls.push(uid === call.toolCallId ? call : { ...call, toolCallId: uid }); + } + }); + + const toolDefinitions: AgentToolDefinition[] = []; + const toolDefsByName = new Map(); + for (const traj of items) { + for (const def of traj.toolDefinitions) { + if (!toolDefsByName.has(def.name)) { + toolDefsByName.set(def.name, def); + toolDefinitions.push(def); + } + } + } + + const steps: StepOverview[] = []; + items.forEach((traj, t) => { + for (const s of traj.steps ?? []) { + steps.push({ ...s, source: `it${t + 1}:${s.source}` }); + } + }); + + const first = items[0]; + const anyAtif = items.some((x) => x.source === "atif"); + return { + source: anyAtif ? "atif" : "tool-calls", + agent: first.agent, + model: first.model, + calls, + byId: buildById(calls), + toolDefinitions, + toolDefsByName, + steps: steps.length > 0 ? steps : undefined, + finalMetrics: items[items.length - 1]?.finalMetrics, + }; +} + +/** + * Normalize a list of raw ATIF payloads (one per iteration) into a single + * merged whole-run {@link AgentTrajectory}. Convenience over + * {@link mergeTrajectories} ∘ {@link fromAtif}. + */ +export function fromAtifList(inputs: unknown[]): AgentTrajectory { + return mergeTrajectories((inputs ?? []).map((x) => fromAtif(x))); +} + // --------------------------------------------------------------------------- // Tools // --------------------------------------------------------------------------- diff --git a/apps/judge/src/index.ts b/apps/judge/src/index.ts index fd5a42e38..7702b381c 100644 --- a/apps/judge/src/index.ts +++ b/apps/judge/src/index.ts @@ -5,7 +5,7 @@ import express, { Request, Response, NextFunction } from "express"; import dotenv from "dotenv"; import { evaluateWorkspace } from "./judge-agent.js"; import { verifyCopilotProtocol } from "./protocol-check.js"; -import { type AgentTrajectory, fromAtif, fromToolCalls } from "./agent-trajectory.js"; +import { type AgentTrajectory, fromAtif, fromAtifList, fromToolCalls } from "./agent-trajectory.js"; import { BlobStorage, RedisLogPublisher } from "shared"; import { mkdtempSync, rmSync } from "fs"; import { tmpdir } from "os"; @@ -53,7 +53,7 @@ app.post( const startTime = Date.now(); try { - const { snapshotUrl, criteria, conversationHistory, personaInstructions, requestId, gate, toolCallsUrl, atifUrl } = req.body; + const { snapshotUrl, criteria, conversationHistory, personaInstructions, requestId, gate, toolCallsUrl, atifUrl, atifUrls } = req.body; // Validate required fields if (!snapshotUrl || typeof snapshotUrl !== "string") { @@ -100,7 +100,23 @@ app.post( // Failures are non-fatal — the judge can still evaluate the workspace // files. See apps/judge/src/agent-trajectory.ts. let trajectory: AgentTrajectory | undefined; - if (atifUrl && typeof atifUrl === "string") { + if (Array.isArray(atifUrls) && atifUrls.length > 0) { + // Run-subject observations (#1156): merge every iteration's ATIF into + // one whole-run trajectory so cross-iteration behavior (e.g. a dep added + // in one turn and removed in another) is visible in a single evaluation. + const urls = atifUrls.filter((u: unknown): u is string => typeof u === "string"); + try { + const atifs = await Promise.all(urls.map((u) => blobStorage.downloadJson(u))); + trajectory = fromAtifList(atifs); + console.log( + `[judge] Loaded merged ATIF trajectory from ${urls.length} iteration(s): ${trajectory.calls.length} call(s), ${trajectory.toolDefinitions.length} tool definition(s)`, + ); + } catch (err) { + console.warn( + `[judge] Failed to load merged ATIF from ${urls.length} url(s): ${err instanceof Error ? err.message : String(err)}`, + ); + } + } else if (atifUrl && typeof atifUrl === "string") { try { const atif = await blobStorage.downloadJson(atifUrl); trajectory = fromAtif(atif); diff --git a/apps/judge/src/observation-trajectory.integration.test.ts b/apps/judge/src/observation-trajectory.integration.test.ts index 185a6b54b..066c40da3 100644 --- a/apps/judge/src/observation-trajectory.integration.test.ts +++ b/apps/judge/src/observation-trajectory.integration.test.ts @@ -27,7 +27,7 @@ import { mkdtempSync, writeFileSync, rmSync, realpathSync } from "fs"; import { tmpdir } from "os"; import { join } from "path"; import { resetCriteriaProvider } from "shared/criteria-provider-factory"; -import { fromAtif } from "./agent-trajectory.js"; +import { fromAtif, fromAtifList } from "./agent-trajectory.js"; import { evaluateWorkspace } from "./judge-agent.js"; const hasToken = !!process.env.GITHUB_TOKEN; @@ -228,4 +228,94 @@ describe("judge ATIF-trajectory observation (integration)", () => { } expect(true).toBe(true); }); + + // The run-subject "killer" test (#1156): the add-then-remove is split ACROSS + // iterations — iteration 1 installs left-pad, iteration 2 uninstalls it. Each + // per-iteration ATIF alone is `false` (one only adds, the other only removes), + // but the MERGED whole-run trajectory reveals the add-then-remove → `true`. + // This is exactly the cross-iteration property the `subject:"run"` path exists + // to evaluate, and the false-negative the user flagged. + const ATIF_INSTALL = { + schema_version: "atif/1", + session_id: "atif-it1-install", + agent: { name: "copilot", version: "1.0.0", model_name: "claude-opus-4.6", tool_definitions: [] }, + steps: [ + { step_id: 1, source: "system" }, + { + step_id: 2, + source: "agent", + model_name: "claude-opus-4.6", + reasoning_content: "Add left-pad to pad output.", + tool_calls: [{ tool_call_id: "i1-c1", function_name: "bash", arguments: { command: "npm install left-pad" } }], + observation: { results: [{ source_call_id: "i1-c1", content: "added 1 package\n+ left-pad@1.3.0" }] }, + }, + ], + }; + + const ATIF_UNINSTALL = { + schema_version: "atif/1", + session_id: "atif-it2-uninstall", + agent: { name: "copilot", version: "1.0.0", model_name: "claude-opus-4.6", tool_definitions: [] }, + steps: [ + { step_id: 1, source: "system" }, + { + step_id: 2, + source: "agent", + model_name: "claude-opus-4.6", + reasoning_content: "Remove left-pad; not needed.", + tool_calls: [{ tool_call_id: "i2-c1", function_name: "bash", arguments: { command: "npm uninstall left-pad" } }], + observation: { results: [{ source_call_id: "i2-c1", content: "removed 1 package\nfound 0 vulnerabilities" }] }, + }, + ], + }; + + it.skipIf(!hasToken)( + "run-subject: false per-iteration but true on the MERGED whole-run trajectory", + { timeout: 600_000 }, + async () => { + // (1) Iteration 1 alone (install only) → no add-then-remove → false. + const it1 = await evaluateWorkspace({ + workspacePath: workspaceDir, + criteria: ["dependency_added_then_removed"], + conversationHistory: [], + trajectory: fromAtif(ATIF_INSTALL), + }); + const it1Obs = it1.criteriaResults.find((r) => r.criterionId === "dependency_added_then_removed"); + expect(it1Obs!.evaluated).toBe(true); + expect( + it1Obs!.passed, + `iteration 1 (install only) should be false. Feedback: ${it1Obs!.feedback}`, + ).toBe(false); + + // (2) Iteration 2 alone (uninstall only) → no add-then-remove → false. + const it2 = await evaluateWorkspace({ + workspacePath: workspaceDir, + criteria: ["dependency_added_then_removed"], + conversationHistory: [], + trajectory: fromAtif(ATIF_UNINSTALL), + }); + const it2Obs = it2.criteriaResults.find((r) => r.criterionId === "dependency_added_then_removed"); + expect(it2Obs!.evaluated).toBe(true); + expect( + it2Obs!.passed, + `iteration 2 (uninstall only) should be false. Feedback: ${it2Obs!.feedback}`, + ).toBe(false); + + // (3) MERGED whole-run trajectory (install in it1 + uninstall in it2) → + // the add-then-remove is now visible → true. This is what subject:"run" + // sends the judge as `atifUrls` and is the false-negative the feature fixes. + const merged = await evaluateWorkspace({ + workspacePath: workspaceDir, + criteria: ["dependency_added_then_removed"], + conversationHistory: [], + trajectory: fromAtifList([ATIF_INSTALL, ATIF_UNINSTALL]), + }); + const mergedObs = merged.criteriaResults.find((r) => r.criterionId === "dependency_added_then_removed"); + expect(mergedObs!.evaluated).toBe(true); + expect( + mergedObs!.passed, + `merged whole-run trajectory should be true (add in it1, remove in it2). Feedback: ${mergedObs!.feedback}`, + ).toBe(true); + }, + ); }); From 4fc56b5c8fa1051be7737b846438355056935733 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 19:19:19 -0700 Subject: [PATCH 14/17] =?UTF-8?q?feat(pp-taxonomy):=20honor=20criterion=20?= =?UTF-8?q?subject=20=E2=80=94=20whole-run=20vs=20per-iteration=20(#1156)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - taxonomy-handler: resolve selected observations with their subject; split into iteration vs run buckets. Iteration-subject -> per-turn observationResults (unchanged). Run-subject -> evaluate once (final snapshot + every turn's atifUrl merged + full conversationHistory) -> run-level run.observationResults. Runs once because the handler runs once per run (the scheduler never sets message.iteration), so no idempotency guard needed. - HandlerContext gains criteriaCollection so handlers can resolve criterion metadata by id; wired in post-processor + atif-handler test contexts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/handlers/atif-handler.test.ts | 1 + .../handlers/atif-handler.websocket.test.ts | 1 + .../src/handlers/taxonomy-handler.test.ts | 198 +++++++++++++++--- .../src/handlers/taxonomy-handler.ts | 126 ++++++++++- .../post-processor/src/post-processor.ts | 1 + apps/workers/post-processor/src/types.ts | 3 + 6 files changed, 287 insertions(+), 43 deletions(-) diff --git a/apps/workers/post-processor/src/handlers/atif-handler.test.ts b/apps/workers/post-processor/src/handlers/atif-handler.test.ts index bda415ad1..41e5520be 100644 --- a/apps/workers/post-processor/src/handlers/atif-handler.test.ts +++ b/apps/workers/post-processor/src/handlers/atif-handler.test.ts @@ -49,6 +49,7 @@ describe("AtifHandler", () => { }), updateOne: vi.fn().mockResolvedValue({ modifiedCount: 1 }), } as any, + criteriaCollection: {} as any, log: vi.fn().mockResolvedValue(undefined), }; }); diff --git a/apps/workers/post-processor/src/handlers/atif-handler.websocket.test.ts b/apps/workers/post-processor/src/handlers/atif-handler.websocket.test.ts index 19c1df8e2..fbe9daffd 100644 --- a/apps/workers/post-processor/src/handlers/atif-handler.websocket.test.ts +++ b/apps/workers/post-processor/src/handlers/atif-handler.websocket.test.ts @@ -53,6 +53,7 @@ describe("AtifHandler WebSocket HAR extraction (real atifact)", () => { }), updateOne: async () => ({ modifiedCount: 1 }), } as any, + criteriaCollection: {} as any, log: async () => undefined, }; }); diff --git a/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts b/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts index 4b328b8e6..0a17a7fe5 100644 --- a/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts +++ b/apps/workers/post-processor/src/handlers/taxonomy-handler.test.ts @@ -6,13 +6,24 @@ import { JudgeInfrastructureError, type JudgeClient } from "shared"; import { TaxonomyHandler } from "./taxonomy-handler.js"; import type { HandlerContext, PostProcessorMessage } from "../types.js"; -function makeCtx(doc: unknown): HandlerContext & { collection: { updateOne: ReturnType } } { +interface CriterionMeta { + id: string; + subject?: "run" | "iteration"; +} + +function makeCtx( + doc: unknown, + metas: CriterionMeta[] = [], +): HandlerContext & { collection: { updateOne: ReturnType } } { return { blobStorage: {} as any, collection: { findOne: vi.fn().mockResolvedValue(doc), updateOne: vi.fn().mockResolvedValue({ modifiedCount: 1 }), } as any, + criteriaCollection: { + find: vi.fn().mockReturnValue({ toArray: vi.fn().mockResolvedValue(metas) }), + } as any, log: vi.fn().mockResolvedValue(undefined), } as any; } @@ -51,44 +62,165 @@ describe("TaxonomyHandler", () => { expect(ctx.collection.updateOne).not.toHaveBeenCalled(); }); - it("writes judge criteriaResults to each turn's observationResults and sends atifUrl", async () => { - judge.evaluate.mockResolvedValue({ - passed: true, - feedback: "ok", - criteriaResults: [{ criterionId: "dep_removed", passed: true, feedback: "left-pad removed", evaluated: true }], - }); - const ctx = makeCtx({ - _id: "req-1", - observations: ["dep_removed"], - run: { - turns: [{ iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }], - }, + describe("subject:iteration", () => { + it("writes judge criteriaResults to each turn's observationResults and sends atifUrl", async () => { + judge.evaluate.mockResolvedValue({ + passed: true, + feedback: "ok", + criteriaResults: [{ criterionId: "uses_ts", passed: true, feedback: "tsconfig present", evaluated: true }], + }); + const ctx = makeCtx( + { + _id: "req-1", + observations: ["uses_ts"], + run: { turns: [{ iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }] }, + }, + [{ id: "uses_ts", subject: "iteration" }], + ); + + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + + expect(judge.evaluate).toHaveBeenCalledWith( + expect.objectContaining({ snapshotUrl: "s://1", criteria: ["uses_ts"], atifUrl: "a://1" }), + ); + const call = judge.evaluate.mock.calls[0][0]; + expect(call.gate).toBeUndefined(); + expect(call.toolCallsUrl).toBeUndefined(); + expect(call.atifUrls).toBeUndefined(); + expect(ctx.collection.updateOne).toHaveBeenCalledWith( + { _id: "req-1", "run.turns.iteration": 1 }, + { $set: { "run.turns.$.observationResults": [{ criterionId: "uses_ts", passed: true, feedback: "tsconfig present", evaluated: true }] } }, + ); }); - await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + it("skips iterations without a snapshotUrl", async () => { + judge.evaluate.mockResolvedValue({ passed: true, feedback: "", criteriaResults: [] }); + const ctx = makeCtx( + { _id: "req-1", observations: ["c1"], run: { turns: [{ iteration: 1 }] } }, + [{ id: "c1", subject: "iteration" }], + ); + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + expect(judge.evaluate).not.toHaveBeenCalled(); + }); - expect(judge.evaluate).toHaveBeenCalledWith( - expect.objectContaining({ snapshotUrl: "s://1", criteria: ["dep_removed"], atifUrl: "a://1" }), - ); - const call = judge.evaluate.mock.calls[0][0]; - expect(call.gate).toBeUndefined(); - expect(call.toolCallsUrl).toBeUndefined(); - expect(ctx.collection.updateOne).toHaveBeenCalledWith( - { _id: "req-1", "run.turns.iteration": 1 }, - { $set: { "run.turns.$.observationResults": [{ criterionId: "dep_removed", passed: true, feedback: "left-pad removed", evaluated: true }] } }, - ); + it("throws on judge infrastructure error so the handler is marked failed", async () => { + judge.evaluate.mockRejectedValue(new JudgeInfrastructureError("down", { httpStatus: 503 })); + const ctx = makeCtx( + { _id: "req-1", observations: ["c1"], run: { turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }, + [{ id: "c1", subject: "iteration" }], + ); + await expect(new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx)).rejects.toThrow("down"); + }); }); - it("skips iterations without a snapshotUrl", async () => { - judge.evaluate.mockResolvedValue({ passed: true, feedback: "", criteriaResults: [] }); - const ctx = makeCtx({ _id: "req-1", observations: ["c1"], run: { turns: [{ iteration: 1 }] } }); - await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); - expect(judge.evaluate).not.toHaveBeenCalled(); + describe("subject:run", () => { + it("evaluates ONCE over the merged trajectory and writes run-level observationResults", async () => { + judge.evaluate.mockResolvedValue({ + passed: true, + feedback: "ok", + criteriaResults: [ + { criterionId: "dep_added_then_removed", passed: true, feedback: "left-pad added then removed", evaluated: true }, + ], + }); + const ctx = makeCtx( + { + _id: "req-1", + observations: ["dep_added_then_removed"], + run: { + turns: [ + { iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }, + { iteration: 2, snapshotUrl: "s://2", atifUrl: "a://2" }, + ], + }, + }, + [{ id: "dep_added_then_removed", subject: "run" }], + ); + + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + + // One judge call, using the FINAL snapshot + EVERY iteration's atif merged. + expect(judge.evaluate).toHaveBeenCalledTimes(1); + expect(judge.evaluate).toHaveBeenCalledWith( + expect.objectContaining({ + snapshotUrl: "s://2", + criteria: ["dep_added_then_removed"], + atifUrls: ["a://1", "a://2"], + }), + ); + const call = judge.evaluate.mock.calls[0][0]; + expect(call.atifUrl).toBeUndefined(); + // Written at the RUN level, not on any turn. + expect(ctx.collection.updateOne).toHaveBeenCalledWith( + { _id: "req-1" }, + { $set: { "run.observationResults": [{ criterionId: "dep_added_then_removed", passed: true, feedback: "left-pad added then removed", evaluated: true }] } }, + ); + }); + + it("defaults a subject-less observation to run (whole-run evaluation)", async () => { + judge.evaluate.mockResolvedValue({ passed: false, feedback: "no", criteriaResults: [] }); + const ctx = makeCtx( + { + _id: "req-1", + observations: ["legacy_obs"], + run: { turns: [{ iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }] }, + }, + [{ id: "legacy_obs" }], // no subject → defaults to run + ); + + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + + expect(judge.evaluate).toHaveBeenCalledTimes(1); + const call = judge.evaluate.mock.calls[0][0]; + expect(call.atifUrls).toEqual(["a://1"]); + expect(ctx.collection.updateOne).toHaveBeenCalledWith( + { _id: "req-1" }, + expect.objectContaining({ $set: expect.objectContaining({ "run.observationResults": [] }) }), + ); + }); + + it("throws on judge infrastructure error during the run-subject evaluation", async () => { + judge.evaluate.mockRejectedValue(new JudgeInfrastructureError("down", { httpStatus: 503 })); + const ctx = makeCtx( + { _id: "req-1", observations: ["r1"], run: { turns: [{ iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }] } }, + [{ id: "r1", subject: "run" }], + ); + await expect(new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx)).rejects.toThrow("down"); + }); }); - it("throws on judge infrastructure error so the handler is marked failed", async () => { - judge.evaluate.mockRejectedValue(new JudgeInfrastructureError("down", { httpStatus: 503 })); - const ctx = makeCtx({ _id: "req-1", observations: ["c1"], run: { turns: [{ iteration: 1, snapshotUrl: "s://1" }] } }); - await expect(new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx)).rejects.toThrow("down"); + describe("mixed subjects", () => { + it("routes iteration and run observations to their respective evaluations", async () => { + judge.evaluate.mockResolvedValue({ passed: true, feedback: "ok", criteriaResults: [] }); + const ctx = makeCtx( + { + _id: "req-1", + observations: ["iter_obs", "run_obs"], + run: { + turns: [ + { iteration: 1, snapshotUrl: "s://1", atifUrl: "a://1" }, + { iteration: 2, snapshotUrl: "s://2", atifUrl: "a://2" }, + ], + }, + }, + [ + { id: "iter_obs", subject: "iteration" }, + { id: "run_obs", subject: "run" }, + ], + ); + + await new TaxonomyHandler(judge as unknown as JudgeClient).process(msg, ctx); + + // 2 iteration calls (one per turn) + 1 run call = 3 total. + expect(judge.evaluate).toHaveBeenCalledTimes(3); + const iterationCalls = judge.evaluate.mock.calls.filter((c) => c[0].criteria.includes("iter_obs")); + const runCalls = judge.evaluate.mock.calls.filter((c) => c[0].criteria.includes("run_obs")); + expect(iterationCalls).toHaveLength(2); + expect(runCalls).toHaveLength(1); + // The iteration eval only carries the iteration-subject criterion. + expect(iterationCalls[0][0].criteria).toEqual(["iter_obs"]); + // The run eval only carries the run-subject criterion, merged trajectory. + expect(runCalls[0][0].criteria).toEqual(["run_obs"]); + expect(runCalls[0][0].atifUrls).toEqual(["a://1", "a://2"]); + }); }); }); diff --git a/apps/workers/post-processor/src/handlers/taxonomy-handler.ts b/apps/workers/post-processor/src/handlers/taxonomy-handler.ts index 571320317..252be22a3 100644 --- a/apps/workers/post-processor/src/handlers/taxonomy-handler.ts +++ b/apps/workers/post-processor/src/handlers/taxonomy-handler.ts @@ -2,7 +2,7 @@ // Licensed under the MIT License. import { JudgeClient, isRetryableJudgeError } from "shared"; -import type { ConversationTurn, CriterionResult } from "shared"; +import type { ConversationTurn, CriterionResult, CriterionSubject } from "shared"; import type { PostProcessHandler, PostProcessorMessage, HandlerContext } from "../types.js"; /** Minimal shape of the request document fields this handler reads/writes. */ @@ -17,15 +17,30 @@ interface RequestDoc { }; } +/** Minimal projection of a criterion needed to route by evaluation granularity. */ +interface CriterionMeta { + id: string; + subject?: CriterionSubject; +} + /** - * Taxonomy Handler (pp-taxonomy): records per-iteration boolean observations. + * Taxonomy Handler (pp-taxonomy): records boolean observations. + * + * Each selected observation criterion is routed by its `subject` (#1156): * - * For each iteration turn with a `snapshotUrl`, reuses the judge service to - * evaluate the run's selected observation criteria (top-level - * `request.observations`) against the codebase snapshot AND the agent - * trajectory (ATIF) — sending `atifUrl` in place of `toolCallsUrl`. Results are - * written to `turn.observationResults`, mirroring gate `criteriaResults`. Never - * gates, never steers the agent. Empty/absent `observations` → no-op. #1156. + * - **`subject:"iteration"`** — evaluated on each iteration in isolation against + * that turn's snapshot + ATIF; written to `turn.observationResults` (mirrors + * gate `criteriaResults`). + * - **`subject:"run"`** (the observation default) — evaluated ONCE against the + * whole run: the final iteration's snapshot + the MERGED trajectory of every + * iteration's ATIF (`atifUrls`) + the full conversation history; written to the + * run-level `run.observationResults`. This is what makes cross-iteration + * behavior (e.g. a dependency added in one turn and removed in another) visible. + * + * Reuses the judge service; never gates, never steers the agent. Empty/absent + * `observations` → no-op. pp-taxonomy is dispatched exactly once per run (the + * scheduler never sets `message.iteration`), so the run-subject branch runs once + * with no idempotency guard. */ export class TaxonomyHandler implements PostProcessHandler { readonly type = "taxonomy"; @@ -51,6 +66,30 @@ export class TaxonomyHandler implements PostProcessHandler { return; } + // Resolve each selected observation's evaluation granularity. A missing/unknown + // subject defaults to "run" (the observation default), so legacy criteria authored + // before the `subject` field get whole-run evaluation. + const metas = (await ctx.criteriaCollection + .find({ id: { $in: observations } }, { projection: { id: 1, subject: 1 } } as any) + .toArray()) as unknown as CriterionMeta[]; + const subjectById = new Map(metas.map((m) => [m.id, m.subject])); + const iterationObs = observations.filter((id) => subjectById.get(id) === "iteration"); + const runObs = observations.filter((id) => subjectById.get(id) !== "iteration"); + + await this.evaluateIterationObservations(iterationObs, turns, iteration, requestId, ctx); + await this.evaluateRunObservations(runObs, turns, iteration, requestId, ctx); + } + + /** Per-iteration observations → `turn.observationResults` (one judge call per turn). */ + private async evaluateIterationObservations( + iterationObs: string[], + turns: ConversationTurn[], + iteration: number | undefined, + requestId: string, + ctx: HandlerContext, + ): Promise { + if (iterationObs.length === 0) return; + const turnsToProcess = iteration ? turns.filter((t) => t.iteration === iteration) : turns; let processedCount = 0; @@ -63,7 +102,7 @@ export class TaxonomyHandler implements PostProcessHandler { try { const result = await this.judge.evaluate({ snapshotUrl: turn.snapshotUrl, - criteria: observations, + criteria: iterationObs, conversationHistory: turns, ...(turn.atifUrl ? { atifUrl: turn.atifUrl } : {}), requestId, @@ -88,6 +127,73 @@ export class TaxonomyHandler implements PostProcessHandler { } } - await ctx.log("info", `Taxonomy evaluation complete: ${processedCount}/${turnsToProcess.length} iterations processed`); + await ctx.log( + "info", + `Iteration-subject evaluation complete: ${processedCount}/${turnsToProcess.length} iterations processed`, + ); + } + + /** + * Whole-run observations → `run.observationResults` (one judge call over the + * merged trajectory). Skipped when a single-iteration dispatch is requested + * (forward-compat: a whole-run eval needs every turn). Today `message.iteration` + * is never set, so this runs once per run. + */ + private async evaluateRunObservations( + runObs: string[], + turns: ConversationTurn[], + iteration: number | undefined, + requestId: string, + ctx: HandlerContext, + ): Promise { + if (runObs.length === 0) return; + if (iteration) { + await ctx.log( + "info", + `Skipping ${runObs.length} run-subject observation(s): single-iteration dispatch cannot evaluate the whole run`, + ); + return; + } + + const turnsWithSnapshot = turns.filter((t) => t.snapshotUrl); + const finalTurn = turnsWithSnapshot[turnsWithSnapshot.length - 1]; + if (!finalTurn?.snapshotUrl) { + await ctx.log("info", "No turn with a snapshot, skipping run-subject observations"); + return; + } + + // Merge every iteration's ATIF into one whole-run trajectory. + const atifUrls = turns + .map((t) => t.atifUrl) + .filter((u): u is string => typeof u === "string" && u.length > 0); + + try { + const result = await this.judge.evaluate({ + snapshotUrl: finalTurn.snapshotUrl, + criteria: runObs, + conversationHistory: turns, + ...(atifUrls.length > 0 ? { atifUrls } : {}), + requestId, + }); + + const runObservationResults: CriterionResult[] = result.criteriaResults ?? []; + + await ctx.collection.updateOne( + { _id: requestId } as any, + { $set: { "run.observationResults": runObservationResults } } as any, + ); + + await ctx.log( + "info", + `Run-subject evaluation complete: ${runObservationResults.length} whole-run observation(s) recorded over ${atifUrls.length} merged trajectory(ies)`, + ); + } catch (err) { + const errMsg = err instanceof Error ? err.message : String(err); + if (isRetryableJudgeError(err)) { + await ctx.log("error", `Run-subject: judge infrastructure error, failing handler: ${errMsg}`); + throw err; + } + await ctx.log("warn", `Run-subject observation evaluation failed: ${errMsg}`); + } } } diff --git a/apps/workers/post-processor/src/post-processor.ts b/apps/workers/post-processor/src/post-processor.ts index 4b1b6e847..e6b10614a 100644 --- a/apps/workers/post-processor/src/post-processor.ts +++ b/apps/workers/post-processor/src/post-processor.ts @@ -81,6 +81,7 @@ export class PostProcessor extends BaseQueueProcessor { const ctx: HandlerContext = { blobStorage: this.blobStorage, collection: this.collection as any, + criteriaCollection: this.db.collection("criteria"), log, }; diff --git a/apps/workers/post-processor/src/types.ts b/apps/workers/post-processor/src/types.ts index 8765dd926..20e1397d0 100644 --- a/apps/workers/post-processor/src/types.ts +++ b/apps/workers/post-processor/src/types.ts @@ -16,6 +16,9 @@ export interface PostProcessorMessage { export interface HandlerContext { blobStorage: BlobStorage; collection: Collection; + /** The `criteria` collection — lets handlers resolve criterion metadata + * (e.g. observation `subject`/`taxonomyElementId`) by id. #1156. */ + criteriaCollection: Collection; log: (level: LogEvent["level"], msg: string, data?: Record) => Promise; } From 1daf022db138e51a8baec3bbe05fcf830a83e834 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 19:20:12 -0700 Subject: [PATCH 15/17] feat(portal): subject toggle + whole-run observations + steering copy (#1156) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - wizard: Run/Iteration toggle (shown for observations, prefilled from AI suggestedSubject, user-overridable); subject in create payload + reset - kind copy reworded to convey STEERING vs record-only: gate steers the agent (pass/fail feedback shapes its next iteration), observation records evidence never shown to the agent — in both the wizard and the inline editor - RunDetail Taxonomy tab: new Whole-run observations subsection (from run.observationResults) above the per-iteration matrix, grouped by dimension - CriteriaSubjectBadge on list + detail; subject in types + api payloads - Storybook: subject-badge story Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/components/CriteriaBadge.stories.tsx | 16 +++- apps/portal/src/components/CriteriaBadge.tsx | 27 +++++- .../src/components/CriteriaWizardStep1.tsx | 43 ++++++++-- apps/portal/src/hooks/useCriteriaWizard.ts | 19 ++++- apps/portal/src/lib/api.ts | 6 +- apps/portal/src/pages/CriteriaList.tsx | 11 ++- apps/portal/src/pages/CriterionDetail.tsx | 53 ++++++++---- apps/portal/src/pages/RunDetail.tsx | 83 ++++++++++++++++++- apps/portal/src/types.ts | 22 ++++- 9 files changed, 241 insertions(+), 39 deletions(-) diff --git a/apps/portal/src/components/CriteriaBadge.stories.tsx b/apps/portal/src/components/CriteriaBadge.stories.tsx index e851b3aec..bb3c1f8cb 100644 --- a/apps/portal/src/components/CriteriaBadge.stories.tsx +++ b/apps/portal/src/components/CriteriaBadge.stories.tsx @@ -2,7 +2,7 @@ // Licensed under the MIT License. import type { Meta, StoryObj } from "@storybook/react-vite"; -import { CriteriaBadge, CriteriaKindBadge } from "./CriteriaBadge"; +import { CriteriaBadge, CriteriaKindBadge, CriteriaSubjectBadge } from "./CriteriaBadge"; const meta = { title: "Components/CriteriaBadge", @@ -47,3 +47,17 @@ export const KindBadges: Story = {
), }; + +export const SubjectBadges: Story = { + args: { + criterionId: "has_unit_tests", + link: false, + }, + render: () => ( +
+ + + +
+ ), +}; diff --git a/apps/portal/src/components/CriteriaBadge.tsx b/apps/portal/src/components/CriteriaBadge.tsx index d1cc1bbc3..e5536e2bd 100644 --- a/apps/portal/src/components/CriteriaBadge.tsx +++ b/apps/portal/src/components/CriteriaBadge.tsx @@ -5,7 +5,7 @@ import { Link } from "react-router-dom"; import { Badge } from "@/components/ui/badge"; import { cn } from "@/lib/utils"; import { criterionResultStyle, type CriterionResultState } from "@/lib/criteria-result"; -import type { CriterionKind } from "@/types"; +import type { CriterionKind, CriterionSubject } from "@/types"; interface CriteriaBadgeProps { criterionId: string; @@ -75,3 +75,28 @@ export function CriteriaKindBadge({ kind, className }: { kind?: CriterionKind; c ); } + +/** + * Small badge showing an observation's evaluation subject (#1156): + * "Whole run" (subject:"run") vs "Per iteration" (subject:"iteration"). + * Defaults to "run" when unset (the observation default). Intended for + * observation criteria; subject is inert for gates. + */ +export function CriteriaSubjectBadge({ subject, className }: { subject?: CriterionSubject; className?: string }) { + const normalized = subject ?? "run"; + return ( + + {normalized === "run" ? "Whole run" : "Per iteration"} + + ); +} diff --git a/apps/portal/src/components/CriteriaWizardStep1.tsx b/apps/portal/src/components/CriteriaWizardStep1.tsx index c6a182b36..3cda0d8e3 100644 --- a/apps/portal/src/components/CriteriaWizardStep1.tsx +++ b/apps/portal/src/components/CriteriaWizardStep1.tsx @@ -32,6 +32,8 @@ export function CriteriaWizardStep1({ wizard, idPrefix = "" }: CriteriaWizardSte setKind, taxonomyElementId, setTaxonomyElementId, + subject, + setSubject, } = wizard; return ( @@ -105,12 +107,14 @@ export function CriteriaWizardStep1({ wizard, idPrefix = "" }: CriteriaWizardSte - Gate — controls pass/fail iteration flow - Observation — records per-iteration evidence + Gate — steers the agent (pass/fail feedback shapes its next iteration) + Observation — records evidence; never shown to the agent

- Gate criteria drive judge feedback. Observations are recorded after each iteration and never gate the agent. + Gate criteria steer the agent — their pass/fail verdict is fed back into the coding + session and shapes what it does next. Observations are recorded for analysis only: + never shown to the agent and never change its behavior.

@@ -125,11 +129,34 @@ export function CriteriaWizardStep1({ wizard, idPrefix = "" }: CriteriaWizardSte

) : ( - + <> + + + {/* Evaluation subject (observations only) */} +
+ + +

+ Whole-run observations span iterations (e.g. a dependency added then later removed), + evaluated once against the final snapshot plus the merged trajectory. Per-iteration + observations are judged independently on each iteration. +

+
+ )}
); diff --git a/apps/portal/src/hooks/useCriteriaWizard.ts b/apps/portal/src/hooks/useCriteriaWizard.ts index 3033c2d49..4115684c3 100644 --- a/apps/portal/src/hooks/useCriteriaWizard.ts +++ b/apps/portal/src/hooks/useCriteriaWizard.ts @@ -7,7 +7,7 @@ import { api } from "@/lib/api"; import { slugify } from "@/lib/utils"; import { toast } from "sonner"; import { gatesSatisfyInvariant, type GateId } from "@/lib/gates"; -import type { CriterionKind, TaxonomyElementId } from "@/types"; +import type { CriterionKind, CriterionSubject, TaxonomyElementId } from "@/types"; export interface UseCriteriaWizardOptions { /** Pre-populated parent dependency IDs */ @@ -18,6 +18,8 @@ export interface UseCriteriaWizardOptions { lockedGates?: GateId[]; initialKind?: CriterionKind; initialTaxonomyElementId?: TaxonomyElementId; + /** Pre-populated evaluation subject (observations default to "run"). */ + initialSubject?: CriterionSubject; /** Called with the new criterion ID after successful creation */ onSuccess: (id: string) => void; } @@ -28,6 +30,7 @@ export function useCriteriaWizard({ lockedGates, initialKind = "gate", initialTaxonomyElementId, + initialSubject = "run", onSuccess, }: UseCriteriaWizardOptions) { const queryClient = useQueryClient(); @@ -43,6 +46,7 @@ export function useCriteriaWizard({ const [gates, setGates] = useState(initialGates ?? ["select"]); const [kind, setKind] = useState(initialKind); const [taxonomyElementId, setTaxonomyElementId] = useState(initialTaxonomyElementId); + const [subject, setSubject] = useState(initialSubject); // Step 2 fields const [prompt, setPrompt] = useState(""); @@ -143,6 +147,11 @@ export function useCriteriaWizard({ setSuggestedChildren([]); setAcceptedChildren([]); } + // Prefill the evaluation subject from the AI suggestion (observations + // only; the toggle is hidden for gates). User can still override it. + if (data.suggestedSubject) { + setSubject(data.suggestedSubject); + } }, onError: () => { // LLM unavailable — proceed with empty prompt for manual entry @@ -197,8 +206,9 @@ export function useCriteriaWizard({ ...(kind === "gate" ? { gates } : {}), kind, ...(kind === "observation" && taxonomyElementId ? { taxonomyElementId } : {}), + ...(kind === "observation" ? { subject } : {}), }); - }, [id, prompt, dependsOn, gates, kind, taxonomyElementId, createMutation]); + }, [id, prompt, dependsOn, gates, kind, taxonomyElementId, subject, createMutation]); // Regenerate prompt const handleRegenerate = useCallback(() => { @@ -218,12 +228,13 @@ export function useCriteriaWizard({ setGates(initialGates ?? ["select"]); setKind(initialKind); setTaxonomyElementId(initialTaxonomyElementId); + setSubject(initialSubject); setPrompt(""); setAiGenerated(false); setSuggestedParents([]); setSuggestedChildren([]); setAcceptedChildren([]); - }, [initialDependsOn, initialGates, initialKind, initialTaxonomyElementId]); + }, [initialDependsOn, initialGates, initialKind, initialTaxonomyElementId, initialSubject]); return { // State @@ -243,6 +254,8 @@ export function useCriteriaWizard({ setKind, taxonomyElementId, setTaxonomyElementId, + subject, + setSubject, prompt, setPrompt, aiGenerated, diff --git a/apps/portal/src/lib/api.ts b/apps/portal/src/lib/api.ts index 281fad7dd..1cef41b3e 100644 --- a/apps/portal/src/lib/api.ts +++ b/apps/portal/src/lib/api.ts @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. -import type { Run, RunState, CriteriaDocument, CriteriaGraphData, GeneratePromptResponse, AnalysisResponse, PromptFeatureDocument, Report, BulkReportStatus, BulkReportSummary, ReportTemplate, ReportTrigger, ReportTemplateSystemPrompt, KeyDocument, KeyValidationResult, CreateKeyRequest, UpdateKeyRequest, CodingAgent, AgentVersion, McpServerDocument, CreateMcpServerRequest, UpdateMcpServerRequest, BulkResubmitOverrides, Insight, InsightWithReference, TaskPrompt, TaskPromptFeatureExtractionResult, Model, FeatureFlag, SkillDocument, SkillSearchResult, SkillDiscoveryResult, SkillRevisionDocument, CodebaseDocument, CodebaseRevisionDocument, CodebaseSourceType, ExtensionDocument, ExtensionSearchResult, ExtensionVersionInfo, MdpResponse, AccountDocument, CreateAccountRequest, UpdateAccountRequest, ProfileWithVersion, ProfileVersionDocument, ProfileDocument, RunGroup, CursorPaginatedResponse, IterationOp, GateConfig, GateId, PromptType, CriterionKind, TaxonomyElementId } from "@/types"; +import type { Run, RunState, CriteriaDocument, CriteriaGraphData, GeneratePromptResponse, AnalysisResponse, PromptFeatureDocument, Report, BulkReportStatus, BulkReportSummary, ReportTemplate, ReportTrigger, ReportTemplateSystemPrompt, KeyDocument, KeyValidationResult, CreateKeyRequest, UpdateKeyRequest, CodingAgent, AgentVersion, McpServerDocument, CreateMcpServerRequest, UpdateMcpServerRequest, BulkResubmitOverrides, Insight, InsightWithReference, TaskPrompt, TaskPromptFeatureExtractionResult, Model, FeatureFlag, SkillDocument, SkillSearchResult, SkillDiscoveryResult, SkillRevisionDocument, CodebaseDocument, CodebaseRevisionDocument, CodebaseSourceType, ExtensionDocument, ExtensionSearchResult, ExtensionVersionInfo, MdpResponse, AccountDocument, CreateAccountRequest, UpdateAccountRequest, ProfileWithVersion, ProfileVersionDocument, ProfileDocument, RunGroup, CursorPaginatedResponse, IterationOp, GateConfig, GateId, PromptType, CriterionKind, TaxonomyElementId, CriterionSubject } from "@/types"; import { qs } from "./url"; import { recordServerDate } from "./serverClock"; @@ -318,7 +318,7 @@ export const api = { }, /** Create a new criterion */ - createCriterion: (body: { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }): Promise => { + createCriterion: (body: { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId; subject?: CriterionSubject }): Promise => { return request("/criteria", { method: "POST", body: JSON.stringify(body), @@ -326,7 +326,7 @@ export const api = { }, /** Update an existing criterion */ - updateCriterion: (id: string, body: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }): Promise => { + updateCriterion: (id: string, body: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId; subject?: CriterionSubject }): Promise => { return request(`/criteria/${id}`, { method: "PUT", body: JSON.stringify(body), diff --git a/apps/portal/src/pages/CriteriaList.tsx b/apps/portal/src/pages/CriteriaList.tsx index 3c07c9800..1425ff064 100644 --- a/apps/portal/src/pages/CriteriaList.tsx +++ b/apps/portal/src/pages/CriteriaList.tsx @@ -35,7 +35,7 @@ import { type CustomizeColumnsOption, } from "@/components/list-layout"; import { HelpTooltip } from "@/components/HelpTooltip"; -import { CriteriaKindBadge } from "@/components/CriteriaBadge"; +import { CriteriaKindBadge, CriteriaSubjectBadge } from "@/components/CriteriaBadge"; import { TAXONOMY_ELEMENT_METADATA } from "@/types"; const FILTER_KEYS = ["gate", "kind"] as const; @@ -179,9 +179,14 @@ export function CriteriaList() { { id: "kind", header: "Kind", - width: "130px", + width: "180px", hidden: visibility.isHidden("kind"), - cell: (c) => , + cell: (c) => ( +
+ + {c.kind === "observation" && } +
+ ), }, { id: "taxonomy", diff --git a/apps/portal/src/pages/CriterionDetail.tsx b/apps/portal/src/pages/CriterionDetail.tsx index 51ec8b358..274637abe 100644 --- a/apps/portal/src/pages/CriterionDetail.tsx +++ b/apps/portal/src/pages/CriterionDetail.tsx @@ -20,7 +20,7 @@ import { import { Skeleton } from "@/components/ui/skeleton"; import { ArrowLeft, Save, Trash2, Loader2, Sparkles, Check, X, Plus, Download } from "lucide-react"; import { CriteriaPicker } from "@/components/CriteriaPicker"; -import { CriteriaKindBadge } from "@/components/CriteriaBadge"; +import { CriteriaKindBadge, CriteriaSubjectBadge } from "@/components/CriteriaBadge"; import { GateCompatibilityPicker } from "@/components/GateCompatibilityPicker"; import { ObservationTaxonomySelect } from "@/components/ObservationTaxonomySelect"; import { formatDate } from "@/lib/utils"; @@ -33,7 +33,7 @@ import { useCommandEnter } from "@/hooks/useCommandEnter"; import { KbdBadge } from "@/components/KbdBadge"; import { criteriaToExportYaml, downloadAsFile } from "@/lib/criteria-export"; import { toast } from "sonner"; -import { TAXONOMY_ELEMENT_METADATA, type CriterionKind, type TaxonomyElementId } from "@/types"; +import { TAXONOMY_ELEMENT_METADATA, type CriterionKind, type CriterionSubject, type TaxonomyElementId } from "@/types"; export function CriterionDetail() { const { id } = useParams<{ id: string }>(); @@ -55,6 +55,7 @@ export function CriterionDetail() { const [editGates, setEditGates] = useState(undefined); const [editKind, setEditKind] = useState("gate"); const [editTaxonomyElementId, setEditTaxonomyElementId] = useState(undefined); + const [editSubject, setEditSubject] = useState("run"); // AI Suggest state const [aiSuggestOpen, setAiSuggestOpen] = useState(false); @@ -70,7 +71,7 @@ export function CriterionDetail() { }); const updateMutation = useMutation({ - mutationFn: (body: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }) => + mutationFn: (body: { prompt?: string; dependsOn?: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId; subject?: CriterionSubject }) => api.updateCriterion(id!, body), onSuccess: async () => { // Update accepted children to depend on this criterion @@ -133,6 +134,7 @@ export function CriterionDetail() { setEditGates(criterion.gates && criterion.gates.length > 0 ? criterion.gates : ["select"]); setEditKind(criterion.kind ?? "gate"); setEditTaxonomyElementId(criterion.taxonomyElementId); + setEditSubject(criterion.subject ?? ((criterion.kind ?? "gate") === "observation" ? "run" : "iteration")); setAiSuggestOpen(false); setBehaviorInput(""); setSuggestedPrompt(null); @@ -153,6 +155,8 @@ export function CriterionDetail() { ...(editKind === "gate" ? { gates: editGates } : {}), kind: editKind, taxonomyElementId: editKind === "observation" ? editTaxonomyElementId : undefined, + // Subject only applies to observations; gates are always per-iteration. + subject: editKind === "observation" ? editSubject : "iteration", }); }; @@ -466,7 +470,7 @@ export function CriterionDetail() { Kind - Whether this criterion gates iteration flow or records observation evidence. + Gate criteria steer the agent (pass/fail feedback shapes its next iteration); observations only record evidence and are never shown to the agent. @@ -479,28 +483,45 @@ export function CriterionDetail() { - Gate — controls pass/fail iteration flow - Observation — records per-iteration evidence + Gate — steers the agent (pass/fail feedback shapes its next iteration) + Observation — records evidence; never shown to the agent {editKind === "observation" && ( - + <> + +
+ + +
+ )} ) : (
{(criterion.kind ?? "gate") === "observation" && ( - - {criterion.taxonomyElementId - ? TAXONOMY_ELEMENT_METADATA[criterion.taxonomyElementId].label - : "Unclassified"} - + <> + + {criterion.taxonomyElementId + ? TAXONOMY_ELEMENT_METADATA[criterion.taxonomyElementId].label + : "Unclassified"} + + + )}
)} diff --git a/apps/portal/src/pages/RunDetail.tsx b/apps/portal/src/pages/RunDetail.tsx index 2b058d107..235a78866 100644 --- a/apps/portal/src/pages/RunDetail.tsx +++ b/apps/portal/src/pages/RunDetail.tsx @@ -295,6 +295,80 @@ function ObservationResultsPanel({ ); } +/** + * Whole-run observation results (#1156, `subject:"run"`). Each criterion is + * evaluated ONCE against the final snapshot + the merged trajectory across all + * iterations, so these live at run level (`run.observationResults`) rather than + * per turn. Rendered as a subsection above the per-iteration matrix, grouped by + * taxonomy dimension like the matrix. + */ +function WholeRunObservationsPanel({ + results, + criteria, +}: { + results: CriterionResult[] | undefined; + criteria: CriteriaDocument[]; +}) { + const criteriaById = useMemo(() => new Map(criteria.map((criterion) => [criterion.id, criterion])), [criteria]); + const groups = useMemo(() => { + const byDimension = new Map>(); + for (const result of results ?? []) { + const criterion = criteriaById.get(result.criterionId); + const key: ObservationDimensionKey = criterion?.taxonomyElementId ?? "unclassified"; + const existing = byDimension.get(key) ?? []; + existing.push({ result, criterion }); + byDimension.set(key, existing); + } + return Array.from(byDimension.entries()).sort(([a], [b]) => observationDimensionLabel(a).localeCompare(observationDimensionLabel(b))); + }, [results, criteriaById]); + + if (groups.length === 0) return null; + + return ( +
+ {groups.map(([dimension, items]) => ( + + + + + Whole run + + {observationDimensionLabel(dimension)} + + + + {items.map(({ result, criterion }) => ( +
+
+ {result.criterionId} + + {result.evaluated ? ( + result.passed ? : + ) : ( + + )} + {result.evaluated ? (result.passed ? "True" : "False") : "Not evaluated"} + +
+ {criterion?.prompt && ( +

{criterion.prompt}

+ )} +
+

Evidence

+

{result.feedback || "No evidence provided."}

+
+
+ ))} + + + ))} +
+ ); +} + export function RunDetail() { const { id, tab } = useParams<{ id: string; tab?: string }>(); const navigate = useNavigate(); @@ -403,7 +477,9 @@ export function RunDetail() { enabled: !!taskPromptId, }); - const hasObservationResults = (activeRun?.turns ?? []).some((turn) => (turn.observationResults?.length ?? 0) > 0); + const hasObservationResults = + (activeRun?.turns ?? []).some((turn) => (turn.observationResults?.length ?? 0) > 0) || + (activeRun?.observationResults?.length ?? 0) > 0; const hasConfiguredObservations = (run?.observations?.length ?? 0) > 0; // Aggregate enrichment status across the whole post-processing DAG so the // "Enriched" badge waits for pp-taxonomy instead of flipping after pp-atif. @@ -979,7 +1055,7 @@ export function RunDetail() { {hasVideoData && } {(hasConfiguredObservations || hasObservationResults) && ( - Taxonomy {hasObservationResults ? `(${activeRun?.turns?.reduce((count, turn) => count + (turn.observationResults?.length ?? 0), 0) ?? 0})` : ""} + Taxonomy {hasObservationResults ? `(${((activeRun?.turns?.reduce((count, turn) => count + (turn.observationResults?.length ?? 0), 0) ?? 0) + (activeRun?.observationResults?.length ?? 0))})` : ""} )} Logs @@ -1083,9 +1159,10 @@ export function RunDetail() {

Observations

- Per-iteration boolean observations recorded over the codebase and agent trajectory, grouped by taxonomy dimension. Non-gating — they never steer the agent. + Boolean observations recorded over the codebase and agent trajectory, grouped by taxonomy dimension. Non-gating — they never steer the agent.

+ diff --git a/apps/portal/src/types.ts b/apps/portal/src/types.ts index a7a6b2ec2..5460b6408 100644 --- a/apps/portal/src/types.ts +++ b/apps/portal/src/types.ts @@ -29,6 +29,16 @@ export interface ToolCall { export type CriterionKind = "gate" | "observation"; +/** + * Evaluation granularity for a criterion (#1156). + * - `iteration` — evaluated on each iteration in isolation (gates are always this). + * - `run` — evaluated once against the whole run (final snapshot + merged + * trajectory). The default for observations; rejected for gates. + */ +export type CriterionSubject = "run" | "iteration"; + +export const CRITERION_SUBJECTS: readonly CriterionSubject[] = ["run", "iteration"]; + export const TAXONOMY_ELEMENT_IDS = [ "dimension:idiomatic-use", "dimension:dependency-currency", @@ -167,6 +177,14 @@ export interface RunState { * tracks pp-atif and is kept for backward compatibility. */ handlerStatus?: Record; + /** + * Whole-run observation results — one entry per selected `subject:"run"` + * observation criterion, evaluated once against the final snapshot + the + * merged trajectory across all iterations (#1156). Per-iteration + * (`subject:"iteration"`) observations live on `turn.observationResults` + * instead. Empty/absent when no run-subject observations were selected. + */ + observationResults?: CriterionResult[]; } export interface Run { @@ -263,6 +281,7 @@ export interface CriteriaConfig { gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId; + subject?: CriterionSubject; } export interface CriteriaDocument extends CriteriaConfig { @@ -272,7 +291,7 @@ export interface CriteriaDocument extends CriteriaConfig { } export interface CriteriaGraphData { - nodes: Array<{ id: string; prompt: string; dependsOn: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId }>; + nodes: Array<{ id: string; prompt: string; dependsOn: string[]; gates?: GateId[]; kind?: CriterionKind; taxonomyElementId?: TaxonomyElementId; subject?: CriterionSubject }>; edges: Array<{ source: string; target: string }>; } @@ -281,6 +300,7 @@ export interface GeneratePromptResponse { suggestedId: string; suggestedParents: string[]; suggestedChildren: string[]; + suggestedSubject?: CriterionSubject; } // Prompt Feature types From f1b24e9fe8aa4c593bb2379d1e89e74914928ff1 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 19:20:32 -0700 Subject: [PATCH 16/17] =?UTF-8?q?feat(cli):=20subject=20parity=20=E2=80=94?= =?UTF-8?q?=20--subject=20flag=20+=20whole-run=20observations=20(#1156)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - yaml-mappers: round-trip subject - criteria command: --subject on create/update; show subject in list/get/export/import - run-get-action: render whole-run run.observationResults block alongside the per-iteration ones - updated cli-commands snapshot for --subject Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../__snapshots__/cli-commands.test.ts.snap | 2 + apps/cli/src/commands/criteria.ts | 24 ++++-- apps/cli/src/run-get-action.ts | 75 +++++++++++++------ apps/cli/src/utils/yaml-mappers.ts | 5 +- 4 files changed, 76 insertions(+), 30 deletions(-) diff --git a/apps/cli/src/__snapshots__/cli-commands.test.ts.snap b/apps/cli/src/__snapshots__/cli-commands.test.ts.snap index f22265fbc..d16e5ad90 100644 --- a/apps/cli/src/__snapshots__/cli-commands.test.ts.snap +++ b/apps/cli/src/__snapshots__/cli-commands.test.ts.snap @@ -133,6 +133,7 @@ exports[`CLI command registration > full command tree matches snapshot 1`] = ` "--id", "--kind", "--prompt", + "--subject", "--taxonomy", "--url", ], @@ -183,6 +184,7 @@ exports[`CLI command registration > full command tree matches snapshot 1`] = ` "--id", "--kind", "--prompt", + "--subject", "--taxonomy", "--url", ], diff --git a/apps/cli/src/commands/criteria.ts b/apps/cli/src/commands/criteria.ts index 7e890f9ac..807b83dcf 100644 --- a/apps/cli/src/commands/criteria.ts +++ b/apps/cli/src/commands/criteria.ts @@ -46,7 +46,7 @@ criteria process.exit(1); } - const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string }>; + const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string; subject?: string }>; if (items.length === 0) { if (!isMachineReadable(format)) console.log(warnBanner("No criteria found.")); return; @@ -61,6 +61,7 @@ criteria tableFormatter: (c: any) => value(c.id), }, { key: 'kind', label: 'Kind', formatter: (c: any) => c.kind ?? 'gate' }, + { key: 'subject', label: 'Subject', formatter: (c: any) => c.kind === 'observation' ? (c.subject ?? 'run') : '—' }, { key: 'dependsOn', label: 'Deps', formatter: (c: any) => String((c.dependsOn ?? []).length) }, { key: 'gates', label: 'Gates', formatter: (c: any) => formatGateList(c.gates) }, { key: 'prompt', label: 'Prompt', tableFormatter: (c: any) => { @@ -97,7 +98,7 @@ criteria const c = await response.json() as { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; - kind?: string; taxonomyElementId?: string; + kind?: string; taxonomyElementId?: string; subject?: string; dependents: string[]; createdAt: string; updatedAt?: string; }; @@ -106,6 +107,7 @@ criteria { key: 'id', label: 'ID' }, { key: 'prompt', label: 'Prompt' }, { key: 'kind', label: 'Kind', formatter: (item: any) => item.kind ?? 'gate' }, + { key: 'subject', label: 'Subject', formatter: (item: any) => item.kind === 'observation' ? (item.subject ?? 'run') : '(n/a)' }, { key: 'taxonomyElementId', label: 'Taxonomy', formatter: (item: any) => item.taxonomyElementId ?? '(none)' }, { key: 'dependsOn', label: 'Depends On', formatter: (item: any) => (item.dependsOn ?? []).join(', ') || '(none)' }, { key: 'gates', label: 'Gates', formatter: (item: any) => formatGateList(item.gates) }, @@ -129,6 +131,9 @@ criteria } console.log(`${label('Gates:')} ${value(formatGateList(c.gates))}`); console.log(`${label('Kind:')} ${value(c.kind ?? 'gate')}`); + if ((c.kind ?? 'gate') === 'observation') { + console.log(`${label('Subject:')} ${value(c.subject ?? 'run')}`); + } if (c.taxonomyElementId) { console.log(`${label('Taxonomy:')} ${value(c.taxonomyElementId)}`); } @@ -152,6 +157,7 @@ criteria .option("--gates ", "Compatible gates (space/comma separated), or all/* for unrestricted") .option("--kind ", "Criterion kind: gate (default) or observation") .option("--taxonomy ", "Taxonomy element id (observation only), e.g. dimension:dependency-currency") + .option("--subject ", "Evaluation subject (observation only): run (default) or iteration") .option("-u, --url ", "API base URL", getDefaultApiUrl()) .action(async (options) => { try { @@ -166,6 +172,7 @@ criteria if (gates !== undefined) body.gates = gates; if (options.kind) body.kind = options.kind; if (options.taxonomy) body.taxonomyElementId = options.taxonomy; + if (options.subject) body.subject = options.subject; const response = await fetch(`${normalizeUrl(options.url)}/api/v1/criteria`, { method: "POST", @@ -196,6 +203,7 @@ criteria .option("--gates ", "New compatible gates (space/comma separated), or all/* for unrestricted") .option("--kind ", "Criterion kind: gate or observation") .option("--taxonomy ", "Taxonomy element id (observation only)") + .option("--subject ", "Evaluation subject (observation only): run or iteration") .option("-u, --url ", "API base URL", getDefaultApiUrl()) .action(async (options) => { try { @@ -206,9 +214,10 @@ criteria if (gates !== undefined) body.gates = gates; if (options.kind) body.kind = options.kind; if (options.taxonomy) body.taxonomyElementId = options.taxonomy; + if (options.subject) body.subject = options.subject; if (Object.keys(body).length === 0) { - console.error(errorText("Error: provide --prompt, --depends-on, --gates, --kind, and/or --taxonomy")); + console.error(errorText("Error: provide --prompt, --depends-on, --gates, --kind, --taxonomy, and/or --subject")); process.exit(1); } @@ -363,7 +372,7 @@ criteria process.exit(1); } - const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string }>; + const items = await response.json() as Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string; subject?: string }>; if (items.length === 0) { console.error(errorText("No criteria found to export.")); @@ -408,6 +417,9 @@ criteria if (c.taxonomyElementId) { doc.taxonomy_element_id = c.taxonomyElementId; } + if (c.subject && c.kind === 'observation') { + doc.subject = c.subject; + } return doc; }); @@ -460,7 +472,7 @@ criteria } // Parse all criteria from files (supports multi-document YAML) - const allCriteria: Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string }> = []; + const allCriteria: Array<{ id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string; subject?: string }> = []; const parseErrors: string[] = []; for (const file of yamlFiles) { @@ -505,7 +517,7 @@ criteria const deps = (c.dependsOn ?? []).length; const depsStr = deps > 0 ? ` ${dimTimestamp(`(${deps} dep${deps > 1 ? 's' : ''})`)}` : ''; const gatesStr = c.gates ? ` ${dimTimestamp(`[${formatGateList(c.gates)}]`)}` : ''; - const kindStr = c.kind === 'observation' ? ` ${dimTimestamp(``)}` : ''; + const kindStr = c.kind === 'observation' ? ` ${dimTimestamp(``)}` : ''; console.log(` ${value(c.id)}${depsStr}${gatesStr}${kindStr}`); } diff --git a/apps/cli/src/run-get-action.ts b/apps/cli/src/run-get-action.ts index e2a9c9a8b..f98907990 100644 --- a/apps/cli/src/run-get-action.ts +++ b/apps/cli/src/run-get-action.ts @@ -173,8 +173,11 @@ export async function runGetAction(options: RunGetOptions): Promise { } // Observations (kind:"observation" criteria), grouped by taxonomy dimension. + // Whole-run (subject:"run") results live at run level; per-iteration + // (subject:"iteration") results live on each turn. + const runObs = rs?.observationResults ?? []; const turnsWithObs = (rs?.turns ?? []).filter((t) => (t.observationResults?.length ?? 0) > 0); - if (turnsWithObs.length > 0) { + if (runObs.length > 0 || turnsWithObs.length > 0) { // Build criterionId → taxonomyElementId map from the criteria store (best effort). const taxById = new Map(); try { @@ -185,32 +188,58 @@ export async function runGetAction(options: RunGetOptions): Promise { } } catch { /* grouping is best-effort */ } - // detected/evaluated tally per criterion id across iterations. - const tally = new Map(); - for (const t of turnsWithObs) { - for (const r of t.observationResults ?? []) { - const e = tally.get(r.criterionId) ?? { detected: 0, evaluated: 0 }; - e.evaluated += 1; - if (r.passed) e.detected += 1; - tally.set(r.criterionId, e); + const dimLabelOf = (dim: string) => + dim === "unclassified" ? "Unclassified" : (TAXONOMY_ELEMENT_METADATA[dim as TaxonomyElementId]?.label ?? dim); + + console.log(`\n${banner('─── Observations ───')}`); + + // Whole-run observations: one verdict + evidence per criterion. + if (runObs.length > 0) { + const byDimRun = new Map(); + for (const r of runObs) { + const dim = taxById.get(r.criterionId) ?? "unclassified"; + const arr = byDimRun.get(dim) ?? []; + arr.push(r); + byDimRun.set(dim, arr); + } + console.log(` ${label('Whole run')}`); + for (const [dim, results] of byDimRun) { + console.log(` ${label(dimLabelOf(dim))}`); + for (const r of results.sort((a, b) => a.criterionId.localeCompare(b.criterionId))) { + const verdict = r.evaluated ? (r.passed ? successText('true') : errorText('false')) : warnBanner('not evaluated'); + console.log(` ${value(r.criterionId)} — ${verdict}`); + if (r.feedback) console.log(` ${dimTimestamp(r.feedback.replace(/\n/g, ' ').slice(0, 200))}`); + } } } - const byDim = new Map(); - for (const id of tally.keys()) { - const dim = taxById.get(id) ?? "unclassified"; - const arr = byDim.get(dim) ?? []; - arr.push(id); - byDim.set(dim, arr); - } + // Per-iteration observations: detected/evaluated tally per criterion id. + if (turnsWithObs.length > 0) { + const tally = new Map(); + for (const t of turnsWithObs) { + for (const r of t.observationResults ?? []) { + const e = tally.get(r.criterionId) ?? { detected: 0, evaluated: 0 }; + e.evaluated += 1; + if (r.passed) e.detected += 1; + tally.set(r.criterionId, e); + } + } - console.log(`\n${banner('─── Observations ───')}`); - for (const [dim, ids] of byDim) { - const dimLabel = dim === "unclassified" ? "Unclassified" : (TAXONOMY_ELEMENT_METADATA[dim as TaxonomyElementId]?.label ?? dim); - console.log(` ${label(dimLabel)}`); - for (const id of ids.sort()) { - const { detected, evaluated } = tally.get(id)!; - console.log(` ${value(id)} — ${detected}/${evaluated} iterations detected`); + const byDim = new Map(); + for (const id of tally.keys()) { + const dim = taxById.get(id) ?? "unclassified"; + const arr = byDim.get(dim) ?? []; + arr.push(id); + byDim.set(dim, arr); + } + + console.log(` ${label('Per iteration')}`); + for (const [dim, ids] of byDim) { + console.log(` ${label(dimLabelOf(dim))}`); + for (const id of ids.sort()) { + const { detected, evaluated } = tally.get(id)!; + console.log(` ${value(id)} — ${detected}/${evaluated} iterations detected`); + } } } } diff --git a/apps/cli/src/utils/yaml-mappers.ts b/apps/cli/src/utils/yaml-mappers.ts index cad0c23dd..315f18c82 100644 --- a/apps/cli/src/utils/yaml-mappers.ts +++ b/apps/cli/src/utils/yaml-mappers.ts @@ -12,7 +12,7 @@ export function mapYamlCriterion( filename: string, _docIndex: number, options: { includeGates?: boolean } = {}, -): { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string } | null { +): { id: string; prompt: string; dependsOn?: string[]; gates?: GateId[]; kind?: string; taxonomyElementId?: string; subject?: string } | null { const id = doc.id as string | undefined; const prompt = doc.prompt as string | undefined; if (!id || !prompt) return null; @@ -28,6 +28,8 @@ export function mapYamlCriterion( const kind = kindRaw ? String(kindRaw).trim() : undefined; const taxRaw = (doc.taxonomy_element_id ?? doc.taxonomyElementId) as string | undefined; const taxonomyElementId = taxRaw ? String(taxRaw).trim() : undefined; + const subjectRaw = doc.subject as string | undefined; + const subject = subjectRaw ? String(subjectRaw).trim() : undefined; return { id: id.trim(), @@ -36,6 +38,7 @@ export function mapYamlCriterion( ...(gates !== undefined ? { gates } : {}), ...(kind ? { kind } : {}), ...(taxonomyElementId ? { taxonomyElementId } : {}), + ...(subject ? { subject } : {}), }; } From a7cfe9b3a6f46a6a4d075be033ed710b8bac89e3 Mon Sep 17 00:00:00 2001 From: Cedric Vidal Date: Mon, 29 Jun 2026 19:21:11 -0700 Subject: [PATCH 17/17] docs: criterion subject (run|iteration) + run-level observations (#1156) Document the new subject discriminator, the kind-dependent default and gate restriction, run-level run.observationResults storage, and the merged whole-run trajectory evaluation across post-processing.md, app-design.md, criteria-provider.md, and overview.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/architecture/app-design.md | 7 ++--- docs/architecture/criteria-provider.md | 7 +++++ docs/architecture/overview.md | 2 +- docs/architecture/post-processing.md | 37 ++++++++++++++++++++------ 4 files changed, 41 insertions(+), 12 deletions(-) diff --git a/docs/architecture/app-design.md b/docs/architecture/app-design.md index 6f1b528a7..ae557aa14 100644 --- a/docs/architecture/app-design.md +++ b/docs/architecture/app-design.md @@ -146,11 +146,12 @@ Criteria are reusable evaluation rules stored in the database and optionally def - **AI-generated prompts** — natural language behavior descriptions can be converted to evaluation prompts via LLM - **Traits** — reusable labels for filtering and composition (e.g., `has_azure`, `has_node`) - **Gate compatibility** — a `gates: GateId[]` list controls which [gates](#gates--multi-phase-evaluation-pipeline) a criterion may be selected for (empty = all); the list is downward-closed along the DAG -- **Criterion kind** — `kind: "gate" | "observation"` distinguishes criteria that gate the agent from record-only observations. Observation criteria can optionally set `taxonomyElementId` to one of the hard-coded observation dimensions; run detail derives dimension grouping by joining each per-turn `observationResults[].criterionId` back to the criterion metadata. -- **Kind** — `kind: "gate" | "observation"` (default `gate`). **Gate** criteria steer the coding agent through the gate DAG. **Observation** criteria (issue #1156) are record-only: they produce a boolean + evidence per iteration, never feed back to the agent, never pass/fail a gate, and never enter the gate DAG. Dependencies may only reference same-kind criteria. +- **Criterion kind** — `kind: "gate" | "observation"` distinguishes criteria that gate the agent from record-only observations. Observation criteria can optionally set `taxonomyElementId` to one of the hard-coded observation dimensions and a `subject` (`run`|`iteration`) that selects whole-run vs per-iteration evaluation; run detail derives dimension grouping by joining each `observationResults[].criterionId` (per-turn or run-level) back to the criterion metadata. +- **Kind** — `kind: "gate" | "observation"` (default `gate`). **Gate** criteria steer the coding agent through the gate DAG. **Observation** criteria (issue #1156) are record-only: they produce a boolean + evidence, never feed back to the agent, never pass/fail a gate, and never enter the gate DAG. Dependencies may only reference same-kind criteria. - **Taxonomy classification** — observation criteria carry an optional `taxonomyElementId` from a hard-coded enum of the three R&A Readout quality dimensions (`dimension:idiomatic-use`, `dimension:dependency-currency`, `dimension:configuration-correctness`). Classification is static/design-time, not per-run. +- **Evaluation subject** — `subject: "run" | "iteration"` decides the evaluation granularity of an observation. **`run`** (the default for observations) is evaluated **once** against the final snapshot + the merged trajectory across all iterations and stored at run level (`run.observationResults`); **`iteration`** is evaluated per turn and stored on each turn (`turn.observationResults`). Gate criteria are always `iteration` (the API rejects `run` for gates); `subject` is honored only by the `pp-taxonomy` pipeline. The AI authoring assistant auto-detects the subject and the user can override it. -Observation evaluation happens off the coding-agent critical path in the `pp-taxonomy` post-processor (see [post-processing](post-processing.md)). A run carries the chosen `observations: string[]` (observation-criteria ids); each evaluated turn stores `observationResults: CriterionResult[]` (same shape as gate `criteriaResults`). Dimension grouping is derived at read time by joining `criterionId` → criterion `taxonomyElementId`. +Observation evaluation happens off the coding-agent critical path in the `pp-taxonomy` post-processor (see [post-processing](post-processing.md)). A run carries the chosen `observations: string[]` (observation-criteria ids); **iteration-subject** results are stored per turn as `turn.observationResults: CriterionResult[]`, while **run-subject** results are stored once at run level as `run.observationResults: CriterionResult[]` (both the same shape as gate `criteriaResults`). Dimension grouping is derived at read time by joining `criterionId` → criterion `taxonomyElementId`. See [`ENV_VARIABLES.md`](../../scope-mt-app/ENV_VARIABLES.md) for related configuration options. diff --git a/docs/architecture/criteria-provider.md b/docs/architecture/criteria-provider.md index 08e623fbb..28ded0e43 100644 --- a/docs/architecture/criteria-provider.md +++ b/docs/architecture/criteria-provider.md @@ -164,6 +164,13 @@ bypassed or drift between callers: criteria of the same `kind`. This keeps the gate DAG closed under `resolveWithAncestors` (observations never leak into a gate's evaluation set) and lets `kind:"observation"` criteria form their own optional dependency chains. +- **Kind-dependent `subject` default + validation** — observation criteria carry a + `subject: "run" | "iteration"` deciding whole-run vs per-iteration evaluation. When + `subject` is omitted the store applies a **kind-dependent default** (not a flat zod + default): `kind:"observation"` ⇒ `"run"`, `kind:"gate"`/legacy ⇒ `"iteration"`. Gate + criteria are rejected if `subject:"run"` is supplied — `"iteration"` is their only legal + value. The field is threaded through `create`/`update`/`find` and both provider backends + so the `pp-taxonomy` worker and the CLI/Portal can read it. - **Referential integrity on delete** — deleting a criterion that still has dependents is rejected. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index ea118f342..d7db0f416 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -80,7 +80,7 @@ flowchart TB | `workers/coder-acp-claude-code` | Claude Code agent via Agent Client Protocol (ACP) | | `workers/coder-acp-copilot` | GitHub Copilot agent via Agent Client Protocol (ACP) | | `workers/report-generator` | Copilot SDK worker that generates post-run markdown reports | -| `workers/post-processor` | DAG-based post-run handlers: `pp-atif` (trajectory ATIF) and `pp-taxonomy` (per-iteration boolean observations via the judge) | +| `workers/post-processor` | DAG-based post-run handlers: `pp-atif` (trajectory ATIF) and `pp-taxonomy` (boolean observations via the judge, per-iteration or whole-run by criterion `subject`) | | `gateway` | AI Gateway — shared Rust TLS-intercepting proxy with plugin architecture (HAR capture, future: token refresh, rate limiting) | ## Data Flow diff --git a/docs/architecture/post-processing.md b/docs/architecture/post-processing.md index baeed6e44..1ec97444b 100644 --- a/docs/architecture/post-processing.md +++ b/docs/architecture/post-processing.md @@ -58,18 +58,39 @@ The `iteration` query parameter is mandatory for the ATIF endpoints. ATIF files ## Taxonomy Observations (`pp-taxonomy`) -The `pp-taxonomy` handler records **observation criteria** — per-iteration boolean -outcomes + evidence about the codebase and the agent's trajectory, **without gating -or steering** the coding agent (issue #1156). It is a DAG node `dependsOn: [pp-atif]`, -because some observations are only derivable from the trajectory (ATIF), not the -final snapshot. +The `pp-taxonomy` handler records **observation criteria** — boolean outcomes + +evidence about the codebase and the agent's trajectory, **without gating or steering** +the coding agent (issue #1156). It is a DAG node `dependsOn: [pp-atif]`, because some +observations are only derivable from the trajectory (ATIF), not the final snapshot. + +### Evaluation subject (`run` vs `iteration`) + +Each observation criterion carries a **`subject: "run" | "iteration"`** that decides +*how* and *where* it is evaluated: + +- **`subject: "iteration"`** — evaluated on **each iteration in isolation**; the verdict is + stored on that turn (`turn.observationResults`). +- **`subject: "run"`** (the default for observations) — evaluated **once** against the + **whole run**: the final iteration's `snapshotUrl` + the **merged trajectory** (every + turn's ATIF concatenated, tool definitions unioned) + the full conversation history. The + single verdict is stored at **run level** (`run.observationResults`). This is what makes + cross-iteration observations correct — e.g. `dependency_added_then_removed`, where a + package is installed in one iteration and removed in a later one, is invisible to any + single-iteration evaluation but clearly `true` across the merged trajectory. + +Gate criteria are always `subject: "iteration"` (the API rejects `"run"` for gates); +`subject` is honored only by this handler. ### Flow 1. Reads the run's `observations: string[]` (observation-criteria ids chosen at submit). Empty/absent ⇒ no-op, handler `done`. -2. For each iteration with a `snapshotUrl`, calls the existing judge via the shared `JudgeClient` (`POST /api/v1/evaluate`) with `snapshotUrl` + `atifUrl` + the observation-criteria ids — no persona, no gate, no taxonomyElementId (the classifier lives on the criterion). ATIF supersedes per-turn `toolCallsUrl`, so only ATIF is sent. -3. Writes the returned per-criterion `{criterionId, passed, feedback}` to that turn's **`observationResults`** — same shape/storage as gate `criteriaResults`. No new blob, no `run.taxonomyUrl`. -4. Stamps `run.handlerStatus["pp-taxonomy"] = { status: "done", version: N }` and notifies the scheduler. +2. Resolves each selected criterion's `subject` and splits them into **iteration-subject** and **run-subject** buckets. +3. **Iteration-subject:** for each iteration with a `snapshotUrl`, calls the existing judge via the shared `JudgeClient` (`POST /api/v1/evaluate`) with `snapshotUrl` + `atifUrl` + the iteration-subject ids. Writes the returned `{criterionId, passed, feedback}` to that turn's **`turn.observationResults`**. +4. **Run-subject:** evaluates **once** with `snapshotUrl` = final turn's snapshot, `atifUrls` = every turn's `atifUrl` (the judge downloads + merges them into one `AgentTrajectory`), and the full `conversationHistory`. Writes the verdicts to **`run.observationResults`** (run level). + - In all judge calls there is no persona, no gate, no taxonomyElementId (the classifier lives on the criterion). ATIF supersedes per-turn `toolCallsUrl`, so only ATIF is sent. + - Result shape/storage mirrors gate `criteriaResults`. No new blob, no `run.taxonomyUrl`. + - **Runs exactly once per run:** the scheduler dispatches `pp-taxonomy` a single time (it never sets `message.iteration`), so the run-subject branch needs no idempotency guard. +5. Stamps `run.handlerStatus["pp-taxonomy"] = { status: "done", version: N }` and notifies the scheduler. Dimension grouping is **derived at read time** by joining each result's `criterionId` → criterion `taxonomyElementId` (the three R&A Readout dimensions; unset = unclassified). It never feeds back to the agent and never affects pass/fail. (#1053 later extends this worker with a richer scorecard artifact.)