feat: DAG-based post-processing + taxonomy readout post-processor

[Cedric Vidal (cedricvidal)]

Original author: Cedric Vidal (@cedricvidal)

Summary

Introduces a DAG-based post-processing pipeline and a new taxonomy ("readout") post-processor that, after every run finishes, uses the GitHub Copilot SDK to generate a structured, schema-validated readout taxonomy JSON (evaluation dimensions, behavioural patterns, cross-profile insights, and ownership-routed action items) and attaches it to the run. It also reworks how post-processing handlers are registered and how report generation is triggered, so the system scales cleanly to N independent post-processors per run.

What's included

Taxonomy post-processor (apps/workers/taxonomy/)

• New worker that consumes a post-processing handler message, fetches the run's ATIF trajectory through the run-scoped API (no direct blob/storage-connection-string dependency), and drives a Copilot SDK session equipped with the readout-generation skill until it emits a taxonomy that passes the shared Zod schema.
• Shared, side-effect-free generateTaxonomy() reused by the worker, a new developer CLI (pnpm taxonomy), and ad-hoc A/B harnesses. The CLI can target any local or integration run and write the validated taxonomy to a file for troubleshooting — keeping CLI ↔ worker parity.
• apps/api/src/routes/requests/taxonomy.ts serves the stored taxonomy at request- and run-scoped endpoints; apps/portal gains a Taxonomy tab on the run detail view.
• Canonical taxonomy schema + tests added to packages/shared.

DAG-based handler orchestration (apps/scheduler/, apps/workers/post-processor/)

• Post-processors are modelled as nodes in a dependency DAG (pp-atif → pp-taxonomy). The scheduler's HandlerDispatcher walks the graph, enqueues each handler when its dependencies are satisfied, and tracks per-handler status.
• Report generation is taken off the DAG. Reports are parameterized (templateId), N-per-run, and trigger-gated, so they don't fit an idempotent per-run DAG node. Instead, the scheduler detects when the handler DAG has fully drained (every registered handler reached a terminal state: done/failed/blocked) and calls the existing POST /api/v1/reports/trigger endpoint, which evaluates each template's trigger and enqueues { reportId } in the shape the report-generator already expects. Exactly-once is guaranteed by an atomic run.reportsTriggeredAt guard with rollback so the poll safety-net can retry.
• YAML-driven handler registration: handlers declare themselves via a handler.yaml registered through a scheduler endpoint at worker-registration time, replacing migration-baked registration. Migrations 018–020 migrate existing services to the handler-DAG model, add a handler-status index, and backfill status.

Default model: gpt-4.1 → gpt-5.4

A/B testing nine Copilot SDK models on a multi-iteration run containing a real defect (the agent lost prior context in iteration 2) showed verdicts ranging 72–100. Weaker models — including the previous default gpt-4.1 — missed the critical failure entirely or rubber-stamped a buggy run with a perfect score, while gpt-5.4 caught the defect at the lowest token cost of the models tested. ENV_VARIABLES.md records the rationale. TAXONOMY_MODEL remains overridable via env.

Validation

• Full Docker docker:dev:copilot end-to-end run: coder → judge → ATIF → taxonomy (gpt-5.4) → schema-valid JSON → uploaded → taxonomyUrl attached → served by the API (200). Output was rich (4 dimensions, 4 patterns, 3 cross-insights, 4 action items) and passed taxonomySchema.safeParse.
• Unit tests added for the scheduler dispatcher (drain detection, exactly-once guard, POST payload, rollback), the taxonomy schema, the API route, handler registration, and queue-processor changes.

Notes

• Docs updated: docs/architecture/post-processing.md, overview.md, db-migrations.md, ENV_VARIABLES.md.
• pp-report is deregistered (no handler.yaml); nothing is deployed yet so no services-collection migration is needed. The scheduler becomes the first automated caller of /reports/trigger.

[Cedric Vidal (cedricvidal)]

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1858

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
1912	1912	0	0	0	0	0	1m58s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #239

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3299	3299	0	0	0	0	0	1m27s

🎉 All tests passed!

Github Test Reporter

[Cedric Vidal (cedricvidal)]

Historical recovery reference: the readout implementation carried forward from this branch is superseded by the canonical attempt-scoped Experiment readout in #1513 and its successor layers in native stack #1518. This PR remains open and unchanged for recovery/history.