Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
aac9c68
feat(taxonomy): restack readout handler on DAG foundation
cedricvidal Oct 12, 2026
e459f27
feat(readout): generate submission-level reports
cedricvidal Oct 13, 2026
be50032
feat(readout): expose submission reports in UI and CLI
cedricvidal Oct 13, 2026
ca5ffdd
docs(readout): document submission report generation
cedricvidal Oct 13, 2026
2401785
fix(readout): harden local container startup
cedricvidal Oct 13, 2026
f2ff607
fix(portal): use generic report terminology
cedricvidal Oct 13, 2026
6458468
fix(readout): include task criteria in evidence
cedricvidal Oct 13, 2026
a440cbb
fix(readout): attach all criteria to run evidence
cedricvidal Oct 13, 2026
0d8723a
fix(reports): align submission report APIs
cedricvidal Oct 13, 2026
ffc4c22
feat: stream exact submission attempt logs
cedricvidal Oct 13, 2026
8181fac
feat: refine submission report experience
cedricvidal Oct 13, 2026
d1a195c
docs: describe attempt-scoped submission reports
cedricvidal Oct 13, 2026
cd97f8f
fix: scope submission reports by type
cedricvidal Oct 13, 2026
d373d41
feat: adapt readout stack to experiments
cedricvidal Oct 16, 2026
af99807
fix(readout): align experiment validation contracts
cedricvidal Oct 16, 2026
55efa81
merge: update experiment readout foundation
cedricvidal Oct 16, 2026
8cb8300
merge: remove experiment tags from foundation
cedricvidal Oct 16, 2026
fb80f62
test(cli): refresh tag-free experiment snapshot
cedricvidal Oct 16, 2026
75f0b3a
Merge remote-tracking branch 'origin/cedricvidal-experiment-foundatio…
cedricvidal Oct 16, 2026
a9598cf
Merge final Experiment portal UX
cedricvidal Oct 16, 2026
a74a950
Merge remote-tracking branch 'origin/cedricvidal-experiment-foundatio…
cedricvidal Oct 16, 2026
afe6e85
feat(portal): organize experiment detail reports
cedricvidal Oct 16, 2026
0e1a4d7
fix(portal): stabilize experiment report states
cedricvidal Oct 16, 2026
89a5c96
Merge experiment foundation into readout
cedricvidal Oct 17, 2026
408b59f
Merge experiment foundation launcher into readout
cedricvidal Oct 17, 2026
6f33318
Merge corrected experiment foundation into readout
cedricvidal Oct 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ apps/
coder-acp-copilot/ # GitHub Copilot worker (ACP SDK)
coder-acp-claude-code/ # Claude Code worker (ACP SDK)
report-generator/ # Post-run report generation
readout/ # Submission-level generated readout
model-scanners/ # Feature detection for Copilot and Anthropic models
version-checkers/ # Poll for new agent/tool releases
key-updaters/ # GitHub auth cookie management for VS Code Web
Expand Down Expand Up @@ -69,6 +70,7 @@ Each worker implements the same queue-processor interface but adapts a different
| `coder-acp-copilot` | GitHub Copilot | ACP SDK v0.14.1 |
| `coder-acp-claude-code` | Claude Code | ACP SDK v0.13.1 |
| `report-generator` | — | Copilot SDK |
| `readout` | — | Copilot SDK |

Workers consume tasks from Azure Storage Queues (named `queue-<worker-name>`) and write results to MongoDB and Blob Storage. Each has its own Dockerfile and docker-compose profile.

Expand Down
31 changes: 29 additions & 2 deletions ENV_VARIABLES.md
Original file line number Diff line number Diff line change
Expand Up @@ -889,6 +889,33 @@ Timeout for the Copilot SDK session used by the report-generator worker. If the

Git commit hash embedded in reporter metadata. Automatically set during CI/CD builds. Used to track which version of the report-generator produced a given report.

## Experiment Readout Handler Configuration

### AZURE_STORAGE_QUEUE_READOUT
**Default:** `pp-readout-queue`
**Type:** string

Azure Storage Queue name for Experiment readout jobs. The coordinator enqueues
an exact execution message containing `projectId`, `experimentId`,
`experimentExecutionId`, handler identity/version, claim token, and execution
version only after `pp-observations` is done for every exact request/run pair.

### READOUT_MODEL
**Default:** `gpt-5.4`
**Type:** string

The Copilot SDK model used by the dedicated `pp-readout` worker to synthesize a
schema-validated structured Experiment readout. Markdown is derived from the
validated structured object.

### READOUT_SESSION_TIMEOUT_MS
**Default:** `300000` (5 minutes)
**Type:** integer (milliseconds)

Timeout for one readout-generation attempt. Exhausted generation or schema
validation marks the exact current Experiment execution's `pp-readout` handler
as failed; historical and superseded executions remain immutable.

## Scheduler Configuration

### SCHEDULER_POLL_INTERVAL_MS
Expand Down Expand Up @@ -936,13 +963,13 @@ TTL for the DAG dispatcher's in-memory cache of handler topology (the `services`
**Default:** `post-processor-queue`
**Type:** string

Azure Storage Queue name used by both the scheduler (to enqueue post-processing work) and the post-processor worker (to dequeue). Must match between the two services. The `pp-taxonomy` observation handler shares this queue.
Azure Storage Queue name used by both the scheduler (to enqueue post-processing work) and the post-processor worker (to dequeue). Must match between the two services. The `pp-observations` handler shares this queue.

### JUDGE_URL
**Default:** falls back to `JUDGE_SERVICE_URL`, then `http://localhost:3002`
**Type:** URL string

Base URL of the judge service used by the `pp-taxonomy` post-process handler to evaluate observation criteria per iteration (snapshot + ATIF trajectory → boolean + evidence). The handler reuses the shared `JudgeClient`, so `JUDGE_CLIENT_TIMEOUT` / `JUDGE_CLIENT_RETRIES` apply. In K8s the post-processor reads `JUDGE_SERVICE_URL` from the `worker-config` ConfigMap (`envFrom`); in docker-compose the `post-processor` service sets `JUDGE_SERVICE_URL: http://judge:80` explicitly (it uses the `common-env` anchor, which — unlike `worker-env` — does not carry it). See issue #1156 and [docs/architecture/post-processing.md](docs/architecture/post-processing.md).
Base URL of the judge service used by the `pp-observations` post-process handler to evaluate observation criteria per iteration (snapshot + ATIF trajectory → boolean + evidence). The handler reuses the shared `JudgeClient`, so `JUDGE_CLIENT_TIMEOUT` / `JUDGE_CLIENT_RETRIES` apply. In K8s the post-processor reads `JUDGE_SERVICE_URL` from the `worker-config` ConfigMap (`envFrom`); in docker-compose the `post-processor` service sets `JUDGE_SERVICE_URL: http://judge:80` explicitly (it uses the `common-env` anchor, which — unlike `worker-env` — does not carry it). See issue #1156 and [docs/architecture/post-processing.md](docs/architecture/post-processing.md).

### SCHEDULER_URL
**Default:** (not set)
Expand Down
Loading
Loading