diff --git a/MANIFEST.json b/MANIFEST.json
index 70c4dc5..a8d9f44 100644
--- a/MANIFEST.json
+++ b/MANIFEST.json
@@ -806,56 +806,6 @@
"path": "validation/package-validation.json",
"bytes": 859,
"sha256": "a5f7bdbdd79c1d022542f7735813a225a424ecd035546fbcd3ed4c90727f704e"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/architecture.md",
- "bytes": 4799,
- "sha256": "f79080de899be87d59f67986952f7f776efd98e7f830df42b58c0deca55ac974"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/architecture.mmd",
- "bytes": 4721,
- "sha256": "715d935fbee41f8a886bc0d3f51ca9edf8ff0a29719ef1da433c60e256334e1e"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/architecture.svg",
- "bytes": 11752,
- "sha256": "e572a3a26528007f034692c8bd6d677e17d951a083d535e2366c0ba7f7102882"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/baseline-config.json",
- "bytes": 424,
- "sha256": "8dc6cc0e070c48edc27963cc1683ca5f2311b223c935cf53e488bc3be018f62b"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/optimize.py",
- "bytes": 13384,
- "sha256": "36893a4379655b1d5b4b3e0326b82231a871401203452e6edfd15c429b3890aa"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/README.md",
- "bytes": 4096,
- "sha256": "63cfa30cd08fe085f587c5332af220c6d64eb3e4f6e035fbd69a83f02e343fe4"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/requirements.txt",
- "bytes": 26,
- "sha256": "64f7ce810445d7a889012d5a96ee260dea3c755c9b86941a31cc0d8fe0cb604b"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/sample-test.jsonl",
- "bytes": 517,
- "sha256": "b25636be2a3c25ccdf1b0d6c0195994c0313e922ea21f075ddba904a8d94ba91"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/sample-train.jsonl",
- "bytes": 579,
- "sha256": "611f87d5e6c093d05b44c37008a3f2c0a0685c6fbd0db5c36d8f437ffc1fa773"
- },
- {
- "path": "graph_synthesis/dspy_jev_optimizer/sample-validation.jsonl",
- "bytes": 559,
- "sha256": "f17c96f92fad2c5bf716ec976dc41d70e64be12509f060d8388c4995a86195aa"
}
]
}
diff --git a/graph_synthesis/dspy_jev_optimizer/README.md b/graph_synthesis/dspy_jev_optimizer/README.md
deleted file mode 100644
index 7dec581..0000000
--- a/graph_synthesis/dspy_jev_optimizer/README.md
+++ /dev/null
@@ -1,115 +0,0 @@
-# DSPy × TypeSafe AI Jev prompt optimizer
-
-Starter experiment for using **DSPy as an outer proposal loop** around **TypeSafe AI Jev as the typed decision/evaluation model**.
-
-> This is additive starter code, not a completed empirical result. It does not change or reinterpret the repository's frozen Jev evidence.
-
-## Architecture
-
-- [GitHub-native Mermaid](architecture.md)
-- [Raw Mermaid source](architecture.mmd)
-- [High-definition vector render](architecture.svg)
-
-The Mermaid diagram is the canonical architecture. The SVG is a publication-oriented vector companion of the same five-stage loop. No generative image tooling was used.
-
-## What the loop does
-
-1. Evaluate the incumbent Jev question on TRAIN and retain mistakes plus probabilities.
-2. Give DSPy the task, fixed label keys, current instructions/criteria, and representative TRAIN failures.
-3. DSPy proposes revised Jev `instructions` and `criteria` wording while preserving label keys.
-4. Jev evaluates the candidate on VALIDATION.
-5. The champion gate defaults to **validation accuracy**. Macro-F1 or a calibration-aware composite can be selected explicitly.
-6. Promote only an improving candidate; persist every candidate and metric.
-7. Freeze the search budget.
-8. Evaluate HIDDEN TEST only after the loop is frozen; never feed that result back into optimization.
-
-TypeSafe's current docs define `Choice` questions with `instructions` and `criteria`, returning a choice plus probabilities/confidence. They also recommend decomposing complex judgments into focused typed questions and composing the answers in code:
-- https://docs.typesafe.ai/primitives
-- https://docs.typesafe.ai/introduction/quickstart
-
-DSPy documentation: https://dspy.ai/
-
-## Quick start
-
-```bash
-cd graph_synthesis/dspy_jev_optimizer
-python -m venv .venv
-source .venv/bin/activate # Windows: .venv\Scripts\activate
-python -m pip install -r requirements.txt
-
-export TYPESAFE_API_KEY="..."
-export OPENAI_API_KEY="..." # or credentials for your DSPy proposal model
-export DSPY_PROPOSER_MODEL="openai/gpt-5.4-nano"
-
-python optimize.py \
- --config baseline-config.json \
- --train sample-train.jsonl \
- --validation sample-validation.jsonl \
- --iterations 12 \
- --selection-metric accuracy \
- --output optimizer-output
-```
-
-The optimization command does **not** read the hidden test.
-
-After the budget is frozen:
-
-```bash
-python optimize.py \
- --config optimizer-output/optimized-config.json \
- --train sample-train.jsonl \
- --validation sample-validation.jsonl \
- --iterations 0 \
- --test sample-test.jsonl \
- --output final-audit
-```
-
-With `--iterations 0`, DSPy is not invoked and no candidate is proposed.
-
-## Metrics
-
-Every evaluation records:
-
-- accuracy
-- macro-F1
-- multiclass MCC
-- multiclass Brier score
-- log loss
-- 10-bin expected calibration error
-- a composite quality score
-
-The default champion gate uses accuracy, matching the highest-accuracy objective. For class imbalance use `--selection-metric macro_f1`. The optional composite is:
-
-```text
-0.55 × macro_F1
-+ 0.20 × accuracy
-+ 0.15 × (1 - multiclass_Brier / 2)
-+ 0.10 × normalized_MCC
-```
-
-## Guardrails
-
-- Never optimize on HIDDEN TEST.
-- Preserve the label schema during a run.
-- Cache calls by state + Jev configuration.
-- Record malformed/rejected candidates, not only winners.
-- Pin proposer/Jev versions and dependencies for a paper-quality experiment.
-- Freeze split, metric, search budget, and stopping rule before final evaluation.
-- Repeat across multiple search seeds before attributing gains to the method.
-- Report null and negative results as well as improvements.
-
-## Next research step
-
-Expand the search space from one `Choice` question to a typed Jev decision graph:
-
-```text
-candidate genome
-├── Choice / Noul / Score question types
-├── number of atomic questions
-├── instructions and criteria
-├── composition weights
-├── decision thresholds
-└── abstention / escalation policy
-```
-
-That tests **workflow optimization**, not merely prompt wording, and should be treated as a separate preregistered experiment.
diff --git a/graph_synthesis/dspy_jev_optimizer/architecture.md b/graph_synthesis/dspy_jev_optimizer/architecture.md
deleted file mode 100644
index e5036ae..0000000
--- a/graph_synthesis/dspy_jev_optimizer/architecture.md
+++ /dev/null
@@ -1,106 +0,0 @@
-# DSPy × TypeSafe AI Jev — closed-loop prompt optimization
-
-```mermaid
-%%{init: {"theme":"base","themeVariables":{"background":"#07111F","primaryColor":"#111C2E","primaryTextColor":"#F8FAFC","primaryBorderColor":"#7C3AED","lineColor":"#64748B","secondaryColor":"#0F1B2D","tertiaryColor":"#12233A","fontFamily":"Inter, ui-sans-serif, system-ui, -apple-system, Segoe UI, sans-serif"},"flowchart":{"curve":"basis","htmlLabels":true,"nodeSpacing":34,"rankSpacing":52,"padding":16}}}%%
-flowchart LR
-
- subgraph DATA["1 · Evidence boundary"]
- direction TB
- TRAIN[("TRAIN
failure analysis only")]
- VAL[("VALIDATION
select candidates")]
- TEST[("HIDDEN TEST
one final evaluation")]
- end
-
- subgraph DSPY["2 · DSPy outer optimizer"]
- direction TB
- BASE["Baseline Jev configuration
instructions · criteria · thresholds"]
- ANALYZE["Failure summarizer
errors · uncertainty · confusion pairs"]
- PROPOSE["DSPy proposal model
revise the Jev question
preserve label schema"]
- CAND["Candidate configuration k
instruction + criteria wording"]
- LEDGER[("Experiment ledger
candidate · hashes · scores · cost")]
- end
-
- subgraph JEV["3 · TypeSafe AI Jev execution"]
- direction TB
- STATE["Structured state
record · claim · context · graph evidence"]
- QUESTIONS["Typed question set
Choice · Noul · Score"]
- MODEL(("Jev
System One"))
- OUTPUT["Typed outputs
choice / score / noul
probabilities · confidence"]
- end
-
- subgraph EVAL["4 · Objective + champion gate"]
- direction TB
- METRICS["Evaluation vector
macro-F1 · accuracy · MCC
Brier · log loss · calibration
latency · tokens · cost"]
- FITNESS["Selection objective
accuracy by default; F1/composite optional"]
- GATE{"Beats incumbent
on validation?"}
- CHAMP["Promote candidate
new incumbent"]
- REJECT["Reject candidate
retain evidence"]
- end
-
- subgraph FINAL["5 · Freeze and report"]
- direction TB
- FREEZE["Freeze champion + budget
no more tuning"]
- FINALRUN["Final hidden-test run
never feeds optimizer"]
- ARTIFACTS["Publish artifacts
optimized-config.json
ledger.json · metrics.json
diagram · reproducible code"]
- end
-
- TRAIN --> ANALYZE
- BASE --> PROPOSE
- ANALYZE --> PROPOSE
- PROPOSE --> CAND
- CAND --> QUESTIONS
- TRAIN --> STATE
- VAL --> STATE
- STATE --> MODEL
- QUESTIONS --> MODEL
- MODEL --> OUTPUT
- OUTPUT --> METRICS
- VAL --> METRICS
- METRICS --> FITNESS
- FITNESS --> GATE
- GATE -- "yes" --> CHAMP
- GATE -- "no" --> REJECT
- CHAMP --> LEDGER
- REJECT --> LEDGER
- CHAMP --> ANALYZE
- REJECT --> PROPOSE
- CHAMP -. "budget / convergence" .-> FREEZE
- FREEZE --> TEST
- TEST --> FINALRUN
- FREEZE --> FINALRUN
- FINALRUN --> ARTIFACTS
-
- class TRAIN,VAL data;
- class TEST hidden;
- class BASE,CAND config;
- class ANALYZE,PROPOSE optimizer;
- class STATE,QUESTIONS input;
- class MODEL jev;
- class OUTPUT output;
- class METRICS,FITNESS metric;
- class GATE gate;
- class CHAMP good;
- class REJECT bad;
- class LEDGER ledger;
- class FREEZE,FINALRUN,ARTIFACTS final;
-
- classDef data fill:#0E2A3F,stroke:#38BDF8,stroke-width:2px,color:#E0F2FE;
- classDef hidden fill:#2B173D,stroke:#C084FC,stroke-width:2.5px,color:#FAF5FF;
- classDef config fill:#172554,stroke:#818CF8,stroke-width:2px,color:#EEF2FF;
- classDef optimizer fill:#2E1065,stroke:#A78BFA,stroke-width:2px,color:#F5F3FF;
- classDef input fill:#0F2835,stroke:#22D3EE,stroke-width:2px,color:#ECFEFF;
- classDef jev fill:#3B0764,stroke:#D946EF,stroke-width:3px,color:#FDF4FF;
- classDef output fill:#132E25,stroke:#34D399,stroke-width:2px,color:#ECFDF5;
- classDef metric fill:#3A2A0B,stroke:#FBBF24,stroke-width:2px,color:#FFFBEB;
- classDef gate fill:#402315,stroke:#FB923C,stroke-width:2.5px,color:#FFF7ED;
- classDef good fill:#123524,stroke:#4ADE80,stroke-width:2.5px,color:#F0FDF4;
- classDef bad fill:#3A161B,stroke:#FB7185,stroke-width:2px,color:#FFF1F2;
- classDef ledger fill:#1F2937,stroke:#94A3B8,stroke-width:2px,color:#F8FAFC;
- classDef final fill:#172033,stroke:#F8FAFC,stroke-width:2px,color:#FFFFFF;
-
- style DATA fill:#081827,stroke:#164E63,stroke-width:1px,color:#BAE6FD
- style DSPY fill:#120C25,stroke:#6D28D9,stroke-width:1px,color:#EDE9FE
- style JEV fill:#0B1C25,stroke:#0E7490,stroke-width:1px,color:#CFFAFE
- style EVAL fill:#211907,stroke:#A16207,stroke-width:1px,color:#FEF3C7
- style FINAL fill:#101827,stroke:#475569,stroke-width:1px,color:#E2E8F0
-```
diff --git a/graph_synthesis/dspy_jev_optimizer/architecture.mmd b/graph_synthesis/dspy_jev_optimizer/architecture.mmd
deleted file mode 100644
index fc14193..0000000
--- a/graph_synthesis/dspy_jev_optimizer/architecture.mmd
+++ /dev/null
@@ -1,102 +0,0 @@
-%%{init: {"theme":"base","themeVariables":{"background":"#07111F","primaryColor":"#111C2E","primaryTextColor":"#F8FAFC","primaryBorderColor":"#7C3AED","lineColor":"#64748B","secondaryColor":"#0F1B2D","tertiaryColor":"#12233A","fontFamily":"Inter, ui-sans-serif, system-ui, -apple-system, Segoe UI, sans-serif"},"flowchart":{"curve":"basis","htmlLabels":true,"nodeSpacing":34,"rankSpacing":52,"padding":16}}}%%
-flowchart LR
-
- subgraph DATA["1 · Evidence boundary"]
- direction TB
- TRAIN[("TRAIN
failure analysis only")]
- VAL[("VALIDATION
select candidates")]
- TEST[("HIDDEN TEST
one final evaluation")]
- end
-
- subgraph DSPY["2 · DSPy outer optimizer"]
- direction TB
- BASE["Baseline Jev configuration
instructions · criteria · thresholds"]
- ANALYZE["Failure summarizer
errors · uncertainty · confusion pairs"]
- PROPOSE["DSPy proposal model
revise the Jev question
preserve label schema"]
- CAND["Candidate configuration k
instruction + criteria wording"]
- LEDGER[("Experiment ledger
candidate · hashes · scores · cost")]
- end
-
- subgraph JEV["3 · TypeSafe AI Jev execution"]
- direction TB
- STATE["Structured state
record · claim · context · graph evidence"]
- QUESTIONS["Typed question set
Choice · Noul · Score"]
- MODEL(("Jev
System One"))
- OUTPUT["Typed outputs
choice / score / noul
probabilities · confidence"]
- end
-
- subgraph EVAL["4 · Objective + champion gate"]
- direction TB
- METRICS["Evaluation vector
macro-F1 · accuracy · MCC
Brier · log loss · calibration
latency · tokens · cost"]
- FITNESS["Selection objective
accuracy by default; F1/composite optional"]
- GATE{"Beats incumbent
on validation?"}
- CHAMP["Promote candidate
new incumbent"]
- REJECT["Reject candidate
retain evidence"]
- end
-
- subgraph FINAL["5 · Freeze and report"]
- direction TB
- FREEZE["Freeze champion + budget
no more tuning"]
- FINALRUN["Final hidden-test run
never feeds optimizer"]
- ARTIFACTS["Publish artifacts
optimized-config.json
ledger.json · metrics.json
diagram · reproducible code"]
- end
-
- TRAIN --> ANALYZE
- BASE --> PROPOSE
- ANALYZE --> PROPOSE
- PROPOSE --> CAND
- CAND --> QUESTIONS
- TRAIN --> STATE
- VAL --> STATE
- STATE --> MODEL
- QUESTIONS --> MODEL
- MODEL --> OUTPUT
- OUTPUT --> METRICS
- VAL --> METRICS
- METRICS --> FITNESS
- FITNESS --> GATE
- GATE -- "yes" --> CHAMP
- GATE -- "no" --> REJECT
- CHAMP --> LEDGER
- REJECT --> LEDGER
- CHAMP --> ANALYZE
- REJECT --> PROPOSE
- CHAMP -. "budget / convergence" .-> FREEZE
- FREEZE --> TEST
- TEST --> FINALRUN
- FREEZE --> FINALRUN
- FINALRUN --> ARTIFACTS
-
- class TRAIN,VAL data;
- class TEST hidden;
- class BASE,CAND config;
- class ANALYZE,PROPOSE optimizer;
- class STATE,QUESTIONS input;
- class MODEL jev;
- class OUTPUT output;
- class METRICS,FITNESS metric;
- class GATE gate;
- class CHAMP good;
- class REJECT bad;
- class LEDGER ledger;
- class FREEZE,FINALRUN,ARTIFACTS final;
-
- classDef data fill:#0E2A3F,stroke:#38BDF8,stroke-width:2px,color:#E0F2FE;
- classDef hidden fill:#2B173D,stroke:#C084FC,stroke-width:2.5px,color:#FAF5FF;
- classDef config fill:#172554,stroke:#818CF8,stroke-width:2px,color:#EEF2FF;
- classDef optimizer fill:#2E1065,stroke:#A78BFA,stroke-width:2px,color:#F5F3FF;
- classDef input fill:#0F2835,stroke:#22D3EE,stroke-width:2px,color:#ECFEFF;
- classDef jev fill:#3B0764,stroke:#D946EF,stroke-width:3px,color:#FDF4FF;
- classDef output fill:#132E25,stroke:#34D399,stroke-width:2px,color:#ECFDF5;
- classDef metric fill:#3A2A0B,stroke:#FBBF24,stroke-width:2px,color:#FFFBEB;
- classDef gate fill:#402315,stroke:#FB923C,stroke-width:2.5px,color:#FFF7ED;
- classDef good fill:#123524,stroke:#4ADE80,stroke-width:2.5px,color:#F0FDF4;
- classDef bad fill:#3A161B,stroke:#FB7185,stroke-width:2px,color:#FFF1F2;
- classDef ledger fill:#1F2937,stroke:#94A3B8,stroke-width:2px,color:#F8FAFC;
- classDef final fill:#172033,stroke:#F8FAFC,stroke-width:2px,color:#FFFFFF;
-
- style DATA fill:#081827,stroke:#164E63,stroke-width:1px,color:#BAE6FD
- style DSPY fill:#120C25,stroke:#6D28D9,stroke-width:1px,color:#EDE9FE
- style JEV fill:#0B1C25,stroke:#0E7490,stroke-width:1px,color:#CFFAFE
- style EVAL fill:#211907,stroke:#A16207,stroke-width:1px,color:#FEF3C7
- style FINAL fill:#101827,stroke:#475569,stroke-width:1px,color:#E2E8F0
diff --git a/graph_synthesis/dspy_jev_optimizer/architecture.svg b/graph_synthesis/dspy_jev_optimizer/architecture.svg
deleted file mode 100644
index fb740a1..0000000
--- a/graph_synthesis/dspy_jev_optimizer/architecture.svg
+++ /dev/null
@@ -1,135 +0,0 @@
-
-
\ No newline at end of file
diff --git a/graph_synthesis/dspy_jev_optimizer/baseline-config.json b/graph_synthesis/dspy_jev_optimizer/baseline-config.json
deleted file mode 100644
index e01e685..0000000
--- a/graph_synthesis/dspy_jev_optimizer/baseline-config.json
+++ /dev/null
@@ -1,9 +0,0 @@
-{
- "task": "Classify the relationship between `claim` and `evidence` for graph-edge verification.",
- "instructions": "Which relation does `evidence` have to `claim`?",
- "criteria": {
- "supports": "The evidence directly supports the claim.",
- "refutes": "The evidence directly contradicts the claim.",
- "insufficient": "The evidence is related or plausible but does not establish support or contradiction."
- }
-}
diff --git a/graph_synthesis/dspy_jev_optimizer/optimize.py b/graph_synthesis/dspy_jev_optimizer/optimize.py
deleted file mode 100644
index be383d8..0000000
--- a/graph_synthesis/dspy_jev_optimizer/optimize.py
+++ /dev/null
@@ -1,401 +0,0 @@
-#!/usr/bin/env python3
-"""DSPy-assisted outer-loop optimization for TypeSafe AI Jev Choice questions.
-
-DSPy proposes candidate Jev instructions/criteria from TRAIN failures.
-Jev evaluates each candidate. VALIDATION chooses the incumbent. HIDDEN TEST
-is optional and is read only after the optimization budget is frozen.
-
-Dataset JSONL:
- {"id": "...", "state": {...}, "label": "supports"}
-"""
-from __future__ import annotations
-
-import argparse
-import hashlib
-import json
-import math
-import os
-from dataclasses import asdict, dataclass
-from pathlib import Path
-from typing import Any, Iterable
-
-import dspy
-from typesafe_sdk import Choice, TypeSafeClient
-
-
-@dataclass(frozen=True)
-class JevConfig:
- task: str
- instructions: str
- criteria: dict[str, str]
-
-
-@dataclass
-class Prediction:
- example_id: str
- gold: str
- predicted: str
- probabilities: dict[str, float]
-
-
-class ReviseJevQuestion(dspy.Signature):
- """Propose one precise revision to a TypeSafe Jev Choice question.
-
- Preserve every label key exactly. Improve only the natural-language
- instructions and label definitions. Keep the judgment atomic and do not
- infer or use hidden-test information.
- """
-
- task: str = dspy.InputField()
- label_keys: str = dspy.InputField(desc="JSON array; preserve keys exactly")
- current_instructions: str = dspy.InputField()
- current_criteria_json: str = dspy.InputField()
- training_failures_json: str = dspy.InputField()
- improved_instructions: str = dspy.OutputField()
- improved_criteria_json: str = dspy.OutputField(
- desc="JSON object with exactly the same keys as label_keys"
- )
-
-
-def canonical_json(value: Any) -> str:
- return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
-
-
-def load_jsonl(path: Path) -> list[dict[str, Any]]:
- rows = []
- with path.open(encoding="utf-8") as handle:
- for line_no, line in enumerate(handle, 1):
- if not line.strip():
- continue
- row = json.loads(line)
- if "state" not in row or "label" not in row:
- raise ValueError(f"{path}:{line_no}: expected state and label")
- row.setdefault("id", f"{path.stem}:{line_no}")
- rows.append(row)
- if not rows:
- raise ValueError(f"{path}: dataset is empty")
- return rows
-
-
-def load_config(path: Path) -> JevConfig:
- raw = json.loads(path.read_text(encoding="utf-8"))
- cfg = JevConfig(
- task=str(raw["task"]),
- instructions=str(raw["instructions"]),
- criteria={str(k): str(v) for k, v in raw["criteria"].items()},
- )
- if len(cfg.criteria) < 2:
- raise ValueError("Choice optimization requires at least two criteria")
- return cfg
-
-
-class JsonCache:
- """Persistent cache so repeated candidates do not repay Jev calls."""
-
- def __init__(self, path: Path):
- self.path = path
- self.data = {}
- if path.exists():
- self.data = json.loads(path.read_text(encoding="utf-8"))
-
- def get(self, key: str):
- return self.data.get(key)
-
- def put(self, key: str, value: dict[str, Any]) -> None:
- self.data[key] = value
- self.path.parent.mkdir(parents=True, exist_ok=True)
- self.path.write_text(
- json.dumps(self.data, indent=2, sort_keys=True, ensure_ascii=False) + "\n",
- encoding="utf-8",
- )
-
-
-def prediction_key(state: Any, cfg: JevConfig) -> str:
- payload = canonical_json(
- {
- "state": state,
- "instructions": cfg.instructions,
- "criteria": cfg.criteria,
- }
- ).encode()
- return hashlib.sha256(payload).hexdigest()
-
-
-def jev_predict(
- client: TypeSafeClient,
- state: Any,
- cfg: JevConfig,
- cache: JsonCache,
-) -> tuple[str, dict[str, float]]:
- key = prediction_key(state, cfg)
- cached = cache.get(key)
- if cached is not None:
- return str(cached["choice"]), {
- k: float(v) for k, v in cached["probabilities"].items()
- }
-
- response = client.system_one(
- state=state,
- questions={
- "target": Choice(
- instructions=cfg.instructions,
- criteria=cfg.criteria,
- )
- },
- )
- answer = response.answers["target"]
- probabilities = {
- str(k): float(v) for k, v in dict(answer.probabilities).items()
- }
- cache.put(key, {"choice": str(answer.choice), "probabilities": probabilities})
- return str(answer.choice), probabilities
-
-
-def evaluate(
- client: TypeSafeClient,
- cfg: JevConfig,
- rows: Iterable[dict[str, Any]],
- cache: JsonCache,
-) -> tuple[dict[str, float], list[Prediction]]:
- labels = list(cfg.criteria)
- label_set = set(labels)
- predictions = []
-
- for row in rows:
- gold = str(row["label"])
- if gold not in label_set:
- raise ValueError(f"Unknown gold label {gold!r}; expected {labels}")
- predicted, probabilities = jev_predict(client, row["state"], cfg, cache)
- predictions.append(
- Prediction(str(row["id"]), gold, predicted, probabilities)
- )
-
- n = len(predictions)
- accuracy = sum(p.gold == p.predicted for p in predictions) / n
-
- f1_values = []
- for label in labels:
- tp = sum(p.gold == label and p.predicted == label for p in predictions)
- fp = sum(p.gold != label and p.predicted == label for p in predictions)
- fn = sum(p.gold == label and p.predicted != label for p in predictions)
- precision = tp / (tp + fp) if tp + fp else 0.0
- recall = tp / (tp + fn) if tp + fn else 0.0
- f1_values.append(
- 2 * precision * recall / (precision + recall)
- if precision + recall
- else 0.0
- )
- macro_f1 = sum(f1_values) / len(f1_values)
-
- brier = 0.0
- log_loss = 0.0
- eps = 1e-12
- for pred in predictions:
- for label in labels:
- p = max(0.0, min(1.0, float(pred.probabilities.get(label, 0.0))))
- y = 1.0 if pred.gold == label else 0.0
- brier += (p - y) ** 2
- gold_p = max(eps, min(1.0, float(pred.probabilities.get(pred.gold, 0.0))))
- log_loss -= math.log(gold_p)
- brier /= n
- log_loss /= n
-
- true_counts = {label: sum(p.gold == label for p in predictions) for label in labels}
- pred_counts = {
- label: sum(p.predicted == label for p in predictions) for label in labels
- }
- correct = sum(p.gold == p.predicted for p in predictions)
- numerator = correct * n - sum(
- pred_counts[label] * true_counts[label] for label in labels
- )
- left = n * n - sum(v * v for v in pred_counts.values())
- right = n * n - sum(v * v for v in true_counts.values())
- denominator = math.sqrt(max(0.0, left * right))
- mcc = numerator / denominator if denominator else 0.0
-
- bins = [[] for _ in range(10)]
- for pred in predictions:
- confidence = max(pred.probabilities.values(), default=0.0)
- index = min(9, int(max(0.0, min(0.999999, confidence)) * 10))
- bins[index].append((confidence, 1.0 if pred.gold == pred.predicted else 0.0))
- ece = 0.0
- for bucket in bins:
- if bucket:
- mean_conf = sum(x[0] for x in bucket) / len(bucket)
- mean_acc = sum(x[1] for x in bucket) / len(bucket)
- ece += len(bucket) / n * abs(mean_conf - mean_acc)
-
- brier_quality = max(0.0, 1.0 - brier / 2.0)
- mcc_quality = (mcc + 1.0) / 2.0
- composite = (
- 0.55 * macro_f1
- + 0.20 * accuracy
- + 0.15 * brier_quality
- + 0.10 * mcc_quality
- )
-
- return {
- "accuracy": accuracy,
- "macro_f1": macro_f1,
- "mcc": mcc,
- "brier": brier,
- "log_loss": log_loss,
- "ece": ece,
- "composite": composite,
- "n": n,
- }, predictions
-
-
-def failure_payload(predictions: list[Prediction], limit: int) -> str:
- failures = [p for p in predictions if p.gold != p.predicted]
- failures.sort(
- key=lambda p: p.probabilities.get(p.predicted, 0.0),
- reverse=True,
- )
- payload = [
- {
- "id": p.example_id,
- "gold": p.gold,
- "predicted": p.predicted,
- "probabilities": p.probabilities,
- }
- for p in failures[:limit]
- ]
- return json.dumps(payload, indent=2, sort_keys=True)
-
-
-def parse_candidate(base: JevConfig, proposal: Any) -> JevConfig:
- instructions = str(proposal.improved_instructions).strip()
- if not instructions:
- raise ValueError("DSPy returned empty instructions")
-
- raw = json.loads(str(proposal.improved_criteria_json))
- criteria = {str(k): str(v).strip() for k, v in raw.items()}
- if set(criteria) != set(base.criteria):
- raise ValueError("DSPy changed the fixed label schema")
- if any(not value for value in criteria.values()):
- raise ValueError("DSPy returned an empty criterion")
- criteria = {key: criteria[key] for key in base.criteria}
- return JevConfig(base.task, instructions, criteria)
-
-
-def save_json(path: Path, value: Any) -> None:
- path.parent.mkdir(parents=True, exist_ok=True)
- path.write_text(
- json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n",
- encoding="utf-8",
- )
-
-
-def main() -> None:
- parser = argparse.ArgumentParser(description=__doc__)
- parser.add_argument("--config", type=Path, required=True)
- parser.add_argument("--train", type=Path, required=True)
- parser.add_argument("--validation", type=Path, required=True)
- parser.add_argument("--test", type=Path)
- parser.add_argument("--iterations", type=int, default=12)
- parser.add_argument(
- "--selection-metric",
- choices=("accuracy", "macro_f1", "composite"),
- default="accuracy",
- )
- parser.add_argument("--failure-sample", type=int, default=20)
- parser.add_argument("--output", type=Path, default=Path("optimizer-output"))
- parser.add_argument("--cache", type=Path, default=Path(".cache/jev-dspy.json"))
- parser.add_argument(
- "--proposer-model",
- default=os.getenv("DSPY_PROPOSER_MODEL", "openai/gpt-5.4-nano"),
- )
- args = parser.parse_args()
-
- if args.iterations < 0:
- parser.error("--iterations must be >= 0")
-
- baseline = load_config(args.config)
- train_rows = load_jsonl(args.train)
- validation_rows = load_jsonl(args.validation)
- cache = JsonCache(args.cache)
- jev = TypeSafeClient()
-
- revise = None
- if args.iterations:
- dspy.configure(lm=dspy.LM(args.proposer_model))
- revise = dspy.Predict(ReviseJevQuestion)
-
- incumbent = baseline
- incumbent_validation, _ = evaluate(jev, incumbent, validation_rows, cache)
- ledger = [{
- "iteration": 0,
- "accepted": True,
- "kind": "baseline",
- "config": asdict(incumbent),
- "validation": incumbent_validation,
- }]
-
- for iteration in range(1, args.iterations + 1):
- train_metrics, train_predictions = evaluate(
- jev, incumbent, train_rows, cache
- )
- assert revise is not None
- proposal = revise(
- task=incumbent.task,
- label_keys=json.dumps(list(incumbent.criteria)),
- current_instructions=incumbent.instructions,
- current_criteria_json=json.dumps(incumbent.criteria, indent=2),
- training_failures_json=failure_payload(
- train_predictions, args.failure_sample
- ),
- )
-
- try:
- candidate = parse_candidate(incumbent, proposal)
- candidate_validation, _ = evaluate(
- jev, candidate, validation_rows, cache
- )
- accepted = (
- candidate_validation[args.selection_metric]
- > incumbent_validation[args.selection_metric]
- )
- row = {
- "iteration": iteration,
- "accepted": accepted,
- "train_incumbent": train_metrics,
- "candidate": asdict(candidate),
- "validation": candidate_validation,
- }
- if accepted:
- incumbent = candidate
- incumbent_validation = candidate_validation
- except Exception as exc:
- row = {
- "iteration": iteration,
- "accepted": False,
- "error": f"{type(exc).__name__}: {exc}",
- }
-
- ledger.append(row)
- save_json(args.output / "ledger.json", ledger)
- save_json(args.output / "optimized-config.json", asdict(incumbent))
- save_json(args.output / "validation-metrics.json", incumbent_validation)
- print(
- f"iteration={iteration:02d} accepted={row['accepted']} "
- f"selection_metric={args.selection_metric} "
- f"best={incumbent_validation[args.selection_metric]:.6f}"
- )
-
- # Deliberately outside the search loop.
- if args.test:
- test_rows = load_jsonl(args.test)
- test_metrics, test_predictions = evaluate(
- jev, incumbent, test_rows, cache
- )
- save_json(args.output / "test-metrics.json", test_metrics)
- save_json(
- args.output / "test-predictions.json",
- [asdict(p) for p in test_predictions],
- )
- print("FINAL_HIDDEN_TEST=" + json.dumps(test_metrics, sort_keys=True))
-
-
-if __name__ == "__main__":
- main()
diff --git a/graph_synthesis/dspy_jev_optimizer/requirements.txt b/graph_synthesis/dspy_jev_optimizer/requirements.txt
deleted file mode 100644
index 7edc6d7..0000000
--- a/graph_synthesis/dspy_jev_optimizer/requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-dspy>=3.3,<4
-typesafe-sdk
diff --git a/graph_synthesis/dspy_jev_optimizer/sample-test.jsonl b/graph_synthesis/dspy_jev_optimizer/sample-test.jsonl
deleted file mode 100644
index 56a193f..0000000
--- a/graph_synthesis/dspy_jev_optimizer/sample-test.jsonl
+++ /dev/null
@@ -1,3 +0,0 @@
-{"id":"te-1","state":{"claim":"The parser improved exact-match accuracy.","evidence":"Exact match increased from 62.4% to 68.9% under the preregistered test."},"label":"supports"}
-{"id":"te-2","state":{"claim":"The treatment shortened hospital stay.","evidence":"Median stay was 5.8 days for treatment and 5.2 days for control."},"label":"refutes"}
-{"id":"te-3","state":{"claim":"The algorithm is robust to distribution shift.","evidence":"The paper reports only in-distribution evaluation."},"label":"insufficient"}
diff --git a/graph_synthesis/dspy_jev_optimizer/sample-train.jsonl b/graph_synthesis/dspy_jev_optimizer/sample-train.jsonl
deleted file mode 100644
index ae3f805..0000000
--- a/graph_synthesis/dspy_jev_optimizer/sample-train.jsonl
+++ /dev/null
@@ -1,3 +0,0 @@
-{"id":"tr-1","state":{"claim":"The trial reported a statistically significant reduction in systolic blood pressure.","evidence":"Mean systolic blood pressure fell 8.2 mmHg versus 1.1 mmHg in control; p=0.004."},"label":"supports"}
-{"id":"tr-2","state":{"claim":"The method improved recall over the baseline.","evidence":"Recall was 0.71 for the method and 0.76 for the baseline."},"label":"refutes"}
-{"id":"tr-3","state":{"claim":"The compound causes liver toxicity in humans.","evidence":"The compound produced elevated liver enzymes in a mouse study."},"label":"insufficient"}
diff --git a/graph_synthesis/dspy_jev_optimizer/sample-validation.jsonl b/graph_synthesis/dspy_jev_optimizer/sample-validation.jsonl
deleted file mode 100644
index 01af49d..0000000
--- a/graph_synthesis/dspy_jev_optimizer/sample-validation.jsonl
+++ /dev/null
@@ -1,3 +0,0 @@
-{"id":"va-1","state":{"claim":"The proposed index reduced query latency.","evidence":"Median latency dropped from 43 ms to 18 ms with the proposed index on the held-out workload."},"label":"supports"}
-{"id":"va-2","state":{"claim":"The model was more accurate than the comparator.","evidence":"Accuracy was 87.1% for the model and 89.4% for the comparator."},"label":"refutes"}
-{"id":"va-3","state":{"claim":"The intervention reduces mortality.","evidence":"The study measured symptom severity but did not report mortality outcomes."},"label":"insufficient"}