Repository navigation
Prepare repeated DSPy/Jev comparisons; record offline reruns and live blockers - #32
Conversation
Prepare five-seed live comparisons with separate calibration and held-out panels. Preserve native baseline formulations and immutable research evidence. Record 1092 passing local tests, six skips, eleven successful archived replays, and exploratory saved-output recalibration. Live inference remains blocked by missing mapped proposal credentials and an Actions approval requirement; no DSPy performance gain is claimed.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Final artifact verification: the complete published tree Validation scope remains the c726b66 baseline plus this additive implementation, not a claim that the newer main branch or PR merge result has been tested locally. Live DSPy/Jev comparisons remain unexecuted. The PR is left open; no inference-credential or Actions approval restriction was bypassed. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2db06433d5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - name: Generate report without inventing missing runs | ||
| if: always() | ||
| run: python -B -m graph_synthesis.dspy_benchmark.report --directory '${{ runner.temp }}/comparison' |
There was a problem hiding this comment.
Each matrix leg runs on an isolated runner and invokes the report before any other seed's artifact is available, so even after all five live jobs succeed every generated report sees only its own seed-* directory, reports the other four seeds as not_run, and cannot calculate the promised five-run means or standard deviations. Add a downstream job that downloads all five seed artifacts into one comparison directory and invokes the reporter once.
Useful? React with 👍 / 👎.
| pull_request: | ||
| paths: ['graph_synthesis/dspy_benchmark/**', '.github/workflows/dspy-benchmark.yml'] |
There was a problem hiding this comment.
The benchmark imports core.py and providers.py from graph_synthesis/dspy_jev_optimizer, but this path filter does not trigger the benchmark workflow for changes under that package. Such a PR only runs the optimizer's own tests, so an API or behavioral change that breaks the benchmark can merge unnoticed and surface only during a costly manual live dispatch; include the optimizer directory in this workflow's pull-request paths.
Useful? React with 👍 / 👎.
| proposer_model = provider_model() | ||
| output.mkdir(parents=True) | ||
| if not os.environ.get('TYPESAFE_API_KEY') or (not proposer_model and not baseline_only): | ||
| write_json(output/'status.json', {'status':'blocked_missing_credentials', | ||
| 'typesafe_key_present':bool(os.environ.get('TYPESAFE_API_KEY')), | ||
| 'proposer_model_configured':bool(proposer_model),'live_calls':0}) | ||
| raise RuntimeError('Live benchmark requires TypeSafe and a generative proposal provider') |
There was a problem hiding this comment.
When DSPY_PROPOSER_MODEL is configured but its corresponding provider secret is absent, this guard passes because it checks only that the model string is nonempty. The optimizer then performs the baseline validation and training Jev evaluations before the first DSPy proposal attempts authentication and fails, wasting roughly 60 paid Jev calls per task/seed instead of recording a zero-call credential block. For the explicitly supported workflow providers, validate the credential matching the configured model—or probe the proposer—before starting target inference.
Useful? React with 👍 / 👎.
| if item['status'] in ('completed','completed_baseline_only'): | ||
| rows.extend(read_json(directory/'metrics.json')) |
There was a problem hiding this comment.
If users run the documented --baseline-only mode, this condition loads those partial metrics into the main comparison summary even though no DSPy arm was executed. With only baseline-only runs, summary is nonempty, so the report also suppresses the explicit “No completed live comparison” warning and presents control scores under “Primary test results”; with mixed runs it aggregates controls over more seeds than DSPy arms. Keep partial baseline results separate from the completed-comparison aggregate.
Useful? React with 👍 / 👎.
| def execute(output: Path, seed: int, *, iterations=8, baseline_only=False, model='jev-1.13.0', workers=4): | ||
| if seed not in SEEDS or not 0 <= iterations <= 12: | ||
| raise ValueError('Unregistered seed or iteration budget') |
There was a problem hiding this comment.
The public CLI accepts --iterations 0, but optimize() then freezes the untouched baseline without ever invoking the DSPy proposer; the runner still names both resulting arms dspy_accuracy/dspy_composite and writes status completed. The reporter consequently counts a zero-DSPy-call execution as a completed full DSPy run and can publish its scores as DSPy results. Require at least one iteration for non-baseline runs, or record and report this mode as a distinct partial/no-search execution.
Useful? React with 👍 / 👎.
CompleteDotTech
added
reproducibility
1 participant
Execution status — no live DSPy result claimed
The requested live with/without-DSPy comparison could not start. Preflight run 35355158096 found the explicitly mapped TypeSafe key populated but none of the mapped OpenRouter/OpenAI/Anthropic proposal keys. It stopped before Jev inference. A second diagnostic 35355491970 was marked
action_requiredwith zero jobs. That gate was not bypassed. Broad secret-context diagnostics were removed; live workflows are manual-only and use named credentials.Implemented comparison matrix
Actually executed
3b0a82c5ebd405fb60faf6fc93e7d79d9f61f0f6).Entry point:
DSPY_BENCHMARKS.md. Detailed status:experiments/dspy-comparison-20260918/STATUS.md. The root MANIFEST and prior research results are unchanged. This is an execution package plus honest partial results, not evidence that DSPy improved Jev.