feat(evaluation)!: add reusable claims and enforce promotion floors - #747
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
drewstone
changed the title
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review The fixture-only follow-up at |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
) * docs(evaluation): map benchmark science to eval engineering (#748) * docs(evaluation): review benchmark science and eval engineering * fix(diagnostics): bind benchmark probes to source contents * feat(evaluation)!: add reusable claims and final evidence tracking * fix(evaluation): freeze judges throughout comparisons * fix(ci): build public imports before checking examples
drewstone
changed the title
|
@codex review Please review the combined change against main at |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 participant
Campaign results could overweight repeated task variants, disagree with their own gates, and reuse supposedly fresh final data.
This change makes the evaluation claim explicit while keeping development comparisons reusable and selected candidates available after negative or inconclusive final results.
Keep headlines, detailed lift, power diagnostics, and saved provenance on the same declared units.
Use paired decisions appropriate to the measured outcome.
Apply promotion floors to the deciding interval, so inconclusive safety evidence cannot authorize release.
Accept a declared binary scale so zero-error metrics can use supported inference.
Explain an unresolved floor without claiming an observed regression.
Capture declared inputs and judge callbacks before search; retain failed measurement exposure.
Host execution still owns access isolation and candidate authoring.
Remove the duplicate reservation on proposer-based self-improvement.
Correct outcome selection, direction, calibration, adaptation cohorts, and power at the declared effect.
Research proposals retain hypotheses instead of fabricated expected gains.
/contract.Update public exports, packed-consumer checks, README, guides, and a runnable offline example.
Retain a cited book mapping and an audit of historical self-improvement evidence.
Build before checking examples that import the public package, so clean checkouts resolve current declarations.
These are breaking API and verification changes; the changelog records the affected signatures and formats.
Historical evidence retains its original bytes and claims.
The caller audit resolved 139 of 201 repository default refs; 62 remained uncovered.
Consumers must migrate the documented root names with their dependency upgrade, including consumers that track
latest.Fixture checks establish execution and measurement behavior, not current-model quality gains or cross-domain optimizer effectiveness.
Validation:
c111c088aca5b6253e48517ed90a7ad84e5861d6: https://github.com/tangle-network/agent-eval/actions/runs/34813213542.TypeScript: 6,121 passed, 3 skipped across 412 files; all three dedicated optimizer integration tests also pass.
Python source: 208 passed, 4 skipped; published GEPA: 45 passed, 1 skipped; DSPy: 18 passed.
All typechecks, lint, build, packed-package checks, and Python distribution checks pass.
Final-evidence and self-improvement integration checks pass 68 tests across six suites.
Exact agreement now preserves the declared score; incompatible averages remain rejected.
All 14 independent review probes pass.
Gate explanations distinguish an unresolved regression floor from a demonstrated loss.
The full diff adds 3,032 production TypeScript lines and removes 1,223: 1,809 net.
The optional final-evidence and evaluator-admission modules total 648 production lines and reuse existing storage and statistics.
Ordinary evaluations require neither optional feature.
These checks establish measurement behavior; they do not establish general optimizer effectiveness.
GitHub still requires one independent approving review.
The hosted reviewer declined this revision because its code-review quota is exhausted; its earlier review covers only
e723098.The normal merge attempt was refused by branch protection; no bypass was used.
This PR delivers the promotion-floor fix and the evaluation-integrity implementation to
main.#749 merged the implementation into this PR's branch; all changes remain included here.