feat(evaluation)!: add reusable claims and final evidence tracking - #749
Merged
Merged
Conversation
* docs(evaluation): review benchmark science and eval engineering * fix(diagnostics): bind benchmark probes to source contents
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
drewstone
added a commit
that referenced
this pull request
Sep 14, 2026
…747) * fix(promotion): enforce binary regression confidence floors * fix(promotion): preserve indeterminate objective evidence * test(promotion): size continuous fixtures for mean inference * feat(evaluation)!: add reusable claims and final evidence tracking (#749) * docs(evaluation): map benchmark science to eval engineering (#748) * docs(evaluation): review benchmark science and eval engineering * fix(diagnostics): bind benchmark probes to source contents * feat(evaluation)!: add reusable claims and final evidence tracking * fix(evaluation): freeze judges throughout comparisons * fix(ci): build public imports before checking examples * fix(evaluation): declare binary outcomes and simplify evidence checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Campaign results could overweight repeated task variants, disagree with their own gates, and reuse supposedly fresh final data.
This change makes the evaluation claim explicit while keeping development comparisons reusable and selected candidates available after negative or inconclusive final results.
Keep headlines, detailed lift, power diagnostics, and saved provenance on the same declared units.
Use paired decisions appropriate to the measured outcome.
Capture declared inputs and judge callbacks before search; retain failed measurement exposure.
Host execution still owns access isolation and candidate authoring.
Correct outcome selection, direction, calibration, adaptation cohorts, and power at the declared effect.
Research proposals retain hypotheses instead of fabricated expected gains.
/contract.Update public exports, packed-consumer checks, README, guides, and a runnable offline example.
Retain a cited book mapping and an audit of historical self-improvement evidence.
Build before checking examples that import the public package, so clean checkouts resolve current declarations.
These are breaking API and verification changes; the changelog records the affected signatures and formats.
Historical evidence retains its original bytes and claims.
The caller audit resolved 139 of 201 repository default refs; 62 remained uncovered.
Consumers must migrate the documented root names with their dependency upgrade, including consumers that track
latest.Fixture checks establish execution and measurement behavior, not current-model quality gains or cross-domain optimizer effectiveness.
Both CI jobs pass at
b21a1fc214ce46c6b57396c60fd8acde32ec9d69: run 34798762971.Two default-suite TypeScript skips run and pass in the dedicated optimizer step.
The remaining TypeScript skip requires a local transcript corpus.
The Python source-environment skips belong to the separate DSPy and published-GEPA environments.
Additional local validation:
pnpm typecheck,pnpm typecheck:examples,pnpm typecheck:scripts, andpnpm lint.pnpm buildandpnpm verify:package, including packed consumer imports and the canonical encoder gate.The regression reproducer previously returned
shipand lift1for identical zero-quality artifacts.It now returns
holdand lift0, with both arms scored zero and the same evaluator digest.The prior full local run passed 6,061 tests, failed 31, and skipped 7 under heavy shared-host load.
Its 32 error blocks comprise 30 timeouts and two elapsed-time assertions.
A one-worker rerun of all 28 affected files passed 506 tests and failed 10, all timeouts.
The dedicated full suite passes on the current revision; the local failures remain part of the verification record.
Hosted Codex review could not run because the account code-review quota is exhausted.
The local reviews do not satisfy the required independent approval for merging to
main.Stacked on #747 at
ba7e4d9aunder the existing owner's fixture-only handoff.The safety gate and its fixture changes belong to that PR.
Retarget to
mainafter #747 merges; this older base also displays the book review already merged in #748.