Skip to content

feat(evaluation)!: add reusable claims and final evidence tracking - #749

Merged
drewstone merged 5 commits into
fix/pareto-safety-floorfrom
feat/evaluation-integrity
Sep 14, 2026
Merged

drewstone merged 5 commits into
fix/pareto-safety-floorfrom
feat/evaluation-integrity

Conversation

@drewstone

@drewstone drewstone commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Campaign results could overweight repeated task variants, disagree with their own gates, and reuse supposedly fresh final data.
This change makes the evaluation claim explicit while keeping development comparisons reusable and selected candidates available after negative or inconclusive final results.

  • Add optional claims and source-unit reporting to the existing improvement and comparison APIs.
    Keep headlines, detailed lift, power diagnostics, and saved provenance on the same declared units.
    Use paired decisions appropriate to the measured outcome.
  • Add optional final-evidence reservation and exposure through the existing locked journal.
    Capture declared inputs and judge callbacks before search; retain failed measurement exposure.
    Host execution still owns access isolation and candidate authoring.
  • Audit actual evaluator controls with simultaneous error bounds and explicit unknowns.
    Correct outcome selection, direction, calibration, adaptation cohorts, and power at the declared effect.
    Research proposals retain hypotheses instead of fabricated expected gains.
  • Bind interval value selectors to experiment seals, remove retired digest readers, and align root types with /contract.
    Update public exports, packed-consumer checks, README, guides, and a runnable offline example.
    Retain a cited book mapping and an audit of historical self-improvement evidence.
  • Run the existing CI jobs for every PR target, including stacked branches.
    Build before checking examples that import the public package, so clean checkouts resolve current declarations.

These are breaking API and verification changes; the changelog records the affected signatures and formats.
Historical evidence retains its original bytes and claims.
The caller audit resolved 139 of 201 repository default refs; 62 remained uncovered.
Consumers must migrate the documented root names with their dependency upgrade, including consumers that track latest.
Fixture checks establish execution and measurement behavior, not current-model quality gains or cross-domain optimizer effectiveness.

Both CI jobs pass at b21a1fc214ce46c6b57396c60fd8acde32ec9d69: run 34798762971.

Check Result
TypeScript suite 6,102 passed; 3 skipped across 412 files
Official optimizer TypeScript integrations 2 passed with source packages; 1 passed with published GEPA
Python client with official GEPA 208 passed; 4 skipped; 1 dependency deprecation warning
Published GEPA compatibility 45 passed; 1 skipped; 1 dependency deprecation warning
DSPy metric compatibility 18 passed
Typechecks, lint, build, packed exports, and Python distributions Passed

Two default-suite TypeScript skips run and pass in the dedicated optimizer step.
The remaining TypeScript skip requires a local transcript corpus.
The Python source-environment skips belong to the separate DSPy and published-GEPA environments.

Additional local validation:

  • pnpm typecheck, pnpm typecheck:examples, pnpm typecheck:scripts, and pnpm lint.
  • pnpm build and pnpm verify:package, including packed consumer imports and the canonical encoder gate.
  • 107 affected campaign tests across six files, including six judge-mutation and class-callback regressions.
  • Public offline candidate search and report export; 178 local links across 14 changed Markdown files.
  • Independent Standards and Spec reviews; both confirmed findings are fixed.

The regression reproducer previously returned ship and lift 1 for identical zero-quality artifacts.
It now returns hold and lift 0, with both arms scored zero and the same evaluator digest.

The prior full local run passed 6,061 tests, failed 31, and skipped 7 under heavy shared-host load.
Its 32 error blocks comprise 30 timeouts and two elapsed-time assertions.
A one-worker rerun of all 28 affected files passed 506 tests and failed 10, all timeouts.
The dedicated full suite passes on the current revision; the local failures remain part of the verification record.

Hosted Codex review could not run because the account code-review quota is exhausted.
The local reviews do not satisfy the required independent approval for merging to main.

Stacked on #747 at ba7e4d9a under the existing owner's fixture-only handoff.
The safety gate and its fixture changes belong to that PR.
Retarget to main after #747 merges; this older base also displays the book review already merged in #748.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@drewstone
drewstone merged commit db03b8e into fix/pareto-safety-floor Sep 14, 2026
2 checks passed
drewstone added a commit that referenced this pull request Sep 14, 2026
…747)

* fix(promotion): enforce binary regression confidence floors

* fix(promotion): preserve indeterminate objective evidence

* test(promotion): size continuous fixtures for mean inference

* feat(evaluation)!: add reusable claims and final evidence tracking (#749)

* docs(evaluation): map benchmark science to eval engineering (#748)

* docs(evaluation): review benchmark science and eval engineering

* fix(diagnostics): bind benchmark probes to source contents

* feat(evaluation)!: add reusable claims and final evidence tracking

* fix(evaluation): freeze judges throughout comparisons

* fix(ci): build public imports before checking examples

* fix(evaluation): declare binary outcomes and simplify evidence checks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant