Skip to content

fix(eval): preserve campaign accounting and evaluator revisions - #595

Merged
drewstone merged 2 commits into
mainfrom
fix/eval-cohort-recovery-20260915
Sep 16, 2026
Merged

drewstone merged 2 commits into
mainfrom
fix/eval-cohort-recovery-20260915

Conversation

@drewstone

@drewstone drewstone commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Ensemble judges now receive the complete campaign scoring input, including cost-ledger context, phase, and tags.
Forwarding judgeVersion into Eval invalidates cached campaign scores when evaluator configuration changes.
The trust gate consumes Eval's maintained score contract directly.

The library and both generated application templates adopt the same published execution and evaluation dependencies.
The Interface override is removed.
README, API references, and shipped optimization skills distinguish rater agreement, evaluator accuracy, exploratory evidence, and release evidence.
The scheduler overlap regression declares its intended concurrency explicitly instead of assuming the host has three CPUs.

Validation:

  • Exact-source signoff passed for 6998b8a09862a3fa757e6bc5fc5e4f11080575a8 with the unchanged checked-in gate.
    Run and complete proof artifact.
  • Both shuffled full suites passed: 6,226 passed, 10 skipped each, with seeds 1370982601 and 8392146.
    Frozen install, typecheck, 391 incident-gate tests, build, generated projects, and Knip also passed.
  • Both generated applications passed strict installs, their installed peer contracts, typechecks, tests, and Wrangler deployment dry runs.
    The PR verification artifact retains their complete logs: 4 default-template tests and 15 chat-template tests.
  • Mutation checks showed the paid-accounting and scorer-revision regressions fail when their protected forwarding behavior is removed.
    Forcing the production scheduler to one concurrent task also failed the overlap regression; restored source passed.

The signoff used a dedicated Linux x64 runner with Node 24.18.0, pnpm 11.24.0, four CPUs, and 15 GiB RAM.
KNIP_DISABLE_RAW_TRANSFER=1 disables Knip's optional large-buffer transfer optimization without disabling checks.
The gate configuration hash matches the reviewed source: 24dd467f0b03ece6aaeafed27c560b0763db9221fae3b1e62adbf030123a0623.

An earlier run on the contended local VM passed 6,225 tests and hit one five-second UI-module import timeout.
That unchanged test file subsequently passed all 14 tests locally; the import took 1.3 seconds.
The dedicated runner then passed both complete randomized suites.
The temporary verification workflow stays outside this PR and will not land on main.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@drewstone
drewstone merged commit 9be4761 into main Sep 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant