Skip to content

fix(eval): distinguish pending and completed Judge results - #33

Merged
xiaour merged 3 commits into
xiaour:mainfrom
BigDataDZ:codex/eval-judge-status
Oct 16, 2026
Merged

xiaour merged 3 commits into
xiaour:mainfrom
BigDataDZ:codex/eval-judge-status

Conversation

@BigDataDZ

@BigDataDZ BigDataDZ commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

When an L3 Judge returned malformed text, reflection was degraded but the eval report showed judgePending=0 and the dashboard counted the attempted call as a finished judgment. Keep every degraded outcome pending and add judgeAttempted/judgeCompleted fields while retaining legacy judgeAvailable/judgeRan semantics. Display completed judgments separately from fixture pass rates, and clarify that correctly rejecting a negative fixture also counts as a passing fixture.

Extract the L3 case runner to respect the existing size ratchet. Production reflection and its parser acceptance rules remain unchanged. Completion means an existing-parser-accepted, nondegraded verdict; this does not add strict schema validation.

Validation:

  • Regression first reproduced missing pending status for malformed/empty/unrelated responses; focused eval/reflection tests 26/26 passed.
  • Full npm test after integrating current main: 503 tests, 491 passed, 12 optional skips, 0 failures.
  • Real Edge browser: pending-only, mixed and completed-only states passed with no page errors; original UI failed the regression. Parent independently reran browser plus eval tests: 11/11 passed, no skips.
  • npm run lint: 0 errors, 7 existing warnings; npm run eval: 8/8 fixtures, 2 L3 judgments pending.
  • Independent task and broad final reviews approved. Added the new browser regression to the existing Chromium CI job. The initial PR checks passed, including the new Chromium test; after integrating current main, exact-head checks rerun real PostgreSQL, browser acceptance and both Docker builds.

@xiaour
xiaour merged commit 12e9712 into xiaour:main Oct 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants