Inspect repeated judge decisions on fixed recordings; release 0.12.0 - #19
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated judgments can disagree even when the agent recording is unchanged. Add an offline Judge Stability report that freezes the supplied recording, dataset, configuration and evaluator definitions, then separates score variation, pass/reject disagreement and unavailable assessments for each target. Preserve every original input and recompute the report with a dedicated verifier; the optional consistency gate explicitly allows all-reject agreement and does not stand in for task acceptance.
Publish five synthetic controls with three authored judgment sets, a homepage comparison matrix, mobile/offline filtering, individual source downloads and a deterministic evidence ZIP. Add bilingual instructions and README screenshots. Keep prior audit/model records unchanged and correct stale casebook counts to the existing 251 rows. Stage version 0.12.0; no new model or AWS evaluation is claimed.
Validation: 357 Python tests; Ruff lint/format; full desktop, 390px and 320px browser checks, offline report and original-download byte checks; wheel installation outside the checkout with an empty PATH. CI will repeat tests on Python 3.11–3.13 and the existing six Docker audit combinations.