Repository navigation
Fix JiuwenSwarm GLM-5.3 evaluation and publish retest results - #295
Conversation
tzhouam
changed the title
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d55be10b7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| record = load(value["study"] / f"private-codex/scores/pr-{sample['pr']}.json", {}) | ||
| if record.get("status") != "complete" or record.get("truth_manifest_sha256") != binding(value["study"] / "private-codex/truth-manifest.json")["sha256"]: | ||
| raise ValueError("sample lacks bound completed blind score") |
There was a problem hiding this comment.
If a private-codex/scores/pr-*.json classification is edited and results.json is regenerated to match, this verifier accepts the rewritten metrics because it checks neither the score record's schema, pr, and inputs_sha256 nor any immutable hash of the record; recomputing metrics from the same mutable data.reviews does not establish that those labels came from the archived blind-scoring call. Hash-bind each score record or reconstruct and validate its complete scoring-input identity before using its comments, otherwise the published accuracy results can be changed without performing another score call.
Useful? React with 👍 / 👎.
The JiuwenSwarm A/B evaluator rejected output allowed by its review prompt and ordinary directory arguments. Default MCP rendering also duplicated structured text, so the old native output cap was not certified. This change aligns the review contract, gates complete prompt reading, bounds serialized tool results in both characters and UTF-8 bytes, and verifies native delivery receipts and source-call budgets.
A fresh frozen campaign ran all 72 GLM-5.3 subscription reviews with concurrency 13, followed by 12 independent Codex blind-scoring calls. It reuses the original sources, document corpora, and finite truth baseline without resampling. Main-sample completion changes from 87.50% to 91.67% for author documents and from 62.50% to 79.17% for the knowledge base. Both groups finish 31/36 valid reviews overall; all 62 valid outputs are scored. Nine documentation-budget refusals and one malformed JSON remain failures.
The main paired precision is 65.56% versus 65.48%; paired native time is 505.64s versus 538.35s. This batch does not demonstrate a knowledge-base accuracy or speed gain. The Chinese retest report preserves these limits and links the unchanged content/coverage comparison. Full traces stay outside Git, with compact statistics and hashes committed.
Validation: 3137 tests passed and 17 skipped; focused protocol/preparation/report checks passed; independent native audit certifies all 72 traces with zero truncation or duplicate rendering and a maximum 51/60 source requests. All 12 scorers completed once, frozen runtime bytes remained unchanged, and develop was rechecked against the fixed source pin. Both final-head CI jobs passed.