Repository navigation
Conversation
ndharasz
commented
Sep 15, 2026
ndharasz
commented
- Neutralize the rank-Gaussianized meta model against the same neutralizers as the predictions before orthogonalizing.
- Add regression coverage for identical predictions and meta model yielding zero neutral MMC (within 1e-12), including ties, different exposures, multiple columns, and top/bottom scoring.
- Update expected scores, the no-variance-normalization test, README, docstring, and changelog.
- Bump the package version to 0.7.2 in pyproject.toml and correct the release guide’s stale changelog instructions.
- Neutralize the rank-Gaussianized meta model against the same neutralizers as the predictions before orthogonalizing. - Add regression coverage for identical predictions and meta model yielding zero neutral MMC (within 1e-12), including ties, different exposures, multiple columns, and top/bottom scoring. - Update expected scores, the no-variance-normalization test, README, docstring, and changelog. - Bump the package version to 0.7.2 in pyproject.toml and correct the release guide’s stale changelog instructions.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1760641cd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it changes the semantics of a core scoring formula used to compute payout-relevant contribution scores, a human look would still be worthwhile.
What was reviewed: the neutral_contribution change in numerai_tools/scoring.py (meta model now neutralized against the same neutralizers as predictions before orthogonalizing), the updated docstring/README wording, the version bump/changelog entries, and the new regression tests asserting identical predictions/meta model score zero (including ties and multiple columns). I also checked whether neutralizing the meta model could shrink its post-neutralization variance toward zero (making orthogonalize's u.T @ u denominator blow up for pathological inputs where the meta model is almost fully explained by the neutralizer features) — this is a new numerical exposure introduced by the fix, but it wasn't reported as a confirmed bug, likely because real meta models aren't expected to be near-perfectly explained by the feature neutralizers.
Extended reasoning...
Overview
The PR changes neutral_contribution in numerai_tools/scoring.py so that the meta model is neutralized against the same neutralizer features as the predictions before orthogonalizing, instead of only rank/gaussianizing it. This is a targeted, well-motivated fix: previously identical predictions and (already-neutral) meta model would not necessarily produce zero contribution because re-ranking/gaussianizing can reintroduce neutralizer exposure. The change is accompanied by docstring and README updates, a CHANGELOG entry, a version bump to 0.7.2, a reversal of stale RELEASING.md guidance about not maintaining a changelog, and updated tests (new regression tests confirming identical predictions/meta model yield ~0 score across ties and multiple columns, plus updated fixture expectations and the no-variance-normalize test's helper computation).
Security risks
None. This is a pure numerical/statistics change in a scoring library; no user input handling, auth, or external I/O is touched.
Level of scrutiny
This is small in diff size but touches core, math-heavy scoring logic whose correctness matters for downstream consumers computing model contribution/payout-relevant scores. That combination (small diff, but semantically significant and used for financial-adjacent calculations) warrants a human look even though the automated hunt found nothing actionable, per the guidance to not approve changes to critical/scoring-sensitive logic outright.
Other factors
The fix is well tested: existing fixture values were updated consistently with the new behavior, and new tests directly validate the stated invariant (identical predictions/meta model → zero score) across edge cases (tied ranks, multiple columns, top/bottom scoring). I also reasoned through neutralize/orthogonalize to check whether neutralizing the meta model could degrade the orthogonalize denominator (u.T @ u) toward zero for meta models that are nearly fully explained by the neutralizer features — a new exposure this change introduces that didn't exist when the meta model was left un-neutralized — but this was not surfaced as a confirmed bug by the hunt, and is unlikely to matter for realistic meta models built from many prediction sources rather than the feature set itself.
1 participant
