Repository navigation
Reject a model whose class codes would be scored against label names - #9
Merged
Merged
Conversation
A model fitted on 0/1 (e.g. is_crop) audited against labels "crop"/"noncrop" had its codes taken as positions in the sorted names, so crop was read as class 0 and the model was scored with its classes swapped (accuracy 0.99 printed as 0.01, no error). Codes now stand for labels only when the labels are numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release-blocking: the only silently wrong number found in a pre-release user study.
Bug.
model=withclasses_ = [0, 1](e.g. trained onis_crop) audited against string labelscrop/noncrop:_as_modelaccepted the codes as positions in the sorted labels, socropwas read as class 0 (= non-crop in the model). Repro: a logistic regression with true test accuracy 0.993 printedtest set (as reported) 0.01, PPI 0.01, no error. The README says the class order is checked when the model hasclasses_.Fix. Codes
0..k-1stand for the labels only when the labels are numbers; otherwise raise and ask for a model fitted on the label names. A functionf(X)(noclasses_) still takes sorted-label order. 5 added lines inpappi/core/pipeline.py::_as_model.Review: the new condition in
_as_model. Behaviour change: a model fitted on label-encoded codes against string labels now errors instead of being accepted.Verified: repro now raises
model classes ['0', '1'] are codes but the labels are names ['crop', 'noncrop']; fit the model on the label names; CI-equivalent env (requirements.txt): 9 passed, ruff clean; quickstart unchanged (0.69 vs truth 0.66). No new test added.🤖 Generated with Claude Code