Repository navigation
Add extended evaluation metrics and Task integration - #195
Open
SergeevVladislav wants to merge 2 commits into
Open
SergeevVladislav wants to merge 2 commits into
SergeevVladislav wants to merge 2 commits into
Conversation
SergeevVladislav
force-pushed
the
feat/work-metrics
branch
from
November 11, 2026 07:59
e796163 to
e27f2fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was done:
Add an evaluation-metric catalog covering all 66 metric spellings requested in the operational metric module, with native
Taskintegration where the existing training task provides the required inputs. Add standalone evaluation helpers for uplift, query ranking and rolling forecasts.Rebased onto current
developmentat212e38e2430376bc9562d53a4dce651f1aaab8a4. The rebase conflict in theTaskdocumentation was resolved by retaining both the new metrics documentation and the corrected description of the five supported task types. PRs #193 and #194 are now part of the upstream base; this PR contains only the metrics changes and does not depend on any unmerged branch.Metric coverage
pr_auc), Brier loss, expected calibration error, recall at a requested specificity and standardized partial ROC AUC.API and training integration
get_metriccallables, so parameterized and matrix-valued metrics do not need direction inference on an incompatible two-row toy example. Explicitgreater_is_betterstill takes precedence.(n,)and neural-network(n, 1)binary prediction formats in top-K evaluation. Exercise fit/predict and pickle round trips through the real tabular preset.Formula corrections and compatibility decisions
1 - hamming_loss, so a perfect prediction scores 1.ap@k/average_precision@knormalize truncated precision contributions by all relevant observations;average_precision_in_top_kcomputes scikit-learn AP only within the selected subset. Add corresponding MAP variants. For relevance[1, 0, 1, 1], descending scores and K=2, these give 1/3 and 1 respectively.roc_auc_score_average_samplesalias as NaN-aware column-wise macro AUC, and provide correctly named sample-wise AUC separately. Preserve percentage scales for SMAPE, median APE, NaN-aware Gini and Powerstat.mapebehavior and binaryaccuracycutoff are retained; the newMAPEandaccuracy_scorespellings have the documented safer denominator/inclusive-cutoff semantics. Existing multiclass F1 names gain the shared parameter-aware wrapper, with undefined components defaulting to zero.Open-source reuse and documentation
Standard metrics delegate to scikit-learn, including average precision and NDCG tie handling. Calibration follows the documented L1 ECE definition, implemented with NumPy bin accumulation and explicit endpoint handling.
Uplift formulas are adapted from scikit-uplift 0.5.1, retaining its MIT copyright/license notice in the package. The adaptation uses NumPy cumulative sums instead of the deprecated scikit-learn
stable_cumsumdependency and validates undefined normalizations/empty selections. No dependency changes are required. Existing LightAutoML addon uplift-curve conventions remain unchanged.docs/pages/Advanced_metrics.rstdocuments the full catalog, task applicability, scales, parameters, backend behavior, both AP definitions and runnable evaluation examples; it is linked from the task API documentation.Review fixes in this revision
Reviewed the complete PR diff, the affected
Task/dataset-metric and booster call sites, all four new test modules, the metric guide, and the coverage of all 66 operational metric names. Fixed the following reproducible defects:drop_intermediate=False; cover weighted and unweighted inputs. This is a discrete threshold calculation, without interpolation.Tasklost the optimization direction of factory callables. (fix) Wrappingget_metric(...)before reading its metadata caused direction inference to evaluate an incompatible toy sample. Adjusted R², full-population precision@K and matrix-valued MAP could fail at task construction. Read the callable's declared direction first, preserve explicit overrides, and cover serialization. Generic callables without direction metadata retain the existing inference behavior.Noneallowed LightGBM's objective metric to remain active alongside the requested custom metric and affect early stopping. Use the LightGBM string"None"consistently for both named and callable Python metrics. Actual training tests verify that evaluation history contains only the requested custom metric; parameter/weight/direction tests cover both forms across the relevant backends.The new regression cases were also run against the rebased implementation before these fixes: 13 failures reproduced the defects. The corrected task suite passes in both dependency environments below. No outstanding defect attributable to this PR was found after the fixes; the full-suite limitations are listed separately.
Validation
Validated code commit:
e27f2fb67c14b141886b4c6a6b060482fc882217(rebased feature commit plus the review fixes).Two isolated environments use the repository's compatibility snapshots:
tests/constraints/legacy.txt, NLP extra)tests/constraints/latest.txt, all extras; AFG snapshot for setuptools)Commands below use the corresponding environment's
python,OMP_NUM_THREADS=1,MKL_NUM_THREADS=1andHF_HUB_OFFLINE=1. Both environments passpython -m pip check. The legacy run uses the existing compatibility-workflow exception-W 'ignore:allow_ops_in_compiled_graph failed to import torch:ImportWarning'; no new warning suppression was added.python -m pytest -q -W error tests/unit/test_tasks— 298 passed in each environment. Covers reference values, weights, boundary/error cases, registered task shapes, callable metadata/overrides, booster callbacks, actual LightGBM/XGBoost/CatBoost training, and tabular-preset fit/predict/pickle checks for LightGBM, XGBoost, CatBoost, MLP and linear L2.tests/unit/test_optionalgives 497 passed, 4 skipped, 4 subtests passed. GPU tests are skipped in these CPU environments; legacy also omits optional CV coverage.212e38e. Using setuptools 81.0.0 from the existingtests/constraints/afg-pandas3.txtsnapshot resolves them: all 13 AFG tests and the broader 515-test run above pass. No unrelated AFG code was changed.tox -e lint— passed, including all configured repository hooks. Commit hooks also passed.tox -e docs -x 'testenv:docs.base_python=/usr/bin/python3.12'— passed, with six warnings in existing API/tutorial documentation; none in the new metrics guide. The existing Sphinx 5 toolchain requires an interpreter that still providesimghdr, so documentation was built with Python 3.12.uv build --wheel— passed; inspected the wheel and verified inclusion of all six metrics modules andSCIKIT_UPLIFT_LICENSE.git diff --checkandcodespellon all 18 PR files — passed.tox -e codespell— fails only on the pre-existinglightautoml/dataset/utils.py:57typo (is's), outside this PR.Full-suite attempt:
python -m pytest tests/unit tests/integration -x -q -W error, withHF_HUB_DISABLE_XET=1, stopped after 72 passed, 2 skipped, 1 failed in 14 minutes. Both existing five-fold TabM training tests passed. The failure is the existing XGBoost binary preset quality assertion: held-out AUC 0.6861005434782608, below the required 0.69. Running that test separately against both this PR and unchanged upstream212e38ereproduces the identical failure and score. The test threshold and model behavior were left unchanged. Because the full run used-x, later tests, including the historical integration suite, were not reached; no full-suite pass is claimed. Existing example datasets were available; external model downloads were disabled. The complete interpreter matrix was not run locally. New metric training tests use CPU; no GPU validation is claimed for this PR.GitHub validation on the pushed commit: all five jobs passed in both the upstream compatibility run and the fork compatibility run. Upstream results: 438 passed, 4 skipped on each of Python 3.10/3.12 legacy; 443 passed, 3 skipped on Python 3.14 latest; 13 passed on each AFG dependency snapshot. Upstream pre-commit also passed. The full upstream CI remains in progress as of this update.
Why / Goal:
Make the requested operational metrics available through LightAutoML's normal task/model-selection API and a consistent standalone evaluation entry point. Reuse established open-source definitions where appropriate, correct defects in the supplied formulas, and make previously ambiguous scales and ranking conventions explicit.
Keep the PR diff focused on evaluation metrics; GPU compatibility and SSWARM changes are already in the upstream base. No goal or milestone identifier was supplied.
Related issues: