Skip to content

Add extended evaluation metrics and Task integration - #195

Open
SergeevVladislav wants to merge 2 commits into
sb-ai-lab:developmentfrom
SergeevVladislav:feat/work-metrics
Open

SergeevVladislav wants to merge 2 commits into
sb-ai-lab:developmentfrom
SergeevVladislav:feat/work-metrics

Conversation

@SergeevVladislav

@SergeevVladislav SergeevVladislav commented Oct 18, 2026 •

Copy link
Copy Markdown

What was done:

Add an evaluation-metric catalog covering all 66 metric spellings requested in the operational metric module, with native Task integration where the existing training task provides the required inputs. Add standalone evaluation helpers for uplift, query ranking and rolling forecasts.

Rebased onto current development at 212e38e2430376bc9562d53a4dce651f1aaab8a4. The rebase conflict in the Task documentation was resolved by retaining both the new metrics documentation and the corrected description of the five supported task types. PRs #193 and #194 are now part of the upstream base; this PR contains only the metrics changes and does not depend on any unmerged branch.

Metric coverage

  • Classification: precision/recall/F1/F-beta, macro/micro/weighted/sample averages, ROC AUC variants, Gini, average precision (pr_auc), Brier loss, expected calibration error, recall at a requested specificity and standardized partial ROC AUC.
  • Multilabel: Hamming accuracy, Jaccard variants, NaN-aware macro AUC/Gini and label-ranking metrics. The NaN-aware metrics evaluate incomplete labels; they do not enable training with missing labels.
  • Regression and multioutput regression: RMSE, MSLE, median absolute/percentage errors, SMAPE, WAPE and relative MAE, MAPE excluding zero targets, adjusted R², Spearman correlation and Powerstat/normalized Gini. Preserve the requested uppercase spellings and document units and output aggregation.
  • Ranking: precision@K, recall@K, truncated AP@K, MAP@K, NDCG, LRAP, ranking loss, coverage error and MRR. Support both query-item score matrices and the requested item-ID MRR representation through distinct interfaces.
  • Uplift: normalized Qini/uplift AUC, uplift@K and weighted average uplift, with overall/by-group selection where applicable.
  • Time series: rolling-origin mean MAE/MSE, explicit forecast origins, the leading-NaN work format, horizon validation and incomplete-tail handling.

API and training integration

from lightautoml.tasks import Task
from lightautoml.tasks.metrics import get_metric

task = Task("binary", metric="pr_auc")
task = Task("binary", metric="precision@k", metric_params={"k": 0.1})
task = Task("binary", metric="fbeta_score",
            metric_params={"beta": 0.5, "cutoff": 0.7})
task = Task("reg", metric="r2_adj", metric_params={"n_features": 12})
task = Task("multilabel", metric="hamming_accuracy")
task = Task("reg", metric=get_metric("r2_adj", task="reg", n_features=12))

metric = get_metric("uplift@k", task="uplift", k=0.5)
score = metric([1, 0, 0, 1], [0.9, 0.8, 0.7, 0.6], [1, 0, 1, 0])
  • Register optimization direction explicitly for named metrics and get_metric callables, so parameterized and matrix-valued metrics do not need direction inference on an incompatible two-row toy example. Explicit greater_is_better still takes precedence.
  • Validate required/optional metric parameter names independently of loss parameters. Parameters remain ordinary dictionaries; no expression evaluation or application-specific report/CSV parsing is introduced.
  • Resolve new named metrics to LightGBM/XGBoost callbacks, pass their parameters and observation weights, and keep Python metric parameters out of booster configuration. Disable LightGBM's default metric for Python callbacks passed either by name or as a callable, so it does not also control early stopping.
  • Preserve CatBoost's existing callable-metric policy, including its multioutput objectives. When no native mapping exists, CatBoost early stopping uses the training objective; LightAutoML model selection, tuning and blending use the requested evaluation metric. This PR does not implement arbitrary CatBoost Python evaluation callbacks or a new group-aware ranking training task.
  • Support (n,) and neural-network (n, 1) binary prediction formats in top-K evaluation. Exercise fit/predict and pickle round trips through the real tabular preset.

Formula corrections and compatibility decisions

  • Keep probability metrics on probabilities; threshold only label metrics. Do not independently re-encode target and predicted multiclass labels.
  • Define Hamming accuracy as 1 - hamming_loss, so a perfect prediction scores 1.
  • Fix MAPE-without-zeros target/prediction/weight alignment. Define zero/zero SMAPE contributions as zero. Use absolute targets in the WAPE denominator, including signed regression targets.
  • Require the actual feature count for adjusted R² instead of assuming 20. Preserve fractional beta values for F-beta.
  • Include probabilities exactly 0 and 1 in ECE bins. Document binary positive-class ECE, multiclass top-label ECE and macro multilabel ECE.
  • Keep both AP@K definitions: ap@k/average_precision@k normalize truncated precision contributions by all relevant observations; average_precision_in_top_k computes scikit-learn AP only within the selected subset. Add corresponding MAP variants. For relevance [1, 0, 1, 1], descending scores and K=2, these give 1/3 and 1 respectively.
  • Preserve the operational roc_auc_score_average_samples alias as NaN-aware column-wise macro AUC, and provide correctly named sample-wise AUC separately. Preserve percentage scales for SMAPE, median APE, NaN-aware Gini and Powerstat.
  • Reject undefined custom-metric cases with explicit errors instead of silently producing negative error sentinels or dividing by zero. Document score ties, weights, empty groups, all-missing targets and forecast boundaries.
  • Existing core metric names remain available. The existing lowercase mape behavior and binary accuracy cutoff are retained; the new MAPE and accuracy_score spellings have the documented safer denominator/inclusive-cutoff semantics. Existing multiclass F1 names gain the shared parameter-aware wrapper, with undefined components defaulting to zero.

Open-source reuse and documentation

Standard metrics delegate to scikit-learn, including average precision and NDCG tie handling. Calibration follows the documented L1 ECE definition, implemented with NumPy bin accumulation and explicit endpoint handling.

Uplift formulas are adapted from scikit-uplift 0.5.1, retaining its MIT copyright/license notice in the package. The adaptation uses NumPy cumulative sums instead of the deprecated scikit-learn stable_cumsum dependency and validates undefined normalizations/empty selections. No dependency changes are required. Existing LightAutoML addon uplift-curve conventions remain unchanged.

docs/pages/Advanced_metrics.rst documents the full catalog, task applicability, scales, parameters, backend behavior, both AP definitions and runnable evaluation examples; it is linked from the task API documentation.

Review fixes in this revision

Reviewed the complete PR diff, the affected Task/dataset-metric and booster call sites, all four new test modules, the metric guide, and the coverage of all 66 operational metric names. Fixed the following reproducible defects:

  1. Major — recall at specificity omitted valid thresholds. (fix) scikit-learn's default ROC simplification removes collinear points even when they represent the best attainable recall under a specificity constraint. For five tied positive/negative pairs with descending scores and specificity 0.5, the old implementation returned 0.2 instead of 0.4. Keep all ROC thresholds with drop_intermediate=False; cover weighted and unweighted inputs. This is a discrete threshold calculation, without interpolation.
  2. Major — Task lost the optimization direction of factory callables. (fix) Wrapping get_metric(...) before reading its metadata caused direction inference to evaluate an incompatible toy sample. Adjusted R², full-population precision@K and matrix-valued MAP could fail at task construction. Read the callable's declared direction first, preserve explicit overrides, and cover serialization. Generic callables without direction metadata retain the existing inference behavior.
  3. Major — LightGBM retained an extra evaluation metric for callable metrics. (fix) Python None allowed LightGBM's objective metric to remain active alongside the requested custom metric and affect early stopping. Use the LightGBM string "None" consistently for both named and callable Python metrics. Actual training tests verify that evaluation history contains only the requested custom metric; parameter/weight/direction tests cover both forms across the relevant backends.
  4. Minor — malformed forecast vectors could be silently skipped. (fix) Empty vectors and wholly NaN vectors were treated as leading missing-forecast placeholders. Restrict placeholders to scalar NaN; reject empty, multidimensional and non-finite forecast vectors, with regression tests. Valid scalar placeholders remain supported.

The new regression cases were also run against the rebased implementation before these fixes: 13 failures reproduced the defects. The corrected task suite passes in both dependency environments below. No outstanding defect attributable to this PR was found after the fixes; the full-suite limitations are listed separately.

Validation

Validated code commit: e27f2fb67c14b141886b4c6a6b060482fc882217 (rebased feature commit plus the review fixes).

Two isolated environments use the repository's compatibility snapshots:

Environment Python NumPy / pandas / scikit-learn LightGBM / XGBoost / CatBoost PyTorch
Legacy (tests/constraints/legacy.txt, NLP extra) 3.10.21 1.26.4 / 2.2.3 / 1.5.2 4.6.0 / 2.1.4 / 1.2.8 2.2.2+cpu
Current (tests/constraints/latest.txt, all extras; AFG snapshot for setuptools) 3.13.12 2.5.4 / 3.0.6 / 1.9.1 4.7.0 / 3.4.1 / 1.2.10 2.14.1+cpu

Commands below use the corresponding environment's python, OMP_NUM_THREADS=1, MKL_NUM_THREADS=1 and HF_HUB_OFFLINE=1. Both environments pass python -m pip check. The legacy run uses the existing compatibility-workflow exception -W 'ignore:allow_ops_in_compiled_graph failed to import torch:ImportWarning'; no new warning suppression was added.

  • python -m pytest -q -W error tests/unit/test_tasks — 298 passed in each environment. Covers reference values, weights, boundary/error cases, registered task shapes, callable metadata/overrides, booster callbacks, actual LightGBM/XGBoost/CatBoost training, and tabular-preset fit/predict/pickle checks for LightGBM, XGBoost, CatBoost, MLP and linear L2.
  • Broader regression command below — 515 passed, 3 skipped, 4 subtests passed in the current environment. In the legacy environment, the same command without tests/unit/test_optional gives 497 passed, 4 skipped, 4 subtests passed. GPU tests are skipped in these CPU environments; legacy also omits optional CV coverage.
python -m pytest -q -W error \
  tests/unit/test_ml_algo tests/unit/test_transformers tests/unit/test_text \
  tests/unit/test_tasks tests/unit/test_validation tests/unit/test_reader \
  tests/unit/test_image tests/unit/test_utils tests/unit/test_experiments \
  tests/unit/test_addons tests/unit/test_optional \
  tests/unit/test_automl/test_presets/test_tabular_compatibility.py
  • The first current-environment run exposed two existing AFG import tests that fail with setuptools 78.1.0. Both failures were reproduced against unchanged upstream commit 212e38e. Using setuptools 81.0.0 from the existing tests/constraints/afg-pandas3.txt snapshot resolves them: all 13 AFG tests and the broader 515-test run above pass. No unrelated AFG code was changed.
  • tox -e lint — passed, including all configured repository hooks. Commit hooks also passed.
  • tox -e docs -x 'testenv:docs.base_python=/usr/bin/python3.12' — passed, with six warnings in existing API/tutorial documentation; none in the new metrics guide. The existing Sphinx 5 toolchain requires an interpreter that still provides imghdr, so documentation was built with Python 3.12.
  • uv build --wheel — passed; inspected the wheel and verified inclusion of all six metrics modules and SCIKIT_UPLIFT_LICENSE.
  • git diff --check and codespell on all 18 PR files — passed.
  • tox -e codespell — fails only on the pre-existing lightautoml/dataset/utils.py:57 typo (is's), outside this PR.

Full-suite attempt: python -m pytest tests/unit tests/integration -x -q -W error, with HF_HUB_DISABLE_XET=1, stopped after 72 passed, 2 skipped, 1 failed in 14 minutes. Both existing five-fold TabM training tests passed. The failure is the existing XGBoost binary preset quality assertion: held-out AUC 0.6861005434782608, below the required 0.69. Running that test separately against both this PR and unchanged upstream 212e38e reproduces the identical failure and score. The test threshold and model behavior were left unchanged. Because the full run used -x, later tests, including the historical integration suite, were not reached; no full-suite pass is claimed. Existing example datasets were available; external model downloads were disabled. The complete interpreter matrix was not run locally. New metric training tests use CPU; no GPU validation is claimed for this PR.

GitHub validation on the pushed commit: all five jobs passed in both the upstream compatibility run and the fork compatibility run. Upstream results: 438 passed, 4 skipped on each of Python 3.10/3.12 legacy; 443 passed, 3 skipped on Python 3.14 latest; 13 passed on each AFG dependency snapshot. Upstream pre-commit also passed. The full upstream CI remains in progress as of this update.

Why / Goal:

Make the requested operational metrics available through LightAutoML's normal task/model-selection API and a consistent standalone evaluation entry point. Reuse established open-source definitions where appropriate, correct defects in the supplied formulas, and make previously ambiguous scales and ranking conventions explicit.

Keep the PR diff focused on evaluation metrics; GPU compatibility and SSWARM changes are already in the upstream base. No goal or milestone identifier was supplied.

Related issues:

  • Related to Add ranking task #100 for ranking evaluation metrics. This does not close Add ranking task #100: group-aware ranking training remains outside this PR.
  • Dedicated issue for the complete operational metric catalog: N/A.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant