Skip to content

Fix boosting training and recover from failed Optuna trials - #198

Open
SergeevVladislav wants to merge 1 commit into
sb-ai-lab:developmentfrom
SergeevVladislav:fix/boosting-training-and-tuning
Open

SergeevVladislav wants to merge 1 commit into
sb-ai-lab:developmentfrom
SergeevVladislav:fix/boosting-training-and-tuning

Conversation

@SergeevVladislav

Copy link
Copy Markdown

What was done:

An exception in an Optuna trial could abort the pipeline, including when the first trial failed before a best trial existed. Tuning now continues after individual failures, stops cleanly at fail_tolerance, retains the best completed parameters, and falls back to initial parameters when no trial completes. Invalid distribution types still fail before tuning, and keyboard interrupts propagate with progress-bar cleanup.

This PR also fixes defects reproduced while auditing the three boosting adapters:

  • XGBoost validation predictions, inference, and tree feature importance use best_iteration, including when training reaches the iteration limit. Importance uses the native get_score() API, retains feature names, and includes unused features with zero importance. Linear-booster predictions retain their supported behavior.
  • Multiclass output dimensions cover classes from both training and validation. CatBoost retains missing training classes; cross-entropy scoring handles validation splits with absent classes while preserving existing sklearn input formats.
  • CatBoost accepts numeric CSR inputs without densifying, counts sparse missing values during tuning, and avoids a discarded pandas dtype conversion. Pandas dataset index normalization avoids the deprecated inplace argument.
  • Custom LightGBM objectives work with both the old fobj API and LightGBM 4's callable objective. Explicit loss-adapter dictionaries are validated correctly. Multiclass F1 objectives retain the backend's derivative shape and apply observation weights; metric callbacks do not apply the prediction softmax to class labels.
  • Existing XGBoost RMSLE training uses squared error on log1p(target) with expm1 predictions. RMSLE evaluation uses log-space RMSE; other requested metrics use the original scale. Quantile q and Huber a reach the native parameters, and quantile callbacks honor their parameter and observation weights.

Added 74 regression cases and documented the behavior. Reviewed the complete diff and affected training, tuning, scoring, inference, and serialization paths, including missing-class scoring, legacy metric input formats, and convergence on large targets.

Validation:

python -m pytest -q -W error \
  tests/unit/test_ml_algo/test_boost_xgb.py \
  tests/unit/test_ml_algo/test_boosting_training.py \
  tests/unit/test_ml_algo/test_optimization \
  tests/unit/test_tasks \
  tests/unit/test_automl/test_presets/test_boosting_tuning_failures.py \
  tests/unit/test_dataset/test_pandas_dataset.py
  • 112 passed on Python 3.13 with Optuna 5.0.0, XGBoost 3.4.1, LightGBM 4.7.0, CatBoost 1.2.10, and pandas 3.0.6.
  • 112 passed on Python 3.12 with pandas 3.1.1 and the same boosting/Optuna versions; used the CUDA initialization warning exception from tox.ini.
  • 112 passed on Python 3.10 with Optuna 4.5.0, XGBoost 2.1.4, LightGBM 4.6.0, CatBoost 1.2.8, and pandas 2.2.3; excluded the environment's optional Torch import warning.
  • LightGBM 3.3.5 compatibility subset: 10 passed, including real custom-objective training through fobj.
  • The same focused suite on upstream 212e38e: 60 failed, 52 passed, plus one teardown error from an unused XGBoost parameter warning.
  • Extended local tuning checks: all nine booster/task combinations completed 50 successful trials after one failed trial with Optuna 5.0.0.
  • Native CUDA smoke checks: 25 scenarios passed, with actual cuda:0 / GPU model configuration verified. These used a separate local harness, not CPU fallback.
  • Simulated three-way merge with metrics PR Add extended evaluation metrics and Task integration #195 (e27f2fb): no conflicts; 381 tests passed across the combined task, boosting, tuning, and preset suites.
  • pre-commit run --files <changed files> and codespell <changed files>: passed. sphinx-build -E -b html docs <build directory>: succeeded with six existing warnings; all 193 generated API pages were verified.

Full-suite status: a complete Python 3.13 run before the final RMSLE scale refinement finished with 343 passed, 1 failed, 9 skipped, and 4 passing subtests. The final full-suite rerun is in progress. The nine skips require CUDA-enabled Torch, while this environment has CPU Torch; native boosting CUDA was tested separately. Other Python versions were covered by the focused suite, not full-suite runs.

The remaining failure is TestTabularAutoMLXGB.test_fit_predict_binary, whose existing fixture requires both OOF and holdout AUC above 0.69. The controlled comparison gives OOF/holdout 0.69794 / 0.68610 with upstream all-tree predictions and 0.67353 / 0.71981 with the corrected best-iteration predictions. Thus the upstream fixture already fails on holdout, while this PR fails its OOF threshold and improves holdout AUC. The fixture and thresholds are unchanged; the full suite is not reported as green. Best-iteration prediction intentionally changes model outputs and requires quality review beyond these regression tests.

Why / Goal:

Keep individual tuning failures from discarding a usable pipeline, and correct the reproduced boosting training and compatibility defects. This is a separate PR against development; it does not contain the metrics expansion from #195.

Related issues:

N/A. No issue was created for this audit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant