Py-boost: a research tool for exploring GBDTs

Modern gradient boosting toolkits are very complex and are written in low-level programming languages. As a result,

• It is hard to customize them to suit one’s needs
• New ideas and methods are not easy to implement
• It is difficult to understand how they work

Py-boost is a Python-based gradient boosting library which aims at overcoming the aforementioned problems.

Authors: Anton Vakhrushev, Leonid Iosipoi , Sergey Kupriyanov.

Py-boost Key Features

Simple. Py-boost is a simplified gradient boosting library, but it supports all main features and hyperparameters available in other implementations.

Fast with GPU. Despite the fact that Py-boost is written in Python, it works only on a GPU and uses Python GPU libraries such as CuPy and Numba.

Efficient inference. Since v0.4, Py-Boost has been able to perform efficient inference of tree ensembles on a GPU. Moreover, once your model is trained on a GPU, it can be used for inference on a CPU-only machine after conversion to the treelite format using the built-in wrapper (limitation: the model should be trained with target_splitter='Single', which is the default).

ONNX compatible. Since v0.5, Py-Boost has been compatible with the ONNX format, which provides more options for CPU inference and model deployment.

Easy to customize. Py-boost can be easily customized even if one is not familiar with GPU programming (just replace np with cp). What can be customized? Almost everything via custom callbacks. Examples: Row/Col sampling strategy, Training control, Losses/metrics, Multioutput handling strategy, Anything via custom callbacks.

SketchBoost paper

Multioutput training. Current state-of-the-art boosting toolkits offer limited support for multioutput training. Even when this option is available, training times for tasks such as multiclass/multilabel classification and multitask regression are quite long due to the training complexity, which scales linearly with the number of outputs. To overcome the existing limitations, we introduce the SketchBoost algorithm, which uses approximate tree structure search. As we show in the paper, this strategy does not lead to a decrease in performance and can often improve accuracy.

SketchBoost. You can try our sketching strategies by using the SketchBoost class, or you can implement your own strategies and pass them to the GradientBoosting constructor as the multioutput_sketch parameter. For details, please see Tutorial_2_Advanced_multioutput

Installation

Py-Boost supports Python 3.8–3.14. Install the CuPy distribution compatible with your Python and CUDA versions before installing Py-Boost.

On Python 3.14, use Numba 0.63 or later and CuPy 14 or later. For example, with CUDA 12.x:

pip install -U "cupy-cuda12x>=14" py-boost

For other Python or CUDA versions, choose the appropriate CuPy package from the CuPy installation guide.

CPU inference with Treelite and TL2cgen

Py-Boost uses Treelite 4 (>=4.1.2,<5) and TL2cgen 1 (>=1,<2) for CPU inference. Both dependencies are installed with Py-Boost; treelite_runtime is no longer used. NumPy 1.x and 2.x are supported, subject to your Python version and the other dependencies.

Convert a fitted model trained with target_splitter='Single' (the default):

from py_boost import TLPredictor, TLCompiledPredictor

tl_model = TLPredictor(model)
prediction = tl_model.predict(X_test, nthread=4)
tl_model.dump('cpu_model')
restored = TLPredictor.load('cpu_model')

Single-output and multi-output regression, binary/multilabel classification, and multiclass classification keep the original model's postprocessing. Predictions have shape (n_samples, n_outputs), including single-output models. The underlying Treelite model stores raw scores as regression targets; the wrapper applies model.postprocess_fn once, after restoring this output shape. A custom CPU postprocessor can be supplied with postprocess_fn=.... Both predictor wrappers support shallow copying with copy.copy(); saving a predictor leaves the original and its copies usable.

Compile the model with TL2cgen for faster CPU inference. Compilation requires a C compiler; prediction with the compiled library does not require a compiler:

# Linux: GCC must be installed and on PATH.
compiled = tl_model.compile('gcc', 'model.so', nthread=4)
# Windows: run from a Visual Studio developer shell, with MSVC on PATH.
# compiled = tl_model.compile('msvc', 'model.dll', nthread=4)

prediction = compiled.predict(X_test)
compiled.dump('compiled.pkl')
restored_compiled = TLCompiledPredictor.load('compiled.pkl')

Threshold quantization can be enabled by passing params={'quantize': 1} to compile(); prediction keeps the original float32 split boundaries.

TLCompiledPredictor.predict() also accepts SciPy sparse matrices (CSR, CSC and COO) without converting them to dense arrays. Unstored entries and stored NaNs are treated as missing values; explicitly stored zeros remain zero. Duplicate coordinates are summed in the input dtype before conversion to float32. Prediction leaves the input matrix unchanged.

Compiled predictor dumps contain metadata and the library path. Deploy the .so/.dll alongside the dump. If its path changes, load it with TLCompiledPredictor.load('compiled.pkl', libpath=new_path). Use set_libpath(new_path) to relocate an already loaded predictor. Use a library built for the target operating system and architecture. The legacy compiler argument accepts only its default value, 'ast_native'. See the TL2cgen migration guide for the underlying API changes.

Treelite 3 model checkpoints and libraries compiled with treelite_runtime should be re-exported from the original Py-Boost model with this version.

Quick tour

Py-boost is easy to use since it has an interface similar to scikit-learn's. For usage examples, please see:

• Tutorial_1_Basics for simple usage examples
• Tutorial_2_Advanced_multioutput for advanced multioutput features
• Tutorial_3_Custom_features for examples of customization
• Tutorial_4_Handle_null_targets for the case when the multioutput target contains NaNs
• Tutorial_5_ONNX_inference examples of parsing and inference on CPU with ONNX

More examples are coming soon.

Missing binary targets

For binary or multilabel targets, use NaN for unknown labels and 0 for observed negatives. BCEWithNaNLoss excludes missing labels from gradients and Hessians. Its initial probabilities are weighted frequencies over observed labels; zero-weight rows do not affect them. Every training target column must contain an observed label with positive row weight. Without sample_weight, each row has weight 1. Both all-NaN columns and columns whose observed labels all have zero weight raise ValueError listing their zero-based indices before initialization or tree construction. This check applies to the training partition of each cross-validation fold; no columns are silently removed. Entirely unobserved columns are allowed in evaluation sets. Keep lambda_l2 > 0 so leaves without observed labels have finite updates.

from py_boost import GradientBoosting
from py_boost.gpu.losses import BCEWithNaNLoss, NaNAucMetric

model = GradientBoosting(BCEWithNaNLoss(), metric=NaNAucMetric(), lambda_l2=1)
model.fit(X_train, y_train, eval_sets=[{"X": X_valid, "y": y_valid}])

The loss also accepts aliases bce_nan and multilabel_nan. Its default metric is NaNBCEMetric; metric names bce/logloss and auc/roc select the corresponding NaN-aware metrics when used with this loss. Explicit metric aliases bce_nan, auc_nan and gini_nan are also available. Existing losses and their default metrics are unchanged.

Import these metrics from py_boost.gpu.losses:

Metric	Result
NaNBCEMetric	Mean BCE over observed labels; return_list=True gives per-output losses as a CuPy array.
BCEWithNaNMetricWeighted(output_weights)	Weighted mean of per-output BCE.
NaNAucMetric	Macro ROC AUC; return_list=True gives a NumPy array.
NaNAucMetricByClass	Per-output ROC AUC as a NumPy array, for reporting.
NaNGiniMetricByClass	Per-output Gini (2 * AUC - 1) as a NumPy array, for reporting.
NaNGiniMetricAtN	Weighted macro Gini; call with prod_weights= and n= to restrict to the first n target columns.

All metrics support finite, nonnegative row weights of shape (n_samples,) or (n_samples, 1). NaN labels and zero-weight rows are excluded independently for each target. ROC AUC/Gini are undefined for outputs without positive weight in both classes; BCE is undefined for outputs without any positively weighted labels. Per-output results contain NaN for these outputs, and macro metrics exclude them. A scalar metric is NaN if no eligible output remains. Positive row weights are rescaled before aggregation so multiplying all weights by a common positive constant does not change the metric, subject to floating-point precision. All-zero weights produce an undefined metric.

With early stopping enabled, an undefined (NaN) scalar metric on the last evaluation set raises a clear error instead of silently pruning to one tree. Choose an evaluation set and metric with enough observed, positively weighted labels, or set es=0 to keep training and record NaN in the evaluation history. The per-class vector metrics are for reporting only.

Observed targets must be 0 or 1; NaN is the only missing-label marker. Training and evaluation labels are validated once in their original dtype before conversion to float32, including labels on zero-weight rows. A float64 label such as 1.00000001 is rejected instead of being rounded to 1. Predictions must be finite wherever a label has positive row weight; invalid predictions are reported as errors rather than treated as missing labels. BCE additionally requires probabilities in [0, 1], while ROC AUC/Gini also accept finite decision scores.

Output/product weights must follow target column order and are normalized over eligible, positively weighted outputs. n selects leading columns, not the outputs with the highest scores. Compared with custom implementations that return an unnormalized weighted sum, non-unit-sum weights therefore have different semantics. Standard CrossValidation is supported; AdaptiveESCV requires an explicit scorer for missing labels.

FilterSketch supports zero Hessians from missing or zero-weight labels. Outputs with zero total Hessian are excluded from selection. If all remaining output importances are zero, selection uses a uniform distribution over the eligible outputs; if none remain, it raises an error identifying the lack of positive Hessian mass. The separate training-column check requires an observed label with positive row weight in each training output. Gradient squaring and per-leaf accumulation use float32 to limit GPU memory; reductions and probability normalization use float64. With the default BCE clipping and ordinary row weights, gradient magnitudes are well within the float32 range. Extreme weights or gradient scales can still cause overflow or underflow during squaring, and roundoff can affect importance estimates when leaf gradients nearly cancel. Float64 reductions do not remove these float32 training limitations. Zero gradients at an optimum are valid and do not trigger a missing-training-label error.

For BCEWithNaNLoss, row weights are validated as finite and nonnegative before conversion. Training weights are stored as float32: overflow to infinity or conversion of a positive weight to zero raises an explicit representation error with the affected row indices. Training weights are not normalized, so their scale retains its effect on lambda_l2. Evaluation weights stay float64 through fit, allowing metrics to handle weights such as 1e308 and 1e-308. All-zero training weights are rejected by the per-output check; all-zero evaluation weights are allowed for reporting and produce an undefined metric. These strict validation and weighted-initialization policies apply only to BCEWithNaNLoss; existing losses retain their preprocessing and initial scores.

GPU regression tests can be run with python -m pytest -q tests after installing Py-Boost, a compatible CuPy build, and pytest.