Repository navigation
Add NaN-aware binary loss and multilabel metrics - #30
Conversation
|
I would separate the API policy decisions from the correctness fixes. Scope: the strict target-validation and weighted-initialization policies below apply to BCEWithNaNLoss. Keep the target validation and initialization of existing losses unchanged. Any changes to the shared FilterSketch and EarlyStopping behavior should be explicit and tested with existing losses as well. API policy decisions
Correctness fixes required under this policy
Documentation and validation Update the README and Tutorial 4 to describe the policy and use the built-in loss and metrics instead of local implementations. Add CUDA regression coverage through fit() and CrossValidation, including all-NaN training outputs, observed labels with only zero weights, weighted initial frequencies, empty evaluation outputs, undefined early-stopping metrics, invalid float64 labels before conversion, extreme weights, and FilterSketch. Test direct metric calls as well, but do not rely on them as a substitute for testing the public training path. Include regression coverage for existing losses affected by changes to the shared sketching or early-stopping code. |
|
@dev-rinchin Thanks for the detailed review. All six points are addressed in af3243e; the branch also includes current master.
Validation on RTX 5090 / CuPy 14.2.0 / CUDA 12.9: 143 GPU tests passed (111 loss/metric/sketch tests plus 32 existing histogram tests); 143 skipped cleanly without CUDA. A 105-test regression subset against the pre-review implementation reproduces 40 failures, all addressed by this update. I also executed all Tutorial 4 code cells, omitting only the timing magics, at its full dimensions: 50k train + 50k validation rows, 100 features, 10 outputs, and 1,000 trees. Predictions and scalar metric history are finite; the deliberately hidden evaluation output has NaN AUC as documented. |
…view # Conflicts: # py_boost/multioutput/sketching.py
|
After revisiting the numerical risks, I would revise the requirement for float64 importance calculations in my earlier review. The proposed follow-up restores float32 for gradient squaring and per-leaf gradient/Hessian accumulation in FilterSketch, while retaining float64 reductions and probability normalization. This avoids the additional GPU memory cost of the large float64 intermediates. Catastrophic overflow or underflow should be very rare edge cases in ordinary BCE training with weights around one. With the default probability clipping, observed-label gradients have magnitudes between approximately 1e-7 and 1, so their squares are well within the float32 range. Even with one million rows, a leaf gradient sum squared is at most 1e12 in unweighted BCE, far below the float32 limit of approximately 3.4e38. Having 1000+ outputs does not itself increase the scale of each output's gradients. The earlier examples with weights of 1e20 and 1e-30 demonstrate possible failures under extreme scales; they do not establish that these failures are common. Such scales, unscaled regression targets, or custom losses can still exceed the safe range of float32 calculations. Ordinary accumulation roundoff also remains possible, especially when positive and negative leaf gradients nearly cancel. We do not have production failure-rate data, so "very rare" is a conditional assessment for ordinary BCE usage, not a measured percentage. In seven controlled GPU scenarios, including 100,000 rows with 1000 outputs, one million rows, moderate weights, and nearly cancelling gradients, the original float32 calculation produced finite importances and the same top eight outputs as float64. The largest absolute difference in normalized importances was approximately 3e-6. These are synthetic checks of importance calculations, not a guarantee of unchanged training quality. The missing-label and zero-importance fixes should remain: reject training outputs without observed labels of positive weight, exclude zero-Hessian outputs from selection, and use uniform selection over eligible outputs when all their importances are zero. These address undefined operations such as 0/0; switching to float64 does not fix them. The proposed precision change also leaves the existing float64 handling of evaluation weights and metrics in place. A possible intermediate option is to retain float64 calculations but process outputs in blocks, reusing temporary buffers for each block. Only the full vector of output importances needs to survive until global normalization and selection. This can reduce temporary memory substantially, at the cost of additional GPU operations and less favorable access to column slices. It could be an optional future mode; the current proposal does not add a blocking API. The following measurements are from a simple CuPy prototype on an NVIDIA A100 80GB PCIe with CuPy 14.2.0. For an isolated importance calculation with 100,000 rows, 1000 outputs, 64 leaves, one previous tree, and separate Hessians for each output, medians over 20 warmed repetitions were:
For complete fit on a separate synthetic BCE task with 80,000 training rows, 10,000 validation rows, 1000 outputs, 20 features, 25% missing labels, and 30 trees of depth 4, medians over seven warmed repetitions were:
These comparisons are full float64 versus blocked float64, not versus the restored float32 implementation. The memory figures include CuPy intermediates inside the importance calculation, not just the leaf-sum buffers; they exclude input arrays, other training allocations, the CUDA context, and allocator cache. They are not estimates of total training memory. The memory and fit tables use different row counts as specified above. Results depend on hardware and workload: the same prototype was substantially slower without tree history or on smaller inputs. In the measured fit runs, the maximum absolute prediction difference between full and blocked float64 was approximately 1.2e-7. An optimized implementation would need its own validation and benchmarks. |
|
Benchmark of PR #30's merge result against master immediately before the merge. All sketch cases use sketch_outputs=5. Synthetic data without NaN labels; 32 training iterations, one warmup and five timing repetitions. Python 3.12.3, NumPy 2.5.4, CuPy 14.2.0. pyboost-gpu-v1-k5Base: GPU count: 1; versions and cases execute sequentially. K is the full target dimension; k is the sketch width used for structure search. Times are warmed wall-clock medians. GPU memory is peak live CuPy allocations,
BCE/FilterSketch fit time increased by 9.1%; other fit-time medians changed by −0.7% to +1.4%. Peak live CuPy allocations were identical in all 12 scenarios. The maximum absolute holdout-metric change was approximately 4e-5. These workloads do not exercise the new NaN loss or NaN-specific metrics. |
Binary and multilabel training with missing targets currently requires custom loss/metric implementations. Add built-in NaN-aware training and evaluation so unknown labels can be excluded without replacing them with observed negatives.
BCEWithNaNLoss(bce_nan/multilabel_nan) gives missing labels zero gradient/Hessian contributions and initializes each output from its observed labels. Training and evaluation targets are validated once before training: observed labels must be 0 or 1. Every training output must contain an observed label; otherwise, aValueErroridentifies the affected column indices. Entirely missing evaluation outputs remain valid.The default metric is NaN-aware BCE;
bce/loglossandauc/rocresolve to NaN-aware metrics for this loss. Add these public metrics:NaNBCEMetric: observed-label BCE, optionally per output.BCEWithNaNMetricWeighted: normalized weighted mean of per-output BCE.NaNAucMetric/NaNAucMetricByClass: macro/per-output ROC AUC.NaNGiniMetricByClass/NaNGiniMetricAtN: per-output and weighted macro Gini, withnselecting leading target columns.Undefined per-output results are NaN and excluded from macro averages. Standalone scalar metrics remain NaN when no eligible output exists. Early stopping raises an explicit error if its selected scalar metric on the last evaluation set is NaN; with
es=0, training continues and the history records NaN. Row weights are rescaled before BCE/AUC aggregation to avoid overflow and underflow. The shared binary AUC helper also uses stable class-wise normalization. Zero-weight exclusions and undefined all-zero-weight results are preserved.FilterSketchexcludes outputs with zero Hessian mass, falls back to uniform selection among eligible outputs when total importance is zero, and raises a clear error when none are eligible. Guard zero denominators in both initial and queued importance calculations, handlesmooth=1, and allocate leaf accumulators usingmax_leaf_index + 1.README and Tutorial 4 document these policies. Tutorial 4 now imports the built-in classes, demonstrates an entirely missing evaluation output, and uses a generic Python kernel. Per-output metrics are intended for reporting;
AdaptiveESCVstill requires an explicit scorer for missing labels.Validation on RTX 5090, Python 3.12.3, CuPy 14.2.0 / CUDA 12.9, NumPy 2.5.4, Numba 0.68.0:
python -m pytest tests -q: 143 passed — 111 NaN loss/metric/sketch tests and 32 existing histogram tests from current master.1e-320and up to1e308, zero-weight observations, invalid train/evaluation labels, missing training columns/folds, undefined early-stopping metrics, FilterSketch, random projection and cross-validation.ndarray.scatter_adddeprecation warnings remain.Updated against current master, which includes #29 and #31.