WIP, ENH: Array API for `GFDLClassifier` by sdtemple · Pull Request #130 · lanl/GFDL

sdtemple · 2026-09-14T18:58:19Z

This PR is scoped to address .fit(), partial_fit(), .predict() and .predict_proba() in GFDLClassifier. Outside of .fit() and partial_fit() in GFDLClassifier the predict method are not directly touched but resolved through .fit() and .predict() in the inheritance (#129). From initial experiments, using accelerator devices with PyTorch or CuPy noticeably speeds up fit calls.

Related to ENH: Array API for GFDLRegressor #129
Related to WIP, ENH: Array API GFDLEnsembleClassifier estimator #142
Related to ENH: Ridge supports array API standard, we could as well? #68
Related to WIP, ENH: exact solve with array backends #115

Test functions in test_model.py are written to evaluate behavior with namespaces, devices, and dtypes that are accessed via sklearn.utils._array_api.yield_namespace_device_dtype_combinations(). We have tested these changes on NumPy, Array API Strict, PyTorch, and CuPy with CPU, MPS, and CUDA devices and float32 and float64 dtypes.

To test, you would use an incantation like the following:

python -m pip install -e .
python -m pip install array-api-strict array-api-compat torch cupy-cuda13x
SCIPY_ARRAY_API=1 PYTORCH_ENABLE_MPS_FALLBACK=1 python -m pytest

The changes use private utilities from sklearn.

Checklist:
[x] Is there a more elegant solution than the private utility functions?

Yes, we implemented such with discussion in ENH: Array API for GFDLRegressor #129 and rebased.

[x] Is there an sklearn decorator we could use in the test suite?

Yes, we implemented such with discussion in ENH: Array API for GFDLRegressor #129 and rebased.

[x] Would some of the test functions already be captured by an sklearn decorator?

Yes, we implemented such with discussion in ENH: Array API for GFDLRegressor #129 and rebased.

[x] Benchmark speed against NumPy

Noticeable difference with accelerator devices of PyTorch and CuPy

[x] Do the activation and weight generation functions need to be in xp? Would that affect speed?

Yes, we implemented such with discussion in ENH: Array API for GFDLRegressor #129 and rebased. We don't see an impact in speed. We can make the activations in array API except for scipy.special.log_sigmoid.

[x] Can we lower the atol in the test_xp_make_regression function?

Yes, we copy and pasted from implementation in ENH: Array API for GFDLRegressor #129. There is discussion about how the sklearn equivalent is not flexible, and that sklearn team didn't seem interested in expanding it.

AI disclosure: see #129. This PR is a lot of copy and paste from #129. I consulted GPT 5 to modify GFDLClassifier implementation and manually edit it.

SI disclosure: Provided advice and small code snippets for which sklearn utilities we could leverage for model.py

sdtemple · 2026-09-14T19:00:06Z

Most of the "new" file changes from this PR are actually repetition from #129. The actual unique file changes to this PR distinct from #129 are the edit to .fit() in GFDLClassifier, the private utility function invoked in it, and the test function in test_model.py.

sdtemple · 2026-09-14T19:05:02Z

@eiviani-lanl I recommend looking and reviewing #129 before this one.

tylerjereddy · 2026-09-15T19:23:12Z

I provided some detailed initial feedback at gh-129, so I won't repeat that here. In short, both the source and testing utilities should reflect the approach used in sklearn for array API support rather than reinventing the wheel or having AI draft utilities/approaches that are already battle tested upstream. Although occasional new tests may be needed, we can probably get a lot of mileage using their testing utilities like _array_api_for_tests in sklearn/utils/_testing.py and so on.

One thing I didn't mention over there is that you should probably try using https://github.com/data-apis/array-api-strict to check our conformance (sklearn does this too).

- staging ground to have array api use - testing for torch cpu only right now - testing regressor only right now - test to match numpy and torch r2 score - test that predictions are in same namespace - many issues with sklearn conformance in classifier - more details to come about onehotencoder issues

- Two private utility functions for the following: - Check and/or convert all to X's dtype, namespace, & device - Ensure that X is of a float type - Test function to ensure predictions have same context as X - Test function that fitted attributes have same context as X - Test function that ints are handled gracefulyl

- Replace custom private utility functions with move_to - Is existing utility from sklearn - Achieves same functionality and more - array_api_compat no longer used in imports - Torch download removed from TOML - Previous tests pass pre PR (on local Windows) - Torch matches NumPy test passes - Other new tests related to namespace, dtype, device fail - Those will be replaced in next commit with sklearn equiv - Remaining ruff errors not related to this PR scope

- Replace np with xp in some activations - I think scipy uses array API standard - When SCIPY_ARRAY_API is set - Hence no change to scipy.special funcs - Responds to eiviani-lanl comment - No speed change from np.tanh to xp.tanh

- xp.maximum requires both to be Tensor - 0 as scalar only works for np - numpy not needed in activations.py - use sklearn way to get namespace

- Control the dtypes with `xp.astype` - `xp.astype` is used >200 times in sklearn - `move_to` is used over >200 times in sklearn - I.e., sklearn learn uses these for device, dtype - `conftest.py` sets global behavior for array API fixture - sklearn tag tells check estimator that array API is supported - All `test_regression.py` functions pass on local Mac OS - Custom tests are yet to be removed - Seeking alternative existing tests from sklearn to replace

* Added `check_array_api_input_and_values` function. This is a public utility that is not covered by `check(estimator)` and will verify if public attributes match to NumPy runs given flexible choice of array namespace. * I checked many combinations over `GFDLRegressor` inputs except for Torch and NumPy seem to handle backends of bounded weight schemes with slight numerical differences. * `check_array_api_input_and_values` does not have enough flexibility like `rtol` or `atol` to control slight backend differences * Modified `test_sklearn_api_conformance` to check both array API and non-array API version. Made a pytest.xfail for array API when _NotAnArray is put in. This is an sklearn specific object that is incompatible with the array API tests. * Two of my custom tests were already covered by `test_sklearn_api_conformance` with the `array_api_dispatch=True` flag * sklearn.linear_model favors `check_same_namespace` as function to ensure `.predict` input matches the `.fit` input * I liked the idea of converting to what the user puts into `.predict` but I will revert to the standard that sklearn has set * This removes `move_to` instances in `GFDL().predict` * For the time being, I commented out a custom test about integer arrays. sklearn does not address this case.

* Removed conftest.py file which globally set some array API settings * Reverted back to original check_estimator test * Made separate check_estimator test for array API Includes skips if array API not set up * Use pytest.importorskip for "torch" test * array_api_strict bawks at hidden_layer_sizes.min() * Removed array_api_compat dependency

* `check_array_api_input_and_values` uses `make_classification` which is unstable for some weight schemes between torch, np * Shrunk number of cases for that test which calls `check_array_api_input_and_values` * Added custom `make_regression`-based test that checks public attributes and predictions

sdtemple · 2026-09-29T03:21:00Z

    X = np.zeros((30, 4))
    y = np.zeros((30,))
-    invalid_act = Classifier(hidden_layer_sizes=100,
+    invalid_act = Classifier(hidden_layer_sizes=(100,),


When I change to min(hidden_layer_sizes), this test failed. However, it works with xp.min(hidden_layer_sizes) as a scalar 100 or tuple (100,).

Possibly this should be reverted and addressed in #136

sdtemple · 2026-09-29T03:30:11Z

@eiviani-lanl This is rebased to the changes at #129 and I made necessary changes to namespace xp calls in .predict and .predict_proba.

This is a better reference for the time being, but I still need to remove the private utility _to_numpy_cpu_y with something that would pass muster in array-api-strict.

* Applied to `hidden_layer_sizes`

* Implemented argument array_namespace instead of torch only * Used `clone()` in custom test to cut redundancy. This is what `sklearn` does in the related test `_check_array_api_core`. * Removed the 3x for loop of different random states. * The custom `xp_make_regression` test remains but ideally it could be removed if we could get an MR change through `sklearn` that allows for control of the simulating data.

* In `test_xp_make_regression` we need to use `with config_context()` otherwise `sklearn` utilities revert to NumPy. This was a bug. * No handling of dtype for the time being. All tests are xp.float64. * Needed to control `atol` and `rtol` * `move_to` to `assert_allclose` on same namespace (NumPy) * Small code change to use the array_api_compat vendor of the other namespaces (PyTorch)

* `check_array_api_input` only * Values are evaluated with custom test that is based on `check_array_api_core` but gives control over simulating data, `atol` and `rtol` * Removed `pytest.importorskip("torch")` * Instead, use the utility from sklearn `yield_namespace_device_dtype_combinations()` * `if` block to scope to torch, cpu, float64 * Don't force to max dtype precision * All `WEIGHTS.keys()` and `ACTIVATIONS.keys()`

* SI disclosure: I did not use SI for this. This is my first set up of a CI, so hopefully it works smoothly. * Added new YML file with SCIPY_ARRAY_API=1 as the environment file * Only macos and ubuntu and python 3.14 * Only torch and array-api-strict installed

* Replace SciPy version of `log_sigmoid` and `sigmoid` with an `xp` version * `log_sigmoid` and `sigmoid` fail some array-api-strict regression tests. It is known from SciPy that `log_expit` is not yet supported. * Update `atol` and `rtol` values for `float32` tests. This is a pain identifying `atol` and `rtol` values suitable for 1e4 tests across platforms, backends, devices, and dtypes. The vast majority of tests pass with stricter than `atol=1e-0` and `rtol=1e-2` for `float32`. For tests that wouldn't pass a stricter threshold, I have verified that the coeffs are qualitatively the same (often a big coefficient 1e2 that may be off by a little over 1e-1). You can also inspect the singular values (manually) to see that they are also qualitatively the same.

* SI disclosure: I did not use SI * Implemented `xp` for base estimator's `partial_fit` method * Be wary of checking namespace without running `.fit()`, hence why you do not put this in `partial_fit` * First draft at adding `partial_fit` boolean to the custom test function * `xp.linalg.lstsq` does SVD as in `Ridge().fit()` but is not supported currently

* SI disclosure: I did not use SI * Replaced `scipy.special` versions of softmax functions with `xp` versions based on Emma's `sklearn` pull request * Updated `atol` and `rtol` bounds for tighter control in `xp_make_regression` test * Added quantitative comments about why we can't tighten further without failing the test (very few cases out of 720 tests over configurations) * We expect some classifier tests to fail, as the classifier will be addressed in lanl#130 which is a descendant of this PR. The vast majority that fail right now concern `partial_fit` in the `GFDLClassifier`

* Removed the `_NotAnArray` try/except block. The `sklearn` compliance function already handles which tests to run if SCIPY_ARRAY_API=1 * Removed the related test

* SI disclosure: I did not use SI, except I did use GPT 6.1 Sol when reviewing Emma's work * There were some Array API styles that Emma used in her sklearn PR that I preferred over mine. I also identified some redundant tests in my sklearn PR review. Here I am porting over those suggested changes. * eiviani-lanl/scikit-learn#2 * Use `xp.asarray` instead of `move_to` and `xp.astype` * Use `dtype=[xp.float64, xp.float32]` in `validate_data` * Only `xp.squeeze` when needed. Fixes a bug with classifier which will go upstream when I rebase lanl#130 * Removed redundant tests which were identified during review

sdtemple · 2026-10-26T18:01:32Z

+
+        self.enc_ = LabelBinarizer(sparse_output=False)
+        Y = self.enc_.fit_transform(Y)
+        self.classes_ = self.enc_.classes_


I changed this to LabelBinarizer, which has Array API support in scikit-learn, from OneHotEncoder. This is also what is used in Emma's sklearn PR.

In general, LabelBinarizer is for targets and OneHotEncoder is for features.

Side not: I think one-hot encoding is available from array-api-extras

https://data-apis.org/array-api-extra/generated/array_api_extra.one_hot.html

* SI disclosure: I used SI to write a simple `partial_fit` test function * Replaced `numpy.testing.assert_allclose` with `xpx.testing.assert_close` for array API tests * Decreased number of Array API test cases

* SI disclosure: I did not use SI in this commit * Copy and pasted the `xp.asarray` set up for the `self.W_` and `self.b_` initialization to `partial_fit` * Simplified some lines to decrease line number

- private utility function for OneHotEncoder - one line call to it in .fit() - GFDLRegressor only has to invoked base GFDL - GFDLClassifier has that 1 line change in .fit() - Test function for numpy vs torch matching - Copy, paste, modify from test_regression.py - Other test functions in regressor PR would be redundant

* Order of `unique_labels` mattered for conformance * Test with larger dataset for `make_classification`. Tyler doesn't like this test but for the commit it confirms similar results for torch versus numpy for ALL weight schemes * Change .predict_proba and .predict in classifier to xp array versions * scipy.special.logsumexp is supported for array API * 6 tests fail with array-api-strict versus 0 for compat. Issue appears at `_to_numpy_cpu_y` * We want to remove that private utility eventually

* AI disclosure: I consulted AI but wrote the edits myself * Copy, paste, and modify the tests from `test_regression.py` in lanl#129 * Use `move_to` on the `self.classes_` * `check_same_namespace` called individually in each estimator's methods

* SI disclosure: I did not use SI for this * Main edit to to put `classes` in `predict` on the same namespace as `X` so that you can index on `xp.argmax` * Revise `xp_make_classification` test to allow for `partial_fit` results

* SI disclosure: I did not use SI * Consistent with the classifier `.fit`, we want `classes` to be on `np` and `cpu` for `OneHotEncoder` if this is the first pass when the encoder is fit.

* `xp.linalg.solve` documation for `partial_fit` in non-ensemble GFDL estimators * `softmax` from @activations.py achieves the same thing as in `partial_fit` custom written with `scipy.special.logsumexp` * These would eventually need to be addressed in an ensemble branch in the same way * Reverted to `rtol=1e-3` for `test_partial_fit` which had `pinv` instability in `EnsembleGFDLClassifier`. It looks like Tyler will `xfail` this case in lanl#138

* SI disclosure: I did not use SI, but the basis of the changes from review of Emma's PR did use GPT 6.1 Sol * Use `LabelBinarizer` instead of `OneHotEncoder`. The `OneHotEncoder` is designed for features, not labels. See the `sklearn` documentation. The `LabelBinarizer` has Array API support, which means we don't have to do device transfers all of the time. * Add `ValueError` if only 1 class is specified in a classification problem. Had to modify the `np.zeros` to `np.arange() % 2` in an invalid activation/weight test so that the `ValueError` doesn't trigger first. * `rtol` spec in a `np.assert_allclose` test

* SI disclosure: I did not use SI * Simplify/shorten some code lines in classifier fit test * Use `xpx.testing.assert_close` instead * Include a partial fit Array API test for classifier which only checks `predict` and `predict_proba`

* SI disclosure: I did not use SI * Include `n_classes` as a parameter for Array API classifier test

* AI disclosure: I did not use AI for this * Skip on `check_array_api` for classifier tests * We will remove those skips once we address the classifier in PR lanl#130

* SI disclosure: I did not use SI * Replaced `scipy.special` versions of softmax functions with `xp` versions based on Emma's `sklearn` pull request * Updated `atol` and `rtol` bounds for tighter control in `xp_make_regression` test * Added quantitative comments about why we can't tighten further without failing the test (very few cases out of 720 tests over configurations) * We expect some classifier tests to fail, as the classifier will be addressed in lanl#130 which is a descendant of this PR. The vast majority that fail right now concern `partial_fit` in the `GFDLClassifier`

* SI disclosure: I did not use SI, except I did use GPT 6.1 Sol when reviewing Emma's work * There were some Array API styles that Emma used in her sklearn PR that I preferred over mine. I also identified some redundant tests in my sklearn PR review. Here I am porting over those suggested changes. * eiviani-lanl/scikit-learn#2 * Use `xp.asarray` instead of `move_to` and `xp.astype` * Use `dtype=[xp.float64, xp.float32]` in `validate_data` * Only `xp.squeeze` when needed. Fixes a bug with classifier which will go upstream when I rebase lanl#130 * Removed redundant tests which were identified during review

sdtemple commented Sep 14, 2026

View reviewed changes

Comment thread src/gfdl/tests/test_model.py

sdtemple commented Sep 14, 2026

View reviewed changes

Comment thread src/gfdl/model.py Outdated

sdtemple commented Sep 14, 2026

View reviewed changes

Comment thread src/gfdl/model.py Outdated

sdtemple requested a review from eiviani-lanl September 14, 2026 19:04

sdtemple self-assigned this Sep 14, 2026

Seth Temple and others added 9 commits September 24, 2026 17:51

WIP: activation functions with xp

1653503

- Replace np with xp in some activations - I think scipy uses array API standard - When SCIPY_ARRAY_API is set - Hence no change to scipy.special funcs - Responds to eiviani-lanl comment - No speed change from np.tanh to xp.tanh

WIP: zero must be tensor in max func

f5283ca

- xp.maximum requires both to be Tensor - 0 as scalar only works for np - numpy not needed in activations.py - use sklearn way to get namespace

sdtemple mentioned this pull request Sep 29, 2026

ENH: Array API for GFDLRegressor #129

Open

sdtemple force-pushed the sdtemple_torch_cpu_classifier branch from 8e156f6 to 0f9721e Compare September 29, 2026 02:02

sdtemple commented Sep 29, 2026

View reviewed changes

Comment thread src/gfdl/tests/test_model.py Outdated

sdtemple commented Sep 29, 2026

View reviewed changes

Comment thread src/gfdl/tests/test_model.py Outdated

sdtemple commented Sep 29, 2026

View reviewed changes

Comment thread src/gfdl/model.py Outdated

sdtemple commented Sep 29, 2026

View reviewed changes

Comment thread src/gfdl/model.py Outdated

sdtemple commented Sep 29, 2026

View reviewed changes

Comment thread src/gfdl/model.py Outdated

Seth Temple and others added 4 commits September 28, 2026 21:32

WIP,MAINT: xp.min instead of min or .min()

45b7090

* Applied to `hidden_layer_sizes`

sdtemple added 5 commits October 19, 2026 12:35

WIP,TST: sklearn api compliance already handles array API

d53840f

* Removed the `_NotAnArray` try/except block. The `sklearn` compliance function already handles which tests to run if SCIPY_ARRAY_API=1 * Removed the related test

sdtemple added array API standard support enhancement New feature or request labels Oct 20, 2026

sdtemple force-pushed the sdtemple_torch_cpu_classifier branch from a13e7b0 to d75b1ab Compare October 20, 2026 21:55

sdtemple mentioned this pull request Oct 21, 2026

WIP, ENH, TST: Array API support initial draft eiviani-lanl/scikit-learn#2

Open

sdtemple force-pushed the sdtemple_torch_cpu_classifier branch from 36ff568 to 4698abc Compare October 23, 2026 23:47

sdtemple commented Oct 26, 2026

View reviewed changes

sdtemple and others added 10 commits October 26, 2026 14:41

WIP,TST: xpx assert, small partial_fit test

01c1228

* SI disclosure: I used SI to write a simple `partial_fit` test function * Replaced `numpy.testing.assert_allclose` with `xpx.testing.assert_close` for array API tests * Decreased number of Array API test cases

WIP,ENH: xp.asarray in partial_fit

1e84cdf

* SI disclosure: I did not use SI in this commit * Copy and pasted the `xp.asarray` set up for the `self.W_` and `self.b_` initialization to `partial_fit` * Simplified some lines to decrease line number

WIP,ENH: partial_fit for classifier

44b6c8e

* SI disclosure: I did not use SI for this * Main edit to to put `classes` in `predict` on the same namespace as `X` so that you can index on `xp.argmax` * Revise `xp_make_classification` test to allow for `partial_fit` results

WIP,ENH: ensure classes for partial_fit are on np cpu

3cf5506

* SI disclosure: I did not use SI * Consistent with the classifier `.fit`, we want `classes` to be on `np` and `cpu` for `OneHotEncoder` if this is the first pass when the encoder is fit.

WIP,TST: modified test_model.py for Array API

c10ca08

* SI disclosure: I did not use SI * Simplify/shorten some code lines in classifier fit test * Use `xpx.testing.assert_close` instead * Include a partial fit Array API test for classifier which only checks `predict` and `predict_proba`

sdtemple force-pushed the sdtemple_torch_cpu_classifier branch from 4698abc to c10ca08 Compare October 26, 2026 21:33

WIP,TST: include multiclass in xp classifier test

d1d6708

* SI disclosure: I did not use SI * Include `n_classes` as a parameter for Array API classifier test

sdtemple mentioned this pull request Oct 27, 2026

WIP, ENH: Array API GFDLEnsembleClassifier estimator #142

Open

sdtemple changed the title ~~WIP, ENH: Torch CPU via Array API for GFDLClassifier~~ WIP, ENH: Array API for GFDLClassifier Oct 27, 2026

sdtemple changed the title ~~WIP, ENH: Array API for GFDLClassifier~~ WIP, ENH: Array API for GFDLClassifier Oct 27, 2026

Navigation Menu

Repository navigation

WIP, ENH: Array API for `GFDLClassifier` - #130

WIP, ENH: Array API for `GFDLClassifier`#130
sdtemple wants to merge 32 commits into
lanl:mainfrom
sdtemple:sdtemple_torch_cpu_classifier

sdtemple commented Sep 14, 2026 •

edited

Loading

Uh oh!

sdtemple commented Sep 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdtemple commented Sep 14, 2026

Uh oh!

tylerjereddy commented Sep 15, 2026

Uh oh!

sdtemple Sep 29, 2026

Uh oh!

sdtemple Oct 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdtemple commented Sep 29, 2026

Uh oh!

sdtemple Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sdtemple commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdtemple commented Sep 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdtemple commented Sep 14, 2026

Uh oh!

tylerjereddy commented Sep 15, 2026

Uh oh!

sdtemple Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

sdtemple Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdtemple commented Sep 29, 2026

Uh oh!

sdtemple Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sdtemple commented Sep 14, 2026 •

edited

Loading