RS_FromNetCDF fails on NetCDF4/HDF5 with filter type=32015 (e.g. ZSTD)

[BruAPAHE]

I encountered a runtime failure when using RS_FromNetCDF(content, 'v') on a valid NetCDF4 file stored in S3.

The error is:

Exception occurred while evaluating expression RS_FromNetCDF - inputs: [[B@xxxx, v], cause: Unknown filter type=32015

This happens because the file uses an HDF5 compression filter (likely BLOSC, filter ID 32015), which is not supported by the version of netcdf-java (4.6.11) used in netcdfAll.

Steps to reproduce:

Load .nc file containing BLOSC/ZSTD compression into Spark via binaryFile.

Try to evaluate RS_FromNetCDF(content, 'variable').

A workaround is to re-encode the file using nccopy or xarray to remove unsupported filters, but this limitation is undocumented and leads to runtime crashes.

Expected:

An informative exception or warning.

Ideally: support for filter 32015 

Environment:

Sedona: 1.7.1

Spark: 3.5.0

File: NetCDF4/HDF5 with ZSTD filter

[github-actions]

Thank you for your interest in Apache Sedona! We appreciate you opening your first issue. Contributions like yours help make Apache Sedona better.

[jiayuasu]

@BruAPAHE Thanks for reporting. I believe recently netcdf-java introduced this support. Do you want to create a PR to upgrade the dependency?

[minhpham1810]

Hi, I'd like to work on this issue. Filter 32015 is Zstandard (BLOSC is 32001). Upgrading netcdf-java won't fix this: even the latest release, 5.10.0, has no Zstandard filter. It only adds BLOSC, through the separate libblosc2-jna module.

Plan:

1. Rethrow UnknownFilterException with a clear message that names the filter ID and the nccopy -F none workaround, and document the supported filters.
2. Add Zstandard support by registering a netcdf-java FilterProvider for ID 32015, backed by zstd-jni, which Spark already ships.

Let me know if that works.

[jiayuasu]

ok. please go ahead