feat(zarr_fdw): add read-only Zarr v2/v3 scientific array FDW

[HamzaMPSY]

Summary

This PR introduces a read-only native Zarr FDW for querying multidimensional scientific arrays directly from PostgreSQL without importing or materializing the source dataset first.

The main use case is scientific and geospatial workloads where large arrays live in object storage while application, relational, or spatial metadata already lives in PostgreSQL/PostGIS.

For example, the goal is to make workflows such as this possible:

Zarr climate / EO data in object storage
                  +
        PostgreSQL / PostGIS data
                  ↓
          one SQL query

rather than requiring a separate ETL or Python/xarray service just to bridge the two systems.

Contribution-policy note: I am aware that the current repository guidance says new native FDWs are paused until the Wrappers API reaches v1.0. I am opening this as a draft/RFC to get maintainer feedback on whether this direction is interesting for Wrappers, how you would prefer the work to be split, and whether any of the framework-level changes should be proposed separately.

What is included

• Read-only Zarr v2 and a bounded Zarr v3 implementation.

• Direct Zarr v3 arrays and indexed sharded arrays with bounded range reads.

• S3-compatible storage.

• Anonymous HTTPS object access with deliberately restricted networking behavior.

• Confined local filesystem access.

• Raw, gzip/zlib, Blosc/LZ4 and supported Zarr v3 codec pipelines including Zstandard and CRC32C.

• Named N-dimensional arrays with rank 1–64.

• Numeric dimension selectors by:

  • exact index/value;
  • index/value lists;
  • index/value ranges.

• Conservative coordinate and chunk pruning with exact residual checks.

• Lazy chunk-index iteration rather than materializing the Cartesian chunk set.

• Bounded ordered prefetching, cancellation and a query-local compressed chunk cache.

• Missing-chunk / fill-value handling.

• CF packed-value decoding and CF time decoding for the currently supported calendar.

• Dataset metadata inspection through zarr_inspect().

• Scalar aggregate pushdown for count, sum, avg, min and max.

• Runtime Zarr I/O metrics exposed through EXPLAIN ANALYZE.

• CRS discovery and PostGIS integration.

• Point sampling, cell extraction, zonal statistics and spatial/time zonal statistics.

• Initial OME-Zarr 0.5 multiscale discovery and explicit rank-2 level execution.

• Deterministic MinIO-backed integration coverage plus unit/integration tests for codecs, chunking, selectors, spatial execution, missing chunks, sharding and safety bounds.

Execution model

A scan is reduced to a generic N-dimensional selection and then to a lazy chunk plan:

SQL predicates / dimension selectors / spatial constraints
                         ↓
                     Selection
                         ↓
                     ScanPlan
                         ↓
              lazy chunk-coordinate iteration
                         ↓
             bounded fetch / cache / range reads
                         ↓
                 codec + dtype decoding
                         ↓
             scientific-value transformations
                         ↓
                 exact residual checks
                    ↙          ↘
              SQL tuples     aggregate state

Selections are conservative: pruning is allowed to read more data than ultimately matches, but exact predicates are retained so pruning cannot change query results.

Large chunk sets are iterated lazily, and remote reads are bounded by both request count and byte budgets.

Why Zarr in PostgreSQL?

Zarr is increasingly used for cloud-native climate, weather, Earth-observation, microscopy and other multidimensional scientific datasets.

Those arrays can be hundreds of gigabytes, terabytes or larger, so loading them into PostgreSQL is generally the wrong storage model.

At the same time, the data scientists and applications need to relate those arrays to often already lives in Postgres:

• geometries and administrative regions;
• farms, assets and infrastructure;
• sensors and observations;
• samples and experimental metadata;
• application/business entities.

The intent of this FDW is therefore not to turn PostgreSQL into xarray or a general numerical-computing engine.

It is to allow PostgreSQL to select and reduce the relevant portions of a remote scientific array, ideally reading only the necessary chunks, and combine the result with relational/PostGIS data.

Current scope and intentional limitations

This implementation is deliberately fail-closed for scientific semantics that it does not understand.

Some current boundaries are:

• read-only;
• one value array per foreign-table scan;
• numeric rank-1 coordinate arrays for ordinary array dimensions;
• no string/categorical coordinates yet;
• coordinate CF packing is not silently decoded;
• one supported temporal dimension and a bounded calendar implementation;
• spatial operations currently target rectilinear rank-2 grids;
• polygon statistics use cell-center coverage rather than fractional cell area;
• no curvilinear grids, interpolation or rotated-grid handling;
• aggregate pushdown is limited to simple ungrouped aggregates;
• OME-Zarr support currently covers a bounded 0.5 multiscale subset.

Unsupported cases return explicit errors rather than attempting potentially incorrect scientific interpretation.

Framework changes

Part of the aggregate work requires Wrappers-level planner/executor hooks so an FDW can safely decide whether a particular aggregate query can be executed remotely and return the aggregate result correctly.

I would especially appreciate guidance on this part.

If preferred, I can separate the generic framework changes from the Zarr FDW into a prerequisite PR so they can be reviewed independently.

Testing

The implementation includes deterministic local and MinIO-backed test fixtures covering both metadata and actual data scans.

The tests exercise areas including:

• edge and missing chunks;
• fill values;
• compressed chunks;
• Zarr v2 and v3 layouts;
• Zarr v3 codecs and indexed sharding;
• range reads;
• generic dimensions;
• exact/list/range dimension selectors;
• aggregate execution;
• spatial and spatial/time operations;
• cancellation and bounded-resource behavior.

I am also validating the FDW against large public Zarr datasets over real S3-compatible and HTTPS endpoints to measure how much remote I/O can be avoided by selection and chunk pruning.

Feedback I am looking for

The main questions I would appreciate maintainer feedback on are:

1. Is a native Zarr/scientific-array FDW something the project would be interested in once native FDW contributions reopen?
2. Would you prefer the Wrappers-level aggregate/runtime changes to be proposed separately?
3. Is the current scope appropriate for one eventual FDW contribution, or would you prefer an initial smaller Zarr reader followed by the scientific/PostGIS functionality incrementally?

I'm happy to reshape or split the work based on what would make it easiest to review and maintain.