Expected behavior
A leading null should not change the dimensions of the other geometries in a GeoSeries. Empty XYZ geometries and points whose Z is NaN should retain their dimensional metadata.
Actual behavior
The constructor path added in #3349 for #3333 converts local geometries through WKB and ST_GeomFromWKB. Adding a leading null causes these geometries to become 2D:
| Input geometry |
GeoSeries([geometry, None]) |
GeoSeries([None, geometry]) |
| POINT Z EMPTY |
3D |
2D |
| LINESTRING Z EMPTY |
3D |
2D |
| POLYGON Z EMPTY |
3D |
2D |
| POINT Z (1 2 NaN) |
3D |
2D |
| POINT Z (1 2 3) |
3D |
3D |
| POINT EMPTY |
2D |
2D |
Steps to reproduce
With a SedonaContext initialized using the current-master JVM jar:
import shapely
from sedona.spark.geopandas import GeoSeries
g = shapely.from_wkt("POINT Z EMPTY")
ordinary = GeoSeries([g, None]).to_geopandas().iloc[0]
leading_null = GeoSeries([None, g]).to_geopandas().iloc[1]
print(ordinary.wkt, shapely.get_coordinate_dimension(ordinary))
# POINT Z EMPTY 3
print(leading_null.wkt, shapely.get_coordinate_dimension(leading_null))
# POINT EMPTY 2Sedona version
2.0.0-SNAPSHOT, master commit 15e415a.
The Spark 4.1 JVM jar was freshly built from this commit with mvn -B -pl spark-shaded -am -Dspark=4.1 -Dscala=2.13 -DskipTests package. Both Python serializer modes reproduced the same results: the pure-Python fallback and a native extension freshly compiled from the same source. The six inputs above were compared through both constructor paths in each mode.
Environment
- API: Python / GeoPandas compatibility API
- Spark: 4.1.1; Scala: 2.13.17
- Java: 17.0.13; Python: 3.11.13
- Shapely: 2.1.2; GEOS: 3.13.1
- GeoPandas: 1.1.1; pandas: 2.2.3
- Local Spark, macOS arm64
Root cause and suggested fix
Direct JVM checks show that the WKB parser retains the 3D coordinate sequence. GeometrySerializer then chooses XY when the sequence has no coordinates or all Z values are NaN. JVM result rows are already 2D before Python decoding.
Replace the leading-null constructor's WKB conversion with construction using an explicit Spark GeometryType schema and Sedona's Python geometry serializer. Preserve the pandas index, row order, name, CRS, embedded SRIDs, and normalization of all pandas missing-value variants. Add dimensional regression cases alongside the existing constructor tests and verify Spark 3.5 and 4.1.
A local explicit-schema prototype preserved the dimensional cases, names, duplicate MultiIndex order, missing values, and CRS on Spark 4.1.1. One additional requirement is to preserve embedded SRIDs in pure-Python serialization: the fallback currently writes zero SRID bytes, so switching to it directly would regress the existing leading-null embedded-SRID case. The native serializer already preserves these SRIDs. A full replacement also needs M/ZM compatibility coverage, since the fallback Python serializer currently rejects measured geometries.
Changing the JVM serializer to treat every three-dimensional JTS sequence as XYZ would also promote ordinary XY data, because JTS can represent XY with a NaN Z slot.
The reproduction measures returned geometry dimensions directly. It does not rely on ST_NDims, ST_CoordDim, or ST_Zmflag, whose empty-geometry results were intentionally changed in #3360.
Existing issues
Expected behavior
A leading null should not change the dimensions of the other geometries in a GeoSeries. Empty XYZ geometries and points whose Z is NaN should retain their dimensional metadata.
Actual behavior
The constructor path added in #3349 for #3333 converts local geometries through WKB and ST_GeomFromWKB. Adding a leading null causes these geometries to become 2D:
Steps to reproduce
With a SedonaContext initialized using the current-master JVM jar:
Sedona version
2.0.0-SNAPSHOT, master commit 15e415a.
The Spark 4.1 JVM jar was freshly built from this commit with
mvn -B -pl spark-shaded -am -Dspark=4.1 -Dscala=2.13 -DskipTests package. Both Python serializer modes reproduced the same results: the pure-Python fallback and a native extension freshly compiled from the same source. The six inputs above were compared through both constructor paths in each mode.Environment
Root cause and suggested fix
Direct JVM checks show that the WKB parser retains the 3D coordinate sequence. GeometrySerializer then chooses XY when the sequence has no coordinates or all Z values are NaN. JVM result rows are already 2D before Python decoding.
Replace the leading-null constructor's WKB conversion with construction using an explicit Spark GeometryType schema and Sedona's Python geometry serializer. Preserve the pandas index, row order, name, CRS, embedded SRIDs, and normalization of all pandas missing-value variants. Add dimensional regression cases alongside the existing constructor tests and verify Spark 3.5 and 4.1.
A local explicit-schema prototype preserved the dimensional cases, names, duplicate MultiIndex order, missing values, and CRS on Spark 4.1.1. One additional requirement is to preserve embedded SRIDs in pure-Python serialization: the fallback currently writes zero SRID bytes, so switching to it directly would regress the existing leading-null embedded-SRID case. The native serializer already preserves these SRIDs. A full replacement also needs M/ZM compatibility coverage, since the fallback Python serializer currently rejects measured geometries.
Changing the JVM serializer to treat every three-dimensional JTS sequence as XYZ would also promote ordinary XY data, because JTS can represent XY with a NaN Z slot.
The reproduction measures returned geometry dimensions directly. It does not rely on ST_NDims, ST_CoordDim, or ST_Zmflag, whose empty-geometry results were intentionally changed in #3360.
Existing issues