From 79ed8e8270a44e7ddbd459ca061a58b80a17e7d8 Mon Sep 17 00:00:00 2001 From: Jia Yu Date: Fri, 28 Aug 2026 12:16:07 -0700 Subject: [PATCH 1/6] [GH-3297] Drop Spark 3.4 support Spark 3.4 reached end of life in October 2024 with 3.4.4 as its final maintenance release, and the major cloud runtimes have since removed it. Build: - Remove the spark/spark-3.4 module and the sedona-spark-3.4 profile - Make 3.5 the default spark.version/spark.compat.version, and the profile that deploys the parent and common modules - Drop spark-3.5 from enable-all-submodules now that it is the default CI: remove the 3.4 rows from the java, python, r and example matrices. The python 3.4 row also carried the Python 3.8 and shapely 1 coverage, so that row moves to Spark 3.5 rather than being deleted. The Spark Connect step's `matrix.spark >= '3.4.0'` guard is now always true. Python: raise the pyspark floor to 3.5.0 and drop the pyspark < 3.4 skip markers from the raster tests. Scala: PartitionedFile's constructor is identical from 3.5.0 onwards, so StacPartitionReader builds one directly instead of reflectively choosing between the 7- and 8-argument forms. SchemaMergeUtils keeps its reflective fallback: StructType.merge(StructType, Boolean) does not exist in 3.5.0, only from 3.5.1. Docs: update the compatibility tables, compile and coordinate examples, and the release and snapshot build matrices, which still named 3.3. Also remove spark-version-converter.py, a Spark 2/3 helper whose target paths have not existed since the module layout changed. Closes #3297 --- .github/workflows/example.yml | 4 - .github/workflows/java.yml | 10 - .github/workflows/python.yml | 3 +- .github/workflows/r.yml | 2 +- docs/community/develop.md | 4 +- docs/community/develop.zh.md | 2 +- docs/community/publish.md | 15 +- docs/community/publish.zh.md | 15 +- docs/community/snapshot.md | 19 +- docs/community/snapshot.zh.md | 19 +- docs/setup/compile.md | 20 +- docs/setup/compile.zh.md | 20 +- docs/setup/emr.md | 2 +- docs/setup/emr.zh.md | 2 +- docs/setup/maven-coordinates.md | 60 +- docs/setup/maven-coordinates.zh.md | 60 +- docs/setup/platform.md | 30 +- docs/setup/platform.zh.md | 30 +- docs/tutorial/raster.md | 2 +- docs/tutorial/raster.zh.md | 2 +- docs/tutorial/sql.md | 2 +- docs/tutorial/sql.zh.md | 2 +- pom.xml | 26 +- python/pyproject.toml | 12 +- python/sedona/spark/sql/dataframe_api.py | 3 +- python/tests/raster/test_flexible_bands.py | 34 - python/tests/raster/test_pandas_udf.py | 5 - python/tests/raster/test_raster_udf_docs.py | 25 - python/tests/raster/test_serde.py | 10 - spark-version-converter.py | 96 -- spark/common/pom.xml | 2 +- .../sedona/sql/UDF/AbstractCatalog.scala | 2 +- .../internal/SchemaMergeUtils.scala | 4 +- .../io/stac/StacPartitionReader.scala | 78 +- spark/pom.xml | 3 +- spark/spark-3.4/.gitignore | 29 - spark/spark-3.4/pom.xml | 185 ---- ...pache.spark.sql.sources.DataSourceRegister | 3 - .../geopackage/GeoPackageDataSource.scala | 73 -- .../GeoPackagePartitionReader.scala | 107 -- .../GeoPackagePartitionReaderFactory.scala | 139 --- .../geopackage/GeoPackageScan.scala | 68 -- .../geopackage/GeoPackageScanBuilder.scala | 73 -- .../geopackage/GeoPackageTable.scala | 119 --- .../shapefile/ShapefileDataSource.scala | 101 -- .../shapefile/ShapefilePartition.scala | 27 - .../shapefile/ShapefilePartitionReader.scala | 287 ----- .../ShapefilePartitionReaderFactory.scala | 154 --- .../shapefile/ShapefileReadOptions.scala | 45 - .../datasources/shapefile/ShapefileScan.scala | 129 --- .../shapefile/ShapefileScanBuilder.scala | 71 -- .../shapefile/ShapefileTable.scala | 159 --- .../shapefile/ShapefileUtils.scala | 202 ---- .../sql/parser/SedonaSqlAstBuilder.scala | 40 - .../sedona/sql/parser/SedonaSqlParser.scala | 49 - .../GeoParquetMetadataDataSource.scala | 65 -- ...arquetMetadataPartitionReaderFactory.scala | 121 --- .../metadata/GeoParquetMetadataScan.scala | 67 -- .../GeoParquetMetadataScanBuilder.scala | 47 - .../metadata/GeoParquetMetadataTable.scala | 78 -- .../src/test/resources/log4j2.properties | 31 - .../apache/sedona/sql/Box2DCastSuite.scala | 144 --- .../apache/sedona/sql/Box3DCastSuite.scala | 118 --- .../sedona/sql/GeoPackageReaderTest.scala | 441 -------- .../sedona/sql/GeoParquetMetadataTests.scala | 152 --- .../sedona/sql/SQLSyntaxTestScala.scala | 72 -- .../apache/sedona/sql/ShapefileTests.scala | 985 ------------------ .../org/apache/sedona/sql/TestBaseScala.scala | 128 --- 68 files changed, 115 insertions(+), 5019 deletions(-) delete mode 100644 spark-version-converter.py delete mode 100644 spark/spark-3.4/.gitignore delete mode 100644 spark/spark-3.4/pom.xml delete mode 100644 spark/spark-3.4/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageDataSource.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReader.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReaderFactory.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScan.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScanBuilder.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageTable.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileDataSource.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartition.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReader.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReaderFactory.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileReadOptions.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScan.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScanBuilder.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileTable.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileUtils.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlAstBuilder.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlParser.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataDataSource.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataPartitionReaderFactory.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScan.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScanBuilder.scala delete mode 100644 spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataTable.scala delete mode 100644 spark/spark-3.4/src/test/resources/log4j2.properties delete mode 100644 spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box2DCastSuite.scala delete mode 100644 spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box3DCastSuite.scala delete mode 100644 spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoPackageReaderTest.scala delete mode 100644 spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoParquetMetadataTests.scala delete mode 100644 spark/spark-3.4/src/test/scala/org/apache/sedona/sql/SQLSyntaxTestScala.scala delete mode 100644 spark/spark-3.4/src/test/scala/org/apache/sedona/sql/ShapefileTests.scala delete mode 100644 spark/spark-3.4/src/test/scala/org/apache/sedona/sql/TestBaseScala.scala diff --git a/.github/workflows/example.yml b/.github/workflows/example.yml index 9d1f76ae391..6cd5e137a9e 100644 --- a/.github/workflows/example.yml +++ b/.github/workflows/example.yml @@ -53,10 +53,6 @@ jobs: spark-compat: '3.5' sedona: 1.8.0 hadoop: 3.3.4 - - spark: 3.4.2 - spark-compat: '3.4' - sedona: 1.8.0 - hadoop: 3.3.4 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml index 33aec60ff09..82c9db10b61 100644 --- a/.github/workflows/java.yml +++ b/.github/workflows/java.yml @@ -81,16 +81,6 @@ jobs: scala: 2.12.15 jdk: '11' skipTests: '' - - spark: 3.4.0 - scala: 2.13.8 - jdk: '11' - skipTests: '' - skipLibPostalTests: 'true' - - spark: 3.4.0 - scala: 2.12.15 - jdk: '11' - skipTests: '' - skipLibPostalTests: 'true' - spark: 3.5.8 scala: 2.12.15 jdk: '17' diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml index 856897c2c0d..e3ce8fc9a2e 100644 --- a/.github/workflows/python.yml +++ b/.github/workflows/python.yml @@ -72,7 +72,7 @@ jobs: scala: '2.12.8' java: '11' python: '3.9' - - spark: '3.4.0' + - spark: '3.5.0' scala: '2.12.8' java: '11' python: '3.8' @@ -158,7 +158,6 @@ jobs: uv sync uv run pytest -v tests/core/test_rdd.py tests/sql/test_dataframe_api.py - name: Run Spark Connect tests - if: ${{ matrix.spark >= '3.4.0' }} run: | cd python export SPARK_REMOTE=local diff --git a/.github/workflows/r.yml b/.github/workflows/r.yml index 2d253a65eb7..421d778e9f1 100644 --- a/.github/workflows/r.yml +++ b/.github/workflows/r.yml @@ -59,7 +59,7 @@ jobs: strategy: fail-fast: true matrix: - spark: [3.4.0, 3.5.0] + spark: [3.5.0] hadoop: [3] scala: [2.12.15] r: [oldrel, release] diff --git a/docs/community/develop.md b/docs/community/develop.md index 53783aeeaf5..3e15a348357 100644 --- a/docs/community/develop.md +++ b/docs/community/develop.md @@ -70,8 +70,8 @@ Make sure you reload the `pom.xml` or reload the maven project. The IDE will ask In a terminal, go to the Sedona root folder. Run `mvn clean install`. All tests will take more than 15 minutes. To only build the project jars, run `mvn clean install -DskipTests`. !!!Note - `mvn clean install` will compile Sedona with Spark 3.3 and Scala 2.12. If you have a different version of Spark in $SPARK_HOME, make sure to specify that using -Dspark command line arg. - For example, to compile sedona with Spark 3.4 and Scala 2.12, use: `mvn clean install -Dspark=3.4 -Dscala=2.12` + `mvn clean install` will compile Sedona with Spark 3.5 and Scala 2.12. If you have a different version of Spark in $SPARK_HOME, make sure to specify that using -Dspark command line arg. + For example, to compile sedona with Spark 4.0 and Scala 2.13, use: `mvn clean install -Dspark=4.0 -Dscala=2.13` More details can be found on [Compile Sedona](../setup/compile.md) diff --git a/docs/community/develop.zh.md b/docs/community/develop.zh.md index f8f9fc56071..b9f73cf0d29 100644 --- a/docs/community/develop.zh.md +++ b/docs/community/develop.zh.md @@ -71,7 +71,7 @@ IDE 通常难以理解 Sedona 中较为复杂的项目结构。 在终端中进入 Sedona 根目录运行 `mvn clean install`。全部测试需要超过 15 分钟。如果只想构建 jar，运行 `mvn clean install -DskipTests`。 !!!Note - `mvn clean install` 默认基于 Spark 3.3、Scala 2.12 编译 Sedona。如果 $SPARK_HOME 中是不同版本的 Spark，请通过 `-Dspark` 命令行参数指定。例如基于 Spark 3.4 与 Scala 2.12 编译：`mvn clean install -Dspark=3.4 -Dscala=2.12`。 + `mvn clean install` 默认基于 Spark 3.5、Scala 2.12 编译 Sedona。如果 $SPARK_HOME 中是不同版本的 Spark，请通过 `-Dspark` 命令行参数指定。例如基于 Spark 4.0 与 Scala 2.13 编译：`mvn clean install -Dspark=4.0 -Dscala=2.13`。 更多细节请参阅 [编译 Sedona](../setup/compile.md)。 diff --git a/docs/community/publish.md b/docs/community/publish.md index b6519cd62d8..a3118e0b6f8 100644 --- a/docs/community/publish.md +++ b/docs/community/publish.md @@ -127,7 +127,7 @@ rm -rf $LOCAL_DIR && git clone --depth 1 --branch $TAG $REPO_URL $LOCAL_DIR && c MAVEN_PLUGIN_VERSION="2.3.2" # Define Spark and Scala versions -declare -a SPARK_VERSIONS=("3.4" "3.5" "4.0" "4.1") +declare -a SPARK_VERSIONS=("3.5" "4.0" "4.1") declare -a SCALA_VERSIONS=("2.12" "2.13") # Function to get Java version for Spark version @@ -383,19 +383,6 @@ verify_java_version() { echo "✓ Verified: Maven wrapper is using Java $mvn_java_version" } -# Compile for Spark 3.4 with Java 11 -JAVA_VERSION=$(get_java_version "3.4") -MVN_WRAPPER=$(create_mvn_wrapper $JAVA_VERSION) -verify_java_version $MVN_WRAPPER $JAVA_VERSION - -echo "Compiling for Spark 3.4 with Scala 2.12 using Java $JAVA_VERSION..." -cd apache-sedona-${SEDONA_VERSION}-src && $MVN_WRAPPER clean && $MVN_WRAPPER install -DskipTests -Dspark=3.4 -Dscala=2.12 && cd .. -cp apache-sedona-${SEDONA_VERSION}-src/spark-shaded/target/sedona-*${SEDONA_VERSION}.jar apache-sedona-${SEDONA_VERSION}-bin/ - -echo "Compiling for Spark 3.4 with Scala 2.13 using Java $JAVA_VERSION..." -cd apache-sedona-${SEDONA_VERSION}-src && $MVN_WRAPPER clean && $MVN_WRAPPER install -DskipTests -Dspark=3.4 -Dscala=2.13 && cd .. -cp apache-sedona-${SEDONA_VERSION}-src/spark-shaded/target/sedona-*${SEDONA_VERSION}.jar apache-sedona-${SEDONA_VERSION}-bin/ - # Compile for Spark 3.5 with Java 11 JAVA_VERSION=$(get_java_version "3.5") MVN_WRAPPER=$(create_mvn_wrapper $JAVA_VERSION) diff --git a/docs/community/publish.zh.md b/docs/community/publish.zh.md index a731c42e03c..2a950445527 100644 --- a/docs/community/publish.zh.md +++ b/docs/community/publish.zh.md @@ -125,7 +125,7 @@ rm -rf $LOCAL_DIR && git clone --depth 1 --branch $TAG $REPO_URL $LOCAL_DIR && c MAVEN_PLUGIN_VERSION="2.3.2" # Spark 与 Scala 版本 -declare -a SPARK_VERSIONS=("3.4" "3.5" "4.0" "4.1") +declare -a SPARK_VERSIONS=("3.5" "4.0" "4.1") declare -a SCALA_VERSIONS=("2.12" "2.13") # 根据 Spark 版本确定所需 Java 版本 @@ -381,19 +381,6 @@ verify_java_version() { echo "✓ Verified: Maven wrapper is using Java $mvn_java_version" } -# 使用 Java 11 编译 Spark 3.4 -JAVA_VERSION=$(get_java_version "3.4") -MVN_WRAPPER=$(create_mvn_wrapper $JAVA_VERSION) -verify_java_version $MVN_WRAPPER $JAVA_VERSION - -echo "Compiling for Spark 3.4 with Scala 2.12 using Java $JAVA_VERSION..." -cd apache-sedona-${SEDONA_VERSION}-src && $MVN_WRAPPER clean && $MVN_WRAPPER install -DskipTests -Dspark=3.4 -Dscala=2.12 && cd .. -cp apache-sedona-${SEDONA_VERSION}-src/spark-shaded/target/sedona-*${SEDONA_VERSION}.jar apache-sedona-${SEDONA_VERSION}-bin/ - -echo "Compiling for Spark 3.4 with Scala 2.13 using Java $JAVA_VERSION..." -cd apache-sedona-${SEDONA_VERSION}-src && $MVN_WRAPPER clean && $MVN_WRAPPER install -DskipTests -Dspark=3.4 -Dscala=2.13 && cd .. -cp apache-sedona-${SEDONA_VERSION}-src/spark-shaded/target/sedona-*${SEDONA_VERSION}.jar apache-sedona-${SEDONA_VERSION}-bin/ - # 使用 Java 11 编译 Spark 3.5 JAVA_VERSION=$(get_java_version "3.5") MVN_WRAPPER=$(create_mvn_wrapper $JAVA_VERSION) diff --git a/docs/community/snapshot.md b/docs/community/snapshot.md index 7ddde59d816..408515e98b3 100644 --- a/docs/community/snapshot.md +++ b/docs/community/snapshot.md @@ -59,15 +59,18 @@ rm -f pom.xml.* mvn -q -B clean release:prepare -Dtag={{ sedona_create_release.current_git_tag }} -DreleaseVersion={{ sedona_create_release.current_version }} -DdevelopmentVersion={{ sedona_create_release.current_snapshot }} -Dresume=false -DdryRun=true -Penable-all-submodules -Darguments="-DskipTests" mvn -q -B release:clean -Penable-all-submodules -# Spark 3.3 and Scala 2.12 -mvn -q deploy -DskipTests -Dspark=3.3 -Dscala=2.12 +# Spark 3.5 and Scala 2.12 +mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.12 -# Spark 3.3 and Scala 2.13 -mvn -q deploy -DskipTests -Dspark=3.3 -Dscala=2.13 +# Spark 3.5 and Scala 2.13 +mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.13 -# Spark 3.4 and Scala 2.12 -mvn -q deploy -DskipTests -Dspark=3.4 -Dscala=2.12 +# Spark 4.0 and Scala 2.12 +mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.12 -# Spark 3.4 and Scala 2.13 -mvn -q deploy -DskipTests -Dspark=3.4 -Dscala=2.13 +# Spark 4.0 and Scala 2.13 +mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.13 + +# Spark 4.1 and Scala 2.13 +mvn -q deploy -DskipTests -Dspark=4.1 -Dscala=2.13 ``` diff --git a/docs/community/snapshot.zh.md b/docs/community/snapshot.zh.md index a904ba5c94a..8aa80ed114c 100644 --- a/docs/community/snapshot.zh.md +++ b/docs/community/snapshot.zh.md @@ -59,15 +59,18 @@ rm -f pom.xml.* mvn -q -B clean release:prepare -Dtag={{ sedona_create_release.current_git_tag }} -DreleaseVersion={{ sedona_create_release.current_version }} -DdevelopmentVersion={{ sedona_create_release.current_snapshot }} -Dresume=false -DdryRun=true -Penable-all-submodules -Darguments="-DskipTests" mvn -q -B release:clean -Penable-all-submodules -# Spark 3.3 与 Scala 2.12 -mvn -q deploy -DskipTests -Dspark=3.3 -Dscala=2.12 +# Spark 3.5 与 Scala 2.12 +mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.12 -# Spark 3.3 与 Scala 2.13 -mvn -q deploy -DskipTests -Dspark=3.3 -Dscala=2.13 +# Spark 3.5 与 Scala 2.13 +mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.13 -# Spark 3.4 与 Scala 2.12 -mvn -q deploy -DskipTests -Dspark=3.4 -Dscala=2.12 +# Spark 4.0 与 Scala 2.12 +mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.12 -# Spark 3.4 与 Scala 2.13 -mvn -q deploy -DskipTests -Dspark=3.4 -Dscala=2.13 +# Spark 4.0 与 Scala 2.13 +mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.13 + +# Spark 4.1 与 Scala 2.13 +mvn -q deploy -DskipTests -Dspark=4.1 -Dscala=2.13 ``` diff --git a/docs/setup/compile.md b/docs/setup/compile.md index e8bd18a3605..cea28c5cb30 100644 --- a/docs/setup/compile.md +++ b/docs/setup/compile.md @@ -46,28 +46,28 @@ To compile all modules, please make sure you are in the root folder of all modul Geotools jars will be packaged into the produced fat jars. !!!note - By default, this command will compile Sedona with Spark 3.4 and Scala 2.12 + By default, this command will compile Sedona with Spark 3.5 and Scala 2.12 ### Compile with different targets User can specify `-Dspark` and `-Dscala` command line options to compile with different targets. Available targets are: -* `-Dspark`: `{major}.{minor}`: For example, specify `-Dspark=3.4` to build for Spark 3.4. +* `-Dspark`: `{major}.{minor}`: For example, specify `-Dspark=3.5` to build for Spark 3.5. * `-Dscala`: `2.12` or `2.13` -=== "Spark 3.4+ Scala 2.12" +=== "Spark 3.5+ Scala 2.12" ``` - mvn clean install -DskipTests -Dspark=3.4 -Dscala=2.12 + mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.12 ``` - Please replace `3.4` with Spark major.minor version when building for higher Spark versions. -=== "Spark 3.4+ Scala 2.13" + Please replace `3.5` with Spark major.minor version when building for higher Spark versions. +=== "Spark 3.5+ Scala 2.13" ``` - mvn clean install -DskipTests -Dspark=3.4 -Dscala=2.13 + mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.13 ``` - Please replace `3.4` with Spark major.minor version when building for higher Spark versions. + Please replace `3.5` with Spark major.minor version when building for higher Spark versions. !!!tip - To get the Sedona Spark Shaded jar with all GeoTools jars included, simply append `-Dgeotools` option. The command is like this:`mvn clean install -DskipTests -Dscala=2.12 -Dspark=3.4 -Dgeotools` + To get the Sedona Spark Shaded jar with all GeoTools jars included, simply append `-Dgeotools` option. The command is like this:`mvn clean install -DskipTests -Dscala=2.12 -Dspark=3.5 -Dgeotools` ### Download staged jars @@ -78,7 +78,7 @@ Sedona uses GitHub Actions to automatically generate jars per commit. You can go 1) Set up Spark (download if needed) and environment variables ```bash -export SPARK_VERSION=3.4.0 # or another supported version +export SPARK_VERSION=3.5.0 # or another supported version wget https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz tar -xvzf spark-${SPARK_VERSION}-bin-hadoop3.tgz rm spark-${SPARK_VERSION}-bin-hadoop3.tgz diff --git a/docs/setup/compile.zh.md b/docs/setup/compile.zh.md index 0c83337eff4..9c1291ddb43 100644 --- a/docs/setup/compile.zh.md +++ b/docs/setup/compile.zh.md @@ -46,28 +46,28 @@ Sedona Scala/Java 代码是一个多模块项目，每个模块都是 Scala/Java Geotools 相关 jar 会被打入生成的 fat jar 中。 !!!note - 默认情况下，该命令会针对 Spark 3.4 和 Scala 2.12 编译 Sedona。 + 默认情况下，该命令会针对 Spark 3.5 和 Scala 2.12 编译 Sedona。 ### 针对不同目标编译 可使用 `-Dspark` 与 `-Dscala` 命令行参数指定不同的目标： -* `-Dspark`：`{major}.{minor}`，例如 `-Dspark=3.4` 表示针对 Spark 3.4 编译。 +* `-Dspark`：`{major}.{minor}`，例如 `-Dspark=3.5` 表示针对 Spark 3.5 编译。 * `-Dscala`：`2.12` 或 `2.13` -=== "Spark 3.4+ Scala 2.12" +=== "Spark 3.5+ Scala 2.12" ``` - mvn clean install -DskipTests -Dspark=3.4 -Dscala=2.12 + mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.12 ``` - 若要针对更高 Spark 版本编译，请将 `3.4` 替换为对应的 Spark major.minor 版本。 -=== "Spark 3.4+ Scala 2.13" + 若要针对更高 Spark 版本编译，请将 `3.5` 替换为对应的 Spark major.minor 版本。 +=== "Spark 3.5+ Scala 2.13" ``` - mvn clean install -DskipTests -Dspark=3.4 -Dscala=2.13 + mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.13 ``` - 若要针对更高 Spark 版本编译，请将 `3.4` 替换为对应的 Spark major.minor 版本。 + 若要针对更高 Spark 版本编译，请将 `3.5` 替换为对应的 Spark major.minor 版本。 !!!tip - 如需获取打包好所有 GeoTools jar 的 Sedona Spark Shaded jar，只需追加 `-Dgeotools` 选项，例如：`mvn clean install -DskipTests -Dscala=2.12 -Dspark=3.4 -Dgeotools` + 如需获取打包好所有 GeoTools jar 的 Sedona Spark Shaded jar，只需追加 `-Dgeotools` 选项，例如：`mvn clean install -DskipTests -Dscala=2.12 -Dspark=3.5 -Dgeotools` ### 下载 staged jar @@ -78,7 +78,7 @@ Sedona 通过 GitHub Actions 在每次提交时自动生成 jar。您可以在 [ 1) 准备 Spark（如未安装则下载）并设置环境变量 ```bash -export SPARK_VERSION=3.4.0 # 或其他受支持的版本 +export SPARK_VERSION=3.5.0 # 或其他受支持的版本 wget https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz tar -xvzf spark-${SPARK_VERSION}-bin-hadoop3.tgz rm spark-${SPARK_VERSION}-bin-hadoop3.tgz diff --git a/docs/setup/emr.md b/docs/setup/emr.md index a5ae043bbba..2d759b81495 100644 --- a/docs/setup/emr.md +++ b/docs/setup/emr.md @@ -22,7 +22,7 @@ We recommend Sedona-1.3.1-incubating and above for EMR. In the tutorial, we use This tutorial is tested on EMR on EC2 with EMR Studio (notebooks). EMR on EC2 uses YARN to manage resources. !!!note - If you are using Spark 3.4+ and Scala 2.12, please use `sedona-spark-shaded-3.4_2.12`. Please pay attention to the Spark version postfix and Scala version postfix. + If you are using Spark 3.5+ and Scala 2.12, please use `sedona-spark-shaded-3.5_2.12`. Please pay attention to the Spark version postfix and Scala version postfix. ## Prepare initialization script diff --git a/docs/setup/emr.zh.md b/docs/setup/emr.zh.md index a0f251ccacc..dc208ed59a6 100644 --- a/docs/setup/emr.zh.md +++ b/docs/setup/emr.zh.md @@ -22,7 +22,7 @@ 本教程在带 EMR Studio（notebook）的 EMR on EC2 上完成测试。EMR on EC2 使用 YARN 进行资源管理。 !!!note - 如果您使用 Spark 3.4+ 与 Scala 2.12，请使用 `sedona-spark-shaded-3.4_2.12`。请注意 Spark 与 Scala 的版本后缀。 + 如果您使用 Spark 3.5+ 与 Scala 2.12，请使用 `sedona-spark-shaded-3.5_2.12`。请注意 Spark 与 Scala 的版本后缀。 ## 准备初始化脚本 diff --git a/docs/setup/maven-coordinates.md b/docs/setup/maven-coordinates.md index 364217463de..7c9b29b1270 100644 --- a/docs/setup/maven-coordinates.md +++ b/docs/setup/maven-coordinates.md @@ -29,7 +29,7 @@ Apache Sedona provides different packages for each supported version of Spark. -Please use the artifact with Spark major.minor version in the artifact name. For example, for Spark 3.4, the artifacts to use should be `sedona-spark-shaded-3.4_2.12`. +Please use the artifact with Spark major.minor version in the artifact name. For example, for Spark 3.5, the artifacts to use should be `sedona-spark-shaded-3.5_2.12`. If you are using the Scala 2.13 builds of Spark, please use the corresponding packages for Scala 2.13, which are suffixed by `_2.13`. @@ -37,21 +37,6 @@ The optional GeoTools library is required if you want to use raster operators. V !!! abstract "Sedona with Apache Spark and Scala 2.12" - === "Spark 3.4 and Scala 2.12" - - ```xml - - org.apache.sedona - sedona-spark-shaded-3.4_2.12 - {{ sedona.current_version }} - - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 and Scala 2.12" ```xml @@ -86,21 +71,6 @@ The optional GeoTools library is required if you want to use raster operators. V !!! abstract "Sedona with Apache Spark and Scala 2.13" - === "Spark 3.4 and Scala 2.13" - - ```xml - - org.apache.sedona - sedona-spark-shaded-3.4_2.13 - {{ sedona.current_version }} - - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 and Scala 2.13" ```xml @@ -192,7 +162,7 @@ The optional GeoTools library is required if you want to use raster operators. V Apache Sedona provides different packages for each supported version of Spark. -Please use the artifacts with Spark major.minor version in the artifact name. For example, for Spark 3.4, the artifacts to use should be `sedona-spark-3.4_2.12`. +Please use the artifacts with Spark major.minor version in the artifact name. For example, for Spark 3.5, the artifacts to use should be `sedona-spark-3.5_2.12`. If you are using the Scala 2.13 builds of Spark, please use the corresponding packages for Scala 2.13, which are suffixed by `_2.13`. @@ -200,19 +170,6 @@ The optional GeoTools library is required if you want to use raster operators. V !!! abstract "Sedona with Apache Spark and Scala 2.12" - === "Spark 3.4 and Scala 2.12" - ```xml - - org.apache.sedona - sedona-spark-3.4_2.12 - {{ sedona.current_version }} - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 and Scala 2.12" ```xml @@ -242,19 +199,6 @@ The optional GeoTools library is required if you want to use raster operators. V !!! abstract "Sedona with Apache Spark and Scala 2.13" - === "Spark 3.4 and Scala 2.13" - ```xml - - org.apache.sedona - sedona-spark-3.4_2.13 - {{ sedona.current_version }} - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 and Scala 2.13" ```xml diff --git a/docs/setup/maven-coordinates.zh.md b/docs/setup/maven-coordinates.zh.md index 96ae59b6a03..a02cff8f9ba 100644 --- a/docs/setup/maven-coordinates.zh.md +++ b/docs/setup/maven-coordinates.zh.md @@ -29,7 +29,7 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 -请使用 artifact 名称中包含 Spark major.minor 版本号的 artifact。例如，对于 Spark 3.4，应使用 `sedona-spark-shaded-3.4_2.12`。 +请使用 artifact 名称中包含 Spark major.minor 版本号的 artifact。例如，对于 Spark 3.5，应使用 `sedona-spark-shaded-3.5_2.12`。 如果使用基于 Scala 2.13 编译的 Spark，则应使用对应的 Scala 2.13 版本，其后缀为 `_2.13`。 @@ -37,21 +37,6 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 !!! abstract "Sedona 与 Apache Spark + Scala 2.12" - === "Spark 3.4 与 Scala 2.12" - - ```xml - - org.apache.sedona - sedona-spark-shaded-3.4_2.12 - {{ sedona.current_version }} - - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 与 Scala 2.12" ```xml @@ -86,21 +71,6 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 !!! abstract "Sedona 与 Apache Spark + Scala 2.13" - === "Spark 3.4 与 Scala 2.13" - - ```xml - - org.apache.sedona - sedona-spark-shaded-3.4_2.13 - {{ sedona.current_version }} - - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 与 Scala 2.13" ```xml @@ -192,7 +162,7 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 -请使用 artifact 名称中包含 Spark major.minor 版本号的 artifact。例如，对于 Spark 3.4，应使用 `sedona-spark-3.4_2.12`。 +请使用 artifact 名称中包含 Spark major.minor 版本号的 artifact。例如，对于 Spark 3.5，应使用 `sedona-spark-3.5_2.12`。 如果使用基于 Scala 2.13 编译的 Spark，则应使用对应的 Scala 2.13 版本（后缀为 `_2.13`）。 @@ -200,19 +170,6 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 !!! abstract "Sedona 与 Apache Spark + Scala 2.12" - === "Spark 3.4 与 Scala 2.12" - ```xml - - org.apache.sedona - sedona-spark-3.4_2.12 - {{ sedona.current_version }} - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 与 Scala 2.12" ```xml @@ -242,19 +199,6 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 !!! abstract "Sedona 与 Apache Spark + Scala 2.13" - === "Spark 3.4 与 Scala 2.13" - ```xml - - org.apache.sedona - sedona-spark-3.4_2.13 - {{ sedona.current_version }} - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` === "Spark 3.5 与 Scala 2.13" ```xml diff --git a/docs/setup/platform.md b/docs/setup/platform.md index c788981ceda..25abf6a6ecf 100644 --- a/docs/setup/platform.md +++ b/docs/setup/platform.md @@ -21,29 +21,29 @@ Sedona binary releases are compiled by Java 11/17 and Scala 2.12/2.13 and tested **Java Requirements:** -- Spark 3.4 & 3.5: Java 11 +- Spark 3.5: Java 11 - Spark 4.0 & 4.1: Java 17 -**Note:** Java 8 support is dropped since Sedona 1.8.0. Spark 3.3 support is dropped since Sedona 1.8.0. +**Note:** Java 8 support is dropped since Sedona 1.8.0. Spark 3.3 support is dropped since Sedona 1.8.0. Spark 3.4 support is dropped since Sedona 2.0.0. === "Sedona Scala/Java" - | | Spark 3.4| Spark 3.5 | Spark 4.0 | Spark 4.1 | - |:---------:|:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 |✅ |✅ |✅ | | - | Scala 2.13 |✅ |✅ |✅ |✅ | + | | Spark 3.5 | Spark 4.0 | Spark 4.1 | + |:---------:|:---------:|:---------:|:---------:| + | Scala 2.12 |✅ |✅ | | + | Scala 2.13 |✅ |✅ |✅ | === "Sedona Python" - | | Spark 3.4 (Scala 2.12)|Spark 3.5 (Scala 2.12)| Spark 4.0 (Scala 2.13)| Spark 4.1 (Scala 2.13)| - |:---------:|:---------:|:---------:|:---------:|:---------:| - | Python 3.7 | ✅ | ✅ | | | - | Python 3.8 | ✅ | ✅ | | | - | Python 3.9 | ✅ | ✅ | ✅ | ✅ | - | Python 3.10 | ✅ | ✅ | ✅ | ✅ | + | | Spark 3.5 (Scala 2.12)| Spark 4.0 (Scala 2.13)| Spark 4.1 (Scala 2.13)| + |:---------:|:---------:|:---------:|:---------:| + | Python 3.7 | ✅ | | | + | Python 3.8 | ✅ | | | + | Python 3.9 | ✅ | ✅ | ✅ | + | Python 3.10 | ✅ | ✅ | ✅ | === "Sedona R" - | | Spark 3.4 | Spark 3.5 | Spark 4.0 | Spark 4.1 | - |:---------:|:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 | ✅ | ✅ | ✅ | | + | | Spark 3.5 | Spark 4.0 | Spark 4.1 | + |:---------:|:---------:|:---------:|:---------:| + | Scala 2.12 | ✅ | ✅ | | diff --git a/docs/setup/platform.zh.md b/docs/setup/platform.zh.md index 0c51162b14e..5f3699382c5 100644 --- a/docs/setup/platform.zh.md +++ b/docs/setup/platform.zh.md @@ -21,29 +21,29 @@ Sedona 二进制版本由 Java 11/17 与 Scala 2.12/2.13 编译，并在以下 **Java 要求：** -- Spark 3.4 与 3.5：Java 11 +- Spark 3.5：Java 11 - Spark 4.0 与 4.1：Java 17 -**注意：** 自 Sedona 1.8.0 起不再支持 Java 8。自 Sedona 1.8.0 起不再支持 Spark 3.3。 +**注意：** 自 Sedona 1.8.0 起不再支持 Java 8。自 Sedona 1.8.0 起不再支持 Spark 3.3。自 Sedona 2.0.0 起不再支持 Spark 3.4。 === "Sedona Scala/Java" - | | Spark 3.4| Spark 3.5 | Spark 4.0 | Spark 4.1 | - |:---------:|:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 |✅ |✅ |✅ | | - | Scala 2.13 |✅ |✅ |✅ |✅ | + | | Spark 3.5 | Spark 4.0 | Spark 4.1 | + |:---------:|:---------:|:---------:|:---------:| + | Scala 2.12 |✅ |✅ | | + | Scala 2.13 |✅ |✅ |✅ | === "Sedona Python" - | | Spark 3.4 (Scala 2.12)|Spark 3.5 (Scala 2.12)| Spark 4.0 (Scala 2.13)| Spark 4.1 (Scala 2.13)| - |:---------:|:---------:|:---------:|:---------:|:---------:| - | Python 3.7 | ✅ | ✅ | | | - | Python 3.8 | ✅ | ✅ | | | - | Python 3.9 | ✅ | ✅ | ✅ | ✅ | - | Python 3.10 | ✅ | ✅ | ✅ | ✅ | + | | Spark 3.5 (Scala 2.12)| Spark 4.0 (Scala 2.13)| Spark 4.1 (Scala 2.13)| + |:---------:|:---------:|:---------:|:---------:| + | Python 3.7 | ✅ | | | + | Python 3.8 | ✅ | | | + | Python 3.9 | ✅ | ✅ | ✅ | + | Python 3.10 | ✅ | ✅ | ✅ | === "Sedona R" - | | Spark 3.4 | Spark 3.5 | Spark 4.0 | Spark 4.1 | - |:---------:|:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 | ✅ | ✅ | ✅ | | + | | Spark 3.5 | Spark 4.0 | Spark 4.1 | + |:---------:|:---------:|:---------:|:---------:| + | Scala 2.12 | ✅ | ✅ | | diff --git a/docs/tutorial/raster.md b/docs/tutorial/raster.md index 40f79d61652..e8403a0ac09 100644 --- a/docs/tutorial/raster.md +++ b/docs/tutorial/raster.md @@ -85,7 +85,7 @@ If you already have a SparkSession (Wherobots, AWS EMR, Databricks), skip ahead ) sedona = SedonaContext.create(config) ``` - Replace `3.3` with the major.minor version of your Spark install (for example `sedona-spark-shaded-3.4_2.12`). + Replace `3.3` with the major.minor version of your Spark install (for example `sedona-spark-shaded-4.0_2.12`). You can also register Sedona by passing `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` to `spark-submit` or `spark-shell`. diff --git a/docs/tutorial/raster.zh.md b/docs/tutorial/raster.zh.md index a2085fc409f..8eab3ed10b9 100644 --- a/docs/tutorial/raster.zh.md +++ b/docs/tutorial/raster.zh.md @@ -85,7 +85,7 @@ Scala、Java、Python、R 等所有 Sedona 语言绑定都已支持栅格能力 ) sedona = SedonaContext.create(config) ``` - 请将 `sedona-spark-shaded-3.3` 中的 `3.3` 替换为对应的 Spark 主.次版本号，例如 `sedona-spark-shaded-3.4_2.12`。 + 请将 `sedona-spark-shaded-3.3` 中的 `3.3` 替换为对应的 Spark 主.次版本号，例如 `sedona-spark-shaded-4.0_2.12`。 你也可以通过给 `spark-submit` 或 `spark-shell` 传入 `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` 来注册 Sedona。 diff --git a/docs/tutorial/sql.md b/docs/tutorial/sql.md index e4371f64b64..eb0df79887e 100644 --- a/docs/tutorial/sql.md +++ b/docs/tutorial/sql.md @@ -107,7 +107,7 @@ You can add additional Spark runtime config to the config builder. For example, 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \ getOrCreate() ``` - If you are using a different Spark version, please replace the `3.3` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-3.4_2.12:{{ sedona.current_version }}`. + If you are using a different Spark version, please replace the `3.3` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-4.0_2.12:{{ sedona.current_version }}`. ## Initiate SedonaContext diff --git a/docs/tutorial/sql.zh.md b/docs/tutorial/sql.zh.md index 2bbf8b5af29..6c00808c451 100644 --- a/docs/tutorial/sql.zh.md +++ b/docs/tutorial/sql.zh.md @@ -107,7 +107,7 @@ SedonaSQL 详细 API 说明请参阅 [SedonaSQL API](../api/sql/Overview.md)。 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \ getOrCreate() ``` - 如使用其他 Spark 版本，请将 sedona-spark-shaded 包名中的 `3.3` 替换为对应的 Spark major.minor 版本，例如 `sedona-spark-shaded-3.4_2.12:{{ sedona.current_version }}`。 + 如使用其他 Spark 版本，请将 sedona-spark-shaded 包名中的 `3.3` 替换为对应的 Spark major.minor 版本，例如 `sedona-spark-shaded-4.0_2.12:{{ sedona.current_version }}`。 ## 初始化 SedonaContext diff --git a/pom.xml b/pom.xml index 60dd981ad34..5d168dc7f08 100644 --- a/pom.xml +++ b/pom.xml @@ -82,8 +82,8 @@ 2.12 - 3.4.0 - 3.4 + 3.5.0 + 3.5 3 2.25.4 0.11.0 @@ -712,21 +712,6 @@ - - sedona-spark-3.4 - - - spark - 3.4 - - - - 3.4.0 - 3.4 - 2.19.0 - false - - sedona-spark-3.5 @@ -739,8 +724,7 @@ 3.5.0 3.5 2.20.0 - - true + false @@ -762,7 +746,7 @@ 2.13.17 2.13 - + true @@ -785,7 +769,7 @@ 2.13.17 2.13 - + true diff --git a/python/pyproject.toml b/python/pyproject.toml index 07a8a2aae2f..e76d2d564d6 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -38,16 +38,16 @@ dependencies = [ [project.optional-dependencies] spark = [ - "pyspark>=3.4.0,<4.1.0; python_version < '3.10'", - "pyspark>=3.4.0,<4.2.0; python_version >= '3.10'", + "pyspark>=3.5.0,<4.1.0; python_version < '3.10'", + "pyspark>=3.5.0,<4.2.0; python_version >= '3.10'", ] pydeck-map = ["geopandas", "pydeck==0.8.0"] kepler-map = ["geopandas", "keplergl==0.3.2"] flink = ["apache-flink>=1.19.0"] db = ["sedonadb[geopandas]; python_version >= '3.9'"] all = [ - "pyspark>=3.4.0,<4.1.0; python_version < '3.10'", - "pyspark>=3.4.0,<4.2.0; python_version >= '3.10'", + "pyspark>=3.5.0,<4.1.0; python_version < '3.10'", + "pyspark>=3.5.0,<4.2.0; python_version >= '3.10'", "geopandas", "pydeck==0.8.0", "keplergl==0.3.2", @@ -76,8 +76,8 @@ dev = [ # cannot set geopandas>=0.14.4 since it doesn't support python 3.8, so we pin fiona to <1.10.0 "fiona<1.10.0", "pyarrow", - "pyspark>=3.4.0,<4.1.0; python_version < '3.10'", - "pyspark>=3.4.0,<4.2.0; python_version >= '3.10'", + "pyspark>=3.5.0,<4.1.0; python_version < '3.10'", + "pyspark>=3.5.0,<4.2.0; python_version >= '3.10'", "keplergl==0.3.2", "pydeck==0.8.0", "pystac==1.5.0", diff --git a/python/sedona/spark/sql/dataframe_api.py b/python/sedona/spark/sql/dataframe_api.py index b42e46a002a..5aadafa0248 100644 --- a/python/sedona/spark/sql/dataframe_api.py +++ b/python/sedona/spark/sql/dataframe_api.py @@ -29,7 +29,8 @@ from pyspark.sql.connect.column import Column as ConnectColumn from pyspark.sql.utils import is_remote except ImportError: - # be backwards compatible with Spark < 3.4 + # pyspark.sql.connect requires pandas and pyarrow, which are not Sedona + # dependencies; fall back to the non-Connect path when they are absent. def is_remote(): return False diff --git a/python/tests/raster/test_flexible_bands.py b/python/tests/raster/test_flexible_bands.py index b7728eae090..f5b95ac2681 100644 --- a/python/tests/raster/test_flexible_bands.py +++ b/python/tests/raster/test_flexible_bands.py @@ -18,7 +18,6 @@ import math import numpy as np -import pyspark import pytest from pyspark.sql.functions import col, expr, udf from tests.test_base import TestBase @@ -27,9 +26,6 @@ class TestFlexibleBands(TestBase): - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_reduce_bands_4_to_1(self): """Reduce from 4 bands to 1 band (NDVI-like calculation).""" spark = self.spark @@ -63,9 +59,6 @@ def ndvi_like(raster): for val in band: assert math.isfinite(val), f"Expected finite value, got {val}" - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_increase_bands_3_to_6(self): """Increase from 3 bands to 6 (original + squared features).""" spark = self.spark @@ -107,9 +100,6 @@ def stack_features(raster): original = float(0 + y * 4 + x) assert band4[y * 4 + x] == original**2 - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_dtype_change_int_to_float(self): """Change dtype from int32 to float32 while keeping same band count.""" spark = self.spark @@ -136,9 +126,6 @@ def normalize(raster): for val in band: assert 0.0 <= val <= 1.0, f"Expected [0,1], got {val}" - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_mixed_band_and_dtype_change(self): """Simultaneous band count reduction and dtype change.""" spark = self.spark @@ -172,9 +159,6 @@ def mean_bands(raster): expected = 1.0 + y * 4 + x assert abs(band[y * 4 + x] - expected) < 0.01 - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_jvm_mapalgebra_after_band_change(self): """JVM-side RS_MapAlgebra works on a raster that had bands changed by UDF.""" spark = self.spark @@ -203,9 +187,6 @@ def select_band1(raster): expected = float(y * 4 + x + 100) assert band[y * 4 + x] == expected - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_reduce_bands_8_to_1_argmax(self): """8 bands to 1 band via argmax (KMeans-like cluster assignment).""" spark = self.spark @@ -237,9 +218,6 @@ def argmax_band(raster): for val in band: assert val == 7.0, f"Expected 7.0, got {val}" - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_metadata_survives_band_change(self): """CRS, affine transform, and dimensions survive a band count change.""" spark = self.spark @@ -275,9 +253,6 @@ def reduce_to_1(raster): assert abs(result["scale_y"] - (-10.0)) < 0.001 assert result["srid"] == 3857 - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_nodata_defaults_to_inherited(self): """Without nodata=, each output band inherits it from the input band.""" spark = self.spark @@ -300,9 +275,6 @@ def reduce_to_1(raster): ) assert result["nodata"] == 0.0 - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_nodata_scalar_overrides_inherited(self): """A scalar nodata= reaches the JVM instead of the inherited value.""" spark = self.spark @@ -333,9 +305,6 @@ def mask(raster): assert result["counted"] == 12 assert result["total"] == 12 - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_nodata_per_band_sequence(self): """A sequence nodata= assigns a different value to each output band.""" spark = self.spark @@ -366,9 +335,6 @@ def widen(raster): for band, expected in enumerate([-1.0, -2.0, -3.0, -4.0] * 2, start=1): assert result[f"nodata{band}"] == expected, f"band {band}" - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_nodata_agrees_between_python_and_jvm_when_widening(self): """bands_meta and RS_BandNoDataValue report the same value for added bands.""" spark = self.spark diff --git a/python/tests/raster/test_pandas_udf.py b/python/tests/raster/test_pandas_udf.py index ad4d54ff247..92c963ab5b5 100644 --- a/python/tests/raster/test_pandas_udf.py +++ b/python/tests/raster/test_pandas_udf.py @@ -17,8 +17,6 @@ import numpy as np import pandas as pd -import pyspark -import pytest import rasterio from pyspark.sql.functions import expr, pandas_udf from pyspark.sql.types import IntegerType @@ -26,9 +24,6 @@ class TestRasterPandasUDF(TestBase): - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_raster_as_param(self): spark = TestRasterPandasUDF.spark df = spark.range(10).withColumn( diff --git a/python/tests/raster/test_raster_udf_docs.py b/python/tests/raster/test_raster_udf_docs.py index a158520da10..aee38b70738 100644 --- a/python/tests/raster/test_raster_udf_docs.py +++ b/python/tests/raster/test_raster_udf_docs.py @@ -23,7 +23,6 @@ import math import numpy as np -import pyspark import pytest import rasterio.fill from pyspark.sql.functions import col, expr, udf @@ -39,10 +38,6 @@ "4, 3, 100, 100, 10, -10, 0, 0, 3857)" ) -requires_spark_34 = pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" -) - class TestRasterUdfDocExamples(TestBase): @@ -51,7 +46,6 @@ def _four_band_df(self): # ---- "Raster to scalar" ---------------------------------------------- - @requires_spark_34 def test_raster_to_scalar(self): df = self._four_band_df() @@ -63,7 +57,6 @@ def mean_udf(raster): # Band means are 5.5, 6.5, 7.5, 8.5. assert result == pytest.approx(7.0) - @requires_spark_34 def test_raster_to_scalar_registered_for_sql(self): """The register-by-name form: pass the decorated UDF, no returnType.""" sedona = self.spark @@ -79,7 +72,6 @@ def mean_udf(raster): # ---- "Raster to raster" ---------------------------------------------- - @requires_spark_34 def test_raster_to_raster(self): df = self._four_band_df() @@ -105,7 +97,6 @@ def mask_udf(raster): # Every band-0 value is below 1400, so the whole mask is set. assert all(value == 1.0 for value in result["band"]) - @requires_spark_34 def test_raster_to_raster_with_nodata(self): """The nodata= form from 'Setting NODATA on the output'.""" NODATA = -9999.0 @@ -136,7 +127,6 @@ def mask_udf(raster): assert result["nodata"] == NODATA assert result["counted"] == 11 - @requires_spark_34 def test_per_band_nodata_sequence(self): """The nodata=[...] form, including float('nan') for 'no NODATA'.""" df = self._four_band_df() @@ -160,7 +150,6 @@ def two_bands(raster): # ---- "NDVI, as map algebra and as a UDF" ------------------------------ - @requires_spark_34 def test_ndvi_udf_matches_map_algebra(self): """Both NDVI forms on the page must produce the same raster.""" df = self._four_band_df() @@ -192,7 +181,6 @@ def ndvi(raster): # ---- "Two rasters" ---------------------------------------------------- - @requires_spark_34 def test_two_raster_udf(self): @udf(returnType=RasterType()) def plus_five(raster): @@ -224,7 +212,6 @@ def delta(after, before): assert result["num_bands"] == 1 assert all(value == pytest.approx(5.0) for value in result["band"]) - @requires_spark_34 def test_two_raster_udf_keeps_holes_invalid(self): """A hole in either input must stay a hole, not become a plausible number. @@ -287,7 +274,6 @@ def delta(a, b): # ---- "Using rasterio inside a UDF" ------------------------------------ - @requires_spark_34 def test_rasterio_inside_udf(self): """fillnodata must actually fill, and the recipe keeps the NODATA declared.""" # Punch a -9999 hole at pixel 5, then run the documented recipe. The sentinel @@ -337,7 +323,6 @@ def fill_udf(raster): assert result["nodata"] == -9999.0 assert result["counted"] == 12 - @requires_spark_34 def test_rasterio_fill_keeps_unreachable_holes_invalid(self): """Cells fillnodata cannot reach keep the sentinel AND stay flagged invalid. @@ -378,7 +363,6 @@ def punch_and_fill(raster): assert result["nodata"] == -9999.0 assert result["counted"] == 144 - remaining - @requires_spark_34 def test_as_rasterio_does_not_carry_nodata(self): """Pins the documented gap: GDAL cannot see Sedona's NODATA.""" df = self.spark.range(1).withColumn( @@ -402,7 +386,6 @@ def probe(raster): assert mask_values == "[255]" # everything reported valid assert sedona_nodata == "5.0" # the raster itself knows - @requires_spark_34 def test_numpy_scalar_nodata_accepted(self): """nodata= accepts NumPy scalars, not just Python floats.""" raster = self.spark.sql(f"SELECT {FOUR_BAND} AS rast").first()["rast"] @@ -413,7 +396,6 @@ def test_numpy_scalar_nodata_accepted(self): value ).__name__ - @requires_spark_34 def test_nodata_nan_clears_inherited_value(self): """float('nan') must clear a NODATA the source has, not silently inherit it.""" df = self.spark.range(1).withColumn( @@ -436,7 +418,6 @@ def clear_nodata(raster): assert result["nodata"] is None assert result["counted"] == 12 # no pixel is skipped - @requires_spark_34 def test_nodata_wider_than_source_dtype(self): """A byte source widened to float64 may carry a nodata a byte could not hold.""" df = self.spark.range(1).withColumn( @@ -459,7 +440,6 @@ def widen(raster): ) assert result["nodata"] == -99999.0 - @requires_spark_34 def test_inherited_nodata_is_retyped_with_output_pixels(self): """Inherited NODATA must not leave byte metadata on a float64 output.""" df = self.spark.range(1).withColumn( @@ -549,7 +529,6 @@ def test_reinterpreted_nodata_is_masked_before_serialization( assert np.count_nonzero(np.isnan(masked)) == 1 assert np.all(masked[~np.isnan(masked)] == 7) - @requires_spark_34 def test_int8_negative_values_are_reinterpreted(self): """The dtype table's claim that int8 -2 reads back as 254.""" df = self._four_band_df() @@ -598,7 +577,6 @@ def test_nodata_must_fit_an_integral_output_dtype(self): out = raster.with_bands(integral, nodata=254.0) assert out.bands_meta[0].nodata == pytest.approx(254.0) - @requires_spark_34 def test_inherited_nodata_must_fit_output_dtype(self): """Narrowing a raster whose inherited NODATA no uint8 pixel can hold raises.""" raster = self.spark.sql( @@ -609,7 +587,6 @@ def test_inherited_nodata_must_fit_output_dtype(self): with pytest.raises(ValueError, match="inherited"): raster.with_bands(arr) - @requires_spark_34 def test_int8_nodata_matches_reinterpreted_pixels(self): """nodata on an int8 band is stored the way the pixels are: -2 becomes 254.""" df = self.spark.range(1).withColumn("rast", expr(FOUR_BAND)) @@ -633,7 +610,6 @@ def i8(raster): assert result["band"][0] == 254.0 assert result["counted"] == 11 # the reinterpreted hole is excluded - @requires_spark_34 def test_uint32_nodata_matches_reinterpreted_pixels(self): """nodata above 2**31-1 on a uint32 band is reinterpreted like the pixels.""" df = self.spark.range(1).withColumn("rast", expr(FOUR_BAND)) @@ -657,7 +633,6 @@ def u32(raster): assert result["band"][0] == -1.0 assert result["counted"] == 11 - @requires_spark_34 def test_float32_nodata_is_coerced_to_float32(self): """A nodata that is not float32-exact is rounded to what the pixels hold.""" df = self.spark.range(1).withColumn("rast", expr(FOUR_BAND)) diff --git a/python/tests/raster/test_serde.py b/python/tests/raster/test_serde.py index 49c59507225..2b9f4456bf3 100644 --- a/python/tests/raster/test_serde.py +++ b/python/tests/raster/test_serde.py @@ -16,7 +16,6 @@ # under the License. import numpy as np -import pyspark import pytest import rasterio from pyspark.sql.functions import col, expr, udf @@ -197,9 +196,6 @@ def test_nodata_override_trailer_survives_python_round_trip(self): ), ], ) - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_with_bands_retypes_packed_sample_models( self, num_bands, data_type, sample_model, expected_pixel_type ): @@ -230,9 +226,6 @@ def rebuild(raster): assert result["pixel_type"] == expected_pixel_type - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_serialize_round_trip_via_udf(self): """Deserialize → serialize round-trip through a raster-returning UDF. Verifies that the Python-serialized bytes can be deserialized back @@ -265,9 +258,6 @@ def test_serialize_round_trip_via_udf(self): round_tripped.close() original.close() - @pytest.mark.skipif( - pyspark.__version__ < "3.4", reason="requires Spark 3.4 or higher" - ) def test_serialize_preserves_metadata(self): """Verify CRS, affine, and dimensions survive serialize round-trip.""" from sedona.spark.raster import raster_serde diff --git a/spark-version-converter.py b/spark-version-converter.py deleted file mode 100644 index 05452e5f693..00000000000 --- a/spark-version-converter.py +++ /dev/null @@ -1,96 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# Python 3 -import fileinput -import sys - -spark2_anchor = "SPARK2 anchor" -spark3_anchor = "SPARK3 anchor" -files = [ - "sql/src/main/scala/org/apache/sedona/sql/UDF/UdfRegistrator.scala", - "sql/src/main/scala/org/apache/spark/sql/sedona_sql/strategy/join/TraitJoinQueryExec.scala", - "sql/src/main/scala/org/apache/spark/sql/sedona_sql/strategy/join/JoinQueryDetector.scala", - "sql/src/main/scala/org/apache/spark/sql/sedona_sql/strategy/join/BroadcastIndexJoinExec.scala", - "sql/src/main/scala/org/apache/spark/sql/sedona_sql/io/GeotiffFileFormat.scala", -] - - -def switch_version(line): - if line[:2] == "//": - print(line[2:], end="") # enable code - return "enabled" - else: - print("//" + line, end="") # disable code - return "disabled" - - -def enable_version(line): - if line[:2] == "//": - print(line[2:], end="") # enable code - return "enabled" - else: - print(line, end="") - return "enabled before" - - -def disable_version(line): - if line[:2] == "//": - print(line, end="") - return "disabled before" - else: - print("//" + line, end="") # disable code - return "disabled" - - -def parse_file(filepath, argv): - conversion_result_spark2 = "" - conversion_result_spark3 = "" - if argv[1] == "spark2": - with fileinput.FileInput(filepath, inplace=True) as file: - for line in file: - if spark2_anchor in line: - conversion_result_spark2 = ( - spark2_anchor + " " + enable_version(line) - ) - elif spark3_anchor in line: - conversion_result_spark3 = ( - spark3_anchor + " " + disable_version(line) - ) - else: - print(line, end="") - return conversion_result_spark2 + " and " + conversion_result_spark3 - elif argv[1] == "spark3": - with fileinput.FileInput(filepath, inplace=True) as file: - for line in file: - if spark2_anchor in line: - conversion_result_spark2 = ( - spark2_anchor + " " + disable_version(line) - ) - elif spark3_anchor in line: - conversion_result_spark3 = ( - spark3_anchor + " " + enable_version(line) - ) - else: - print(line, end="") - return conversion_result_spark2 + " and " + conversion_result_spark3 - else: - return "wrong spark version" - - -for filepath in files: - print(filepath + ": " + parse_file(filepath, sys.argv)) diff --git a/spark/common/pom.xml b/spark/common/pom.xml index bcaca8c737d..4e6701518ce 100644 --- a/spark/common/pom.xml +++ b/spark/common/pom.xml @@ -95,7 +95,7 @@ org.slf4j slf4j-log4j12 - + org.apache.logging.log4j log4j-slf4j2-impl diff --git a/spark/common/src/main/scala/org/apache/sedona/sql/UDF/AbstractCatalog.scala b/spark/common/src/main/scala/org/apache/sedona/sql/UDF/AbstractCatalog.scala index eb1ee1e9891..8989c035b0d 100644 --- a/spark/common/src/main/scala/org/apache/sedona/sql/UDF/AbstractCatalog.scala +++ b/spark/common/src/main/scala/org/apache/sedona/sql/UDF/AbstractCatalog.scala @@ -116,7 +116,7 @@ abstract class AbstractCatalog { // Builds the catalyst aggregate for a UDAF so the FunctionRegistry.builtin entry can be the // real implementation rather than a non-invocable placeholder. The required Spark API is // package-private in Scala (public in bytecode) and moved between versions, hence reflection: - // Spark 3.4/3.5 expose UserDefinedAggregator.scalaAggregator(children), Spark 4.x the + // Spark 3.5 exposes UserDefinedAggregator.scalaAggregator(children), Spark 4.x the // ScalaAggregator companion's apply(udaf, children). private lazy val builtinAggregateBuilder : Option[(UserDefinedFunction, Seq[Expression]) => Expression] = { diff --git a/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/internal/SchemaMergeUtils.scala b/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/internal/SchemaMergeUtils.scala index d49ab69ab3f..6e638fe27e8 100644 --- a/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/internal/SchemaMergeUtils.scala +++ b/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/geoparquet/internal/SchemaMergeUtils.scala @@ -55,12 +55,12 @@ object SchemaMergeUtils extends Logging { caseSensitive: Boolean): StructType = { mergeMethodWithCaseSensitive match { case Some(method) => - // Use the cached method with caseSensitive parameter (Spark 3.5 or later) + // Use the cached method with caseSensitive parameter (Spark 3.5.1 or later) method .invoke(schema1, schema2, caseSensitive.asInstanceOf[AnyRef]) .asInstanceOf[StructType] case None => - // Fall back to merge without caseSensitive parameter (Spark 3.4) + // Fall back to merge without caseSensitive parameter (Spark 3.5.0) schema1.merge(schema2) } } diff --git a/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacPartitionReader.scala b/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacPartitionReader.scala index 64960a4ed3a..307a5195614 100644 --- a/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacPartitionReader.scala +++ b/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacPartitionReader.scala @@ -21,6 +21,7 @@ package org.apache.spark.sql.sedona_sql.io.stac import com.fasterxml.jackson.databind.ObjectMapper import org.apache.hadoop.conf.Configuration import org.apache.spark.broadcast.Broadcast +import org.apache.spark.paths.SparkPath import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.json.JSONOptionsInRead import org.apache.spark.sql.connector.read.PartitionReader @@ -34,7 +35,6 @@ import org.apache.spark.sql.types.{StringType, StructType} import org.apache.spark.util.SerializableConfiguration import java.io.{File, PrintWriter} -import java.lang.reflect.Constructor import scala.io.Source class StacPartitionReader( @@ -209,79 +209,17 @@ class StacPartitionReader( } /** - * Create a PartitionedFile instance using reflection. The constructor parameters differ between - * these versions, so we need to handle both cases. For Spark 3.4 and below, the constructor has - * 7 parameters, while for Spark 3.5 and above, it has 8 parameters. Additionally, the type of - * the second parameter may be `SparkPath` in some cases, which requires special handling. + * Create a PartitionedFile instance for a locally staged STAC item file. * * @param currentFile * The file to create the PartitionedFile for. * @return * The created PartitionedFile instance. - * @throws NoSuchMethodException - * If no suitable constructor is found. */ - def createPartitionedFile(currentFile: File): PartitionedFile = { - val partitionedFileClass = - Class.forName("org.apache.spark.sql.execution.datasources.PartitionedFile") - val constructors = partitionedFileClass.getConstructors - val constructor = constructors - .find(_.getParameterCount == 7) - .getOrElse( - constructors - .find(_.getParameterCount == 8) - .getOrElse( - throw new NoSuchMethodException("No constructor with 7 or 8 parameters found"))) - - val params = if (constructor.getParameterCount == 7) { - val secondParamType = constructor.getParameterTypes()(1) - if (secondParamType.getName == "org.apache.spark.paths.SparkPath") { - Array( - null, - createSparkPath(currentFile.getPath), - java.lang.Long.valueOf(0L), - java.lang.Long.valueOf(currentFile.length()), - Array.empty[String], - java.lang.Long.valueOf(0L), - java.lang.Long.valueOf(0L)) - } else { - Array( - null, - currentFile.getPath, - java.lang.Long.valueOf(0L), - java.lang.Long.valueOf(currentFile.length()), - Array.empty[String], - java.lang.Long.valueOf(0L), - java.lang.Long.valueOf(0L)) - } - } else { - Array( - null, - createSparkPath(currentFile.getPath), - java.lang.Long.valueOf(0L), - java.lang.Long.valueOf(currentFile.length()), - Array.empty[String], - java.lang.Long.valueOf(0L), - java.lang.Long.valueOf(0L), - null) - } - - constructor.newInstance(params: _*).asInstanceOf[PartitionedFile] - } - - /** - * Create a SparkPath instance using reflection. This is needed to support both Spark 3.3 and - * below and Spark 3.4 and above. - * - * @param pathString - * The path to create the SparkPath for. - * @return - * The created SparkPath instance. - */ - def createSparkPath(pathString: String): Object = { - val sparkPathClass = Class.forName("org.apache.spark.paths.SparkPath") - val constructor: Constructor[_] = sparkPathClass.getDeclaredConstructor(classOf[String]) - constructor.setAccessible(true) // Make the private constructor accessible - constructor.newInstance(pathString).asInstanceOf[Object] - } + def createPartitionedFile(currentFile: File): PartitionedFile = + PartitionedFile( + partitionValues = null, + filePath = SparkPath.fromPathString(currentFile.getPath), + start = 0L, + length = currentFile.length()) } diff --git a/spark/pom.xml b/spark/pom.xml index 7c58581fc8b..bc21727640c 100644 --- a/spark/pom.xml +++ b/spark/pom.xml @@ -48,7 +48,6 @@ enable-all-submodules - spark-3.5 spark-4.0 spark-4.1 @@ -77,7 +76,7 @@ org.slf4j slf4j-log4j12 - + org.apache.logging.log4j log4j-slf4j2-impl diff --git a/spark/spark-3.4/.gitignore b/spark/spark-3.4/.gitignore deleted file mode 100644 index f34cc0c65b4..00000000000 --- a/spark/spark-3.4/.gitignore +++ /dev/null @@ -1,29 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -/target/ -/.settings/ -/.classpath -/.project -/dependency-reduced-pom.xml -/doc/ -/.idea/ -*.iml -/latest/ -/spark-warehouse/ -/metastore_db/ -*.log diff --git a/spark/spark-3.4/pom.xml b/spark/spark-3.4/pom.xml deleted file mode 100644 index f3b220351a1..00000000000 --- a/spark/spark-3.4/pom.xml +++ /dev/null @@ -1,185 +0,0 @@ - - - - 4.0.0 - - org.apache.sedona - sedona-spark-parent-${spark.compat.version}_${scala.compat.version} - 2.0.0-SNAPSHOT - ../pom.xml - - sedona-spark-3.4_${scala.compat.version} - - ${project.groupId}:${project.artifactId} - A cluster computing system for processing large-scale spatial data: SQL API for Spark 3.4. - https://sedona.apache.org/ - jar - - - false - - - - - org.apache.sedona - sedona-common - ${project.version} - - - com.fasterxml.jackson.core - * - - - it.geosolutions.jaiext.jiffle - * - - - org.codehaus.janino - * - - - - - org.apache.sedona - sedona-spark-common-${spark.compat.version}_${scala.compat.version} - ${project.version} - - - - org.apache.spark - spark-core_${scala.compat.version} - - - org.apache.spark - spark-sql_${scala.compat.version} - - - org.apache.hadoop - hadoop-client - - - org.apache.logging.log4j - log4j-1.2-api - - - org.geotools - gt-main - - - org.geotools - gt-referencing - - - org.geotools - gt-epsg-hsql - - - org.geotools - gt-geotiff - - - org.geotools - gt-coverage - - - org.geotools - gt-arcgrid - - - org.locationtech.jts - jts-core - - - org.wololo - jts2geojson - - - com.fasterxml.jackson.core - * - - - - - org.scala-lang - scala-library - - - org.scala-lang.modules - scala-collection-compat_${scala.compat.version} - - - org.scalatest - scalatest_${scala.compat.version} - - - org.mockito - mockito-inline - - - org.testcontainers - testcontainers - 2.0.2 - test - - - org.testcontainers - testcontainers-minio - 2.0.2 - test - - - io.minio - minio - - - com.squareup.okhttp3 - okhttp - - - org.apache.hadoop - hadoop-aws - ${hadoop.version} - test - - - org.apache.hadoop - hadoop-client-api - ${hadoop.version} - test - - - - src/main/scala - - - net.alchim31.maven - scala-maven-plugin - - - org.scalatest - scalatest-maven-plugin - - - org.scalastyle - scalastyle-maven-plugin - - - - diff --git a/spark/spark-3.4/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister b/spark/spark-3.4/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister deleted file mode 100644 index ae1de3d8bd2..00000000000 --- a/spark/spark-3.4/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister +++ /dev/null @@ -1,3 +0,0 @@ -org.apache.spark.sql.execution.datasources.v2.geoparquet.metadata.GeoParquetMetadataDataSource -org.apache.sedona.sql.datasources.shapefile.ShapefileDataSource -org.apache.sedona.sql.datasources.geopackage.GeoPackageDataSource diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageDataSource.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageDataSource.scala deleted file mode 100644 index 11f2db38e84..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageDataSource.scala +++ /dev/null @@ -1,73 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.geopackage - -import org.apache.hadoop.fs.Path -import org.apache.sedona.sql.datasources.geopackage.model.GeoPackageOptions -import org.apache.spark.sql.connector.catalog.Table -import org.apache.spark.sql.execution.datasources.FileFormat -import org.apache.spark.sql.execution.datasources.v2.FileDataSourceV2 -import org.apache.spark.sql.sources.DataSourceRegister -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -import java.util.Locale -import scala.jdk.CollectionConverters._ -import scala.util.Try - -class GeoPackageDataSource extends FileDataSourceV2 with DataSourceRegister { - - override def fallbackFileFormat: Class[_ <: FileFormat] = { - null - } - - override protected def getTable(options: CaseInsensitiveStringMap): Table = { - GeoPackageTable( - "", - sparkSession, - options, - getPaths(options), - None, - fallbackFileFormat, - getLoadOptions(options)) - } - - private def getLoadOptions(options: CaseInsensitiveStringMap): GeoPackageOptions = { - val path = options.get("path") - if (path.isEmpty) { - throw new IllegalArgumentException("GeoPackage path is not specified") - } - - val showMetadata = options.getBoolean("showMetadata", false) - val maybeTableName = options.get("tableName") - - if (!showMetadata && maybeTableName == null) { - throw new IllegalArgumentException("Table name is not specified") - } - - val tableName = if (showMetadata) { - "gpkg_contents" - } else { - maybeTableName - } - - GeoPackageOptions(tableName = tableName, showMetadata = showMetadata) - } - - override def shortName(): String = "geopackage" -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReader.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReader.scala deleted file mode 100644 index 4e59163922d..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReader.scala +++ /dev/null @@ -1,107 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.geopackage - -import org.apache.hadoop.fs.Path -import org.apache.sedona.sql.datasources.geopackage.connection.{FileSystemUtils, GeoPackageConnectionManager} -import org.apache.sedona.sql.datasources.geopackage.model.TableType.{FEATURES, METADATA, TILES, UNKNOWN} -import org.apache.sedona.sql.datasources.geopackage.model.{GeoPackageReadOptions, PartitionOptions, TileRowMetadata} -import org.apache.sedona.sql.datasources.geopackage.transform.ValuesMapper -import org.apache.spark.broadcast.Broadcast -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.connector.read.PartitionReader -import org.apache.spark.util.SerializableConfiguration - -import java.io.File -import java.sql.ResultSet - -case class GeoPackagePartitionReader( - var rs: ResultSet, - options: GeoPackageReadOptions, - broadcastedConf: Broadcast[SerializableConfiguration], - var currentTempFile: File, - copying: Boolean = false) - extends PartitionReader[InternalRow] { - - private var values: Seq[Any] = Seq.empty - private var currentFile = options.currentFile - private val partitionedFiles = options.partitionedFiles - - override def next(): Boolean = { - if (rs.next()) { - values = ValuesMapper.mapValues(adjustPartitionOptions, rs) - return true - } - - partitionedFiles.remove(currentFile) - - if (partitionedFiles.isEmpty) { - return false - } - - rs.close() - - currentFile = partitionedFiles.head - val (tempFile, _) = FileSystemUtils.copyToLocal( - options = broadcastedConf.value.value, - file = new Path(currentFile.filePath.toString())) - - if (copying) { - currentTempFile.deleteOnExit() - } - - currentTempFile = tempFile - - rs = GeoPackageConnectionManager.getTableCursor(currentTempFile.getPath, options.tableName) - - if (!rs.next()) { - return false - } - - values = ValuesMapper.mapValues(adjustPartitionOptions, rs) - - true - } - - private def adjustPartitionOptions: PartitionOptions = { - options.partitionOptions.tableType match { - case FEATURES | METADATA => options.partitionOptions - case TILES => - val tileRowMetadata = TileRowMetadata( - zoomLevel = rs.getInt("zoom_level"), - tileColumn = rs.getInt("tile_column"), - tileRow = rs.getInt("tile_row")) - - options.partitionOptions.withTileRowMetadata(tileRowMetadata) - case UNKNOWN => options.partitionOptions - } - - } - - override def get(): InternalRow = { - InternalRow.fromSeq(values) - } - - override def close(): Unit = { - rs.close() - if (copying) { - options.tempFile.delete() - } - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReaderFactory.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReaderFactory.scala deleted file mode 100644 index 0f2e9a87b81..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackagePartitionReaderFactory.scala +++ /dev/null @@ -1,139 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.geopackage - -import org.apache.hadoop.fs.Path -import org.apache.sedona.sql.datasources.geopackage.connection.{FileSystemUtils, GeoPackageConnectionManager} -import org.apache.sedona.sql.datasources.geopackage.model.TableType.TILES -import org.apache.sedona.sql.datasources.geopackage.model.{GeoPackageOptions, GeoPackageReadOptions, PartitionOptions, TableType} -import org.apache.spark.broadcast.Broadcast -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{BoundReference, JoinedRow} -import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection -import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader, PartitionReaderFactory} -import org.apache.spark.sql.execution.datasources.FilePartition -import org.apache.spark.sql.types.StructType -import org.apache.spark.unsafe.types.UTF8String -import org.apache.spark.util.SerializableConfiguration - -case class GeoPackagePartitionReaderFactory( - sparkSession: SparkSession, - broadcastedConf: Broadcast[SerializableConfiguration], - loadOptions: GeoPackageOptions, - dataSchema: StructType, - metadataSchema: StructType) - extends PartitionReaderFactory { - - override def createReader(partition: InputPartition): PartitionReader[InternalRow] = { - val partitionFiles = partition match { - case filePartition: FilePartition => filePartition.files - case _ => - throw new IllegalArgumentException( - s"Unexpected partition type: ${partition.getClass.getCanonicalName}") - } - - val (tempFile, copied) = FileSystemUtils.copyToLocal( - options = broadcastedConf.value.value, - file = new Path(partitionFiles.head.filePath.toString())) - - val tableType = if (loadOptions.showMetadata) { - TableType.METADATA - } else { - GeoPackageConnectionManager.findFeatureMetadata(tempFile.getPath, loadOptions.tableName) - } - - val rs = - GeoPackageConnectionManager.getTableCursor(tempFile.getAbsolutePath, loadOptions.tableName) - - val schema = GeoPackageConnectionManager.getSchema(tempFile.getPath, loadOptions.tableName) - - if (StructType(schema.map(_.toStructField(tableType))) != dataSchema) { - throw new IllegalArgumentException( - s"Schema mismatch: expected $dataSchema, got ${StructType(schema.map(_.toStructField(tableType)))}") - } - - val tileMetadata = tableType match { - case TILES => - Some( - GeoPackageConnectionManager.findTilesMetadata(tempFile.getPath, loadOptions.tableName)) - case _ => None - } - - val baseReader = GeoPackagePartitionReader( - rs = rs, - options = GeoPackageReadOptions( - tableName = loadOptions.tableName, - tempFile = tempFile, - partitionOptions = - PartitionOptions(tableType = tableType, columns = schema, tile = tileMetadata), - partitionedFiles = scala.collection.mutable.HashSet(partitionFiles: _*), - currentFile = partitionFiles.head), - broadcastedConf = broadcastedConf, - currentTempFile = tempFile, - copying = copied) - - if (metadataSchema.nonEmpty) { - val gpkgFile = partitionFiles.head - val filePath = gpkgFile.filePath.toString - val fileName = new Path(filePath).getName - - val allMetadataValues: Map[String, Any] = Map( - "file_path" -> UTF8String.fromString(filePath), - "file_name" -> UTF8String.fromString(fileName), - "file_size" -> gpkgFile.fileSize, - "file_block_start" -> gpkgFile.start, - "file_block_length" -> gpkgFile.length, - "file_modification_time" -> (gpkgFile.modificationTime * 1000L)) - - val innerStructType = metadataSchema.fields.head.dataType.asInstanceOf[StructType] - val prunedValues = innerStructType.fields.map(f => allMetadataValues(f.name)) - val metadataStruct = InternalRow.fromSeq(prunedValues.toSeq) - val metadataRow = InternalRow.fromSeq(Seq(metadataStruct)) - - new PartitionReaderWithMetadata(baseReader, dataSchema, metadataSchema, metadataRow) - } else { - baseReader - } - } -} - -private[geopackage] class PartitionReaderWithMetadata( - reader: PartitionReader[InternalRow], - baseSchema: StructType, - metadataSchema: StructType, - metadataValues: InternalRow) - extends PartitionReader[InternalRow] { - - private val joinedRow = new JoinedRow() - private val unsafeProjection = - GenerateUnsafeProjection.generate(baseSchema.fields.zipWithIndex.map { case (f, i) => - BoundReference(i, f.dataType, f.nullable) - } ++ metadataSchema.fields.zipWithIndex.map { case (f, i) => - BoundReference(baseSchema.length + i, f.dataType, f.nullable) - }) - - override def next(): Boolean = reader.next() - - override def get(): InternalRow = { - unsafeProjection(joinedRow(reader.get(), metadataValues)) - } - - override def close(): Unit = reader.close() -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScan.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScan.scala deleted file mode 100644 index edca3d35ff1..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScan.scala +++ /dev/null @@ -1,68 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.geopackage - -import org.apache.sedona.sql.datasources.geopackage.model.GeoPackageOptions -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.catalyst.expressions.Expression -import org.apache.spark.sql.connector.read.PartitionReaderFactory -import org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex -import org.apache.spark.sql.execution.datasources.v2.FileScan -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap -import org.apache.spark.util.SerializableConfiguration - -import scala.jdk.CollectionConverters._ - -case class GeoPackageScan( - dataSchema: StructType, - sparkSession: SparkSession, - fileIndex: PartitioningAwareFileIndex, - readDataSchema: StructType, - readPartitionSchema: StructType, - metadataSchema: StructType, - options: CaseInsensitiveStringMap, - loadOptions: GeoPackageOptions) - extends FileScan { - - override def readSchema(): StructType = - StructType(readDataSchema.fields ++ readPartitionSchema.fields ++ metadataSchema.fields) - - override def partitionFilters: Seq[Expression] = { - Seq.empty - } - - override def dataFilters: Seq[Expression] = { - Seq.empty - } - - override def createReaderFactory(): PartitionReaderFactory = { - val caseSensitiveMap = options.asScala.toMap - val hadoopConf = sparkSession.sessionState.newHadoopConfWithOptions(caseSensitiveMap) - val broadcastedConf = - sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf)) - - GeoPackagePartitionReaderFactory( - sparkSession, - broadcastedConf, - loadOptions, - dataSchema, - metadataSchema) - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScanBuilder.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScanBuilder.scala deleted file mode 100644 index d363406de19..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageScanBuilder.scala +++ /dev/null @@ -1,73 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.geopackage - -import org.apache.sedona.sql.datasources.geopackage.model.GeoPackageOptions -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.connector.read.Scan -import org.apache.spark.sql.execution.datasources.{InMemoryFileIndex, PartitioningAwareFileIndex} -import org.apache.spark.sql.execution.datasources.v2.FileScanBuilder -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap -import scala.jdk.CollectionConverters._ - -class GeoPackageScanBuilder( - sparkSession: SparkSession, - fileIndex: PartitioningAwareFileIndex, - dataSchema: StructType, - options: CaseInsensitiveStringMap, - loadOptions: GeoPackageOptions, - userDefinedSchema: Option[StructType] = None) - extends FileScanBuilder(sparkSession, fileIndex, dataSchema) { - - private var _requiredMetadataSchema: StructType = StructType(Seq.empty) - - override def pruneColumns(requiredSchema: StructType): Unit = { - val resolver = sparkSession.sessionState.conf.resolver - val metaFields = requiredSchema.fields.filter { field => - !dataSchema.fields.exists(df => resolver(df.name, field.name)) && - !fileIndex.partitionSchema.fields.exists(pf => resolver(pf.name, field.name)) - } - _requiredMetadataSchema = StructType(metaFields) - super.pruneColumns(requiredSchema) - } - - override def build(): Scan = { - val paths = fileIndex.allFiles().map(_.getPath.toString) - - val fileIndexAdjusted = - if (loadOptions.showMetadata) - new InMemoryFileIndex( - sparkSession, - paths.slice(0, 1).map(new org.apache.hadoop.fs.Path(_)), - options.asCaseSensitiveMap.asScala.toMap, - userDefinedSchema) - else fileIndex - - GeoPackageScan( - dataSchema, - sparkSession, - fileIndexAdjusted, - dataSchema, - readPartitionSchema(), - _requiredMetadataSchema, - options, - loadOptions) - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageTable.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageTable.scala deleted file mode 100644 index 078c9f23545..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/geopackage/GeoPackageTable.scala +++ /dev/null @@ -1,119 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.geopackage - -import org.apache.hadoop.fs.FileStatus -import org.apache.sedona.sql.datasources.geopackage.connection.{FileSystemUtils, GeoPackageConnectionManager} -import org.apache.sedona.sql.datasources.geopackage.model.{GeoPackageOptions, MetadataSchema, TableType} -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.connector.catalog.{MetadataColumn, SupportsMetadataColumns} -import org.apache.spark.sql.connector.read.ScanBuilder -import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder} -import org.apache.spark.sql.execution.datasources.{FileFormat, PartitioningAwareFileIndex, SedonaFileIndexHelper} -import org.apache.spark.sql.execution.datasources.v2.FileTable -import org.apache.spark.sql.types.{DataType, DoubleType, IntegerType, LongType, StringType, StructField, StructType, TimestampType} -import org.apache.spark.sql.util.CaseInsensitiveStringMap -import org.apache.spark.util.SerializableConfiguration - -import scala.jdk.CollectionConverters._ - -case class GeoPackageTable( - name: String, - sparkSession: SparkSession, - options: CaseInsensitiveStringMap, - paths: Seq[String], - userSpecifiedSchema: Option[StructType], - fallbackFileFormat: Class[_ <: FileFormat], - loadOptions: GeoPackageOptions) - extends FileTable(sparkSession, options, paths, userSpecifiedSchema) - with SupportsMetadataColumns { - - // Override fileIndex to skip the FileStreamSink.hasMetadata check that causes - // spurious FileNotFoundException warnings when reading from cloud storage (e.g., S3). - // GeoPackage tables are always non-streaming batch sources, so the streaming - // metadata check is unnecessary. - override lazy val fileIndex: PartitioningAwareFileIndex = - SedonaFileIndexHelper.createFileIndex(sparkSession, options, paths, userSpecifiedSchema) - - override def inferSchema(files: Seq[FileStatus]): Option[StructType] = { - if (loadOptions.showMetadata) { - return MetadataSchema.schema - } - - val serializableConf = new SerializableConfiguration( - sparkSession.sessionState.newHadoopConfWithOptions(options.asScala.toMap)) - - val (tempFile, copied) = - FileSystemUtils.copyToLocal(serializableConf.value, files.head.getPath) - - if (copied) { - tempFile.deleteOnExit() - } - - val tableType = if (loadOptions.showMetadata) { - TableType.METADATA - } else { - GeoPackageConnectionManager.findFeatureMetadata(tempFile.getPath, loadOptions.tableName) - } - - Some( - StructType( - GeoPackageConnectionManager - .getSchema(tempFile.getPath, loadOptions.tableName) - .map(field => field.toStructField(tableType)))) - } - - override def formatName: String = { - "GeoPackage" - } - - override def metadataColumns(): Array[MetadataColumn] = GeoPackageTable.fileMetadataColumns - - override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = { - new GeoPackageScanBuilder( - sparkSession, - fileIndex, - schema, - options, - loadOptions, - userSpecifiedSchema) - } - - override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = { - null - } -} - -object GeoPackageTable { - - private val FILE_METADATA_STRUCT_TYPE: StructType = StructType( - Seq( - StructField("file_path", StringType, nullable = false), - StructField("file_name", StringType, nullable = false), - StructField("file_size", LongType, nullable = false), - StructField("file_block_start", LongType, nullable = false), - StructField("file_block_length", LongType, nullable = false), - StructField("file_modification_time", TimestampType, nullable = false))) - - private[geopackage] val fileMetadataColumns: Array[MetadataColumn] = Array(new MetadataColumn { - override def name: String = "_metadata" - override def dataType: DataType = FILE_METADATA_STRUCT_TYPE - override def isNullable: Boolean = false - }) -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileDataSource.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileDataSource.scala deleted file mode 100644 index 7cd6d03a6d9..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileDataSource.scala +++ /dev/null @@ -1,101 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.hadoop.fs.Path -import org.apache.spark.sql.connector.catalog.Table -import org.apache.spark.sql.execution.datasources.v2.FileDataSourceV2 -import org.apache.spark.sql.execution.datasources.FileFormat -import org.apache.spark.sql.sources.DataSourceRegister -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -import java.util.Locale -import scala.collection.JavaConverters._ -import scala.util.Try - -/** - * A Spark SQL data source for reading ESRI Shapefiles. This data source supports reading the - * following components of shapefiles: - * - *
  • .shp: the main file
  • .dbf: (optional) the attribute file
  • .shx: (optional) the - * index file
  • .cpg: (optional) the code page file
  • .prj: (optional) the projection file - *
- * - *

The load path can be a directory containing the shapefiles, or a path to the .shp file. If - * the path refers to a .shp file, the data source will also read other components such as .dbf - * and .shx files in the same directory. - */ -class ShapefileDataSource extends FileDataSourceV2 with DataSourceRegister { - - override def shortName(): String = "shapefile" - - override def fallbackFileFormat: Class[_ <: FileFormat] = null - - override protected def getTable(options: CaseInsensitiveStringMap): Table = { - val paths = getTransformedPath(options) - val tableName = getTableName(options, paths) - val optionsWithoutPaths = getOptionsWithoutPaths(options) - ShapefileTable(tableName, sparkSession, optionsWithoutPaths, paths, None, fallbackFileFormat) - } - - override protected def getTable( - options: CaseInsensitiveStringMap, - schema: StructType): Table = { - val paths = getTransformedPath(options) - val tableName = getTableName(options, paths) - val optionsWithoutPaths = getOptionsWithoutPaths(options) - ShapefileTable( - tableName, - sparkSession, - optionsWithoutPaths, - paths, - Some(schema), - fallbackFileFormat) - } - - private def getTransformedPath(options: CaseInsensitiveStringMap): Seq[String] = { - val paths = getPaths(options) - transformPaths(paths, options) - } - - private def transformPaths( - paths: Seq[String], - options: CaseInsensitiveStringMap): Seq[String] = { - val caseSensitiveMap = options.asCaseSensitiveMap.asScala.toMap - val hadoopConf = sparkSession.sessionState.newHadoopConfWithOptions(caseSensitiveMap) - paths.map { pathString => - if (pathString.toLowerCase(Locale.ROOT).endsWith(".shp")) { - // If the path refers to a file, we need to change it to a glob path to support reading - // .dbf and .shx files as well. For example, if the path is /path/to/file.shp, we need to - // change it to /path/to/file.??? - val path = new Path(pathString) - val fs = path.getFileSystem(hadoopConf) - val isDirectory = Try(fs.getFileStatus(path).isDirectory).getOrElse(false) - if (isDirectory) { - pathString - } else { - pathString.substring(0, pathString.length - 3) + "???" - } - } else { - pathString - } - } - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartition.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartition.scala deleted file mode 100644 index 306b1df4f6c..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartition.scala +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.spark.Partition -import org.apache.spark.sql.connector.read.InputPartition -import org.apache.spark.sql.execution.datasources.PartitionedFile - -case class ShapefilePartition(index: Int, files: Array[PartitionedFile]) - extends Partition - with InputPartition diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReader.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReader.scala deleted file mode 100644 index 301d63296fb..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReader.scala +++ /dev/null @@ -1,287 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.commons.io.FilenameUtils -import org.apache.commons.io.IOUtils -import org.apache.hadoop.conf.Configuration -import org.apache.hadoop.fs.FSDataInputStream -import org.apache.hadoop.fs.Path -import org.apache.sedona.common.FunctionsGeoTools -import org.apache.sedona.core.formatMapper.shapefileParser.shapes.DbfFileReader -import org.apache.sedona.core.formatMapper.shapefileParser.shapes.PrimitiveShape -import org.apache.sedona.core.formatMapper.shapefileParser.shapes.ShapeFileReader -import org.apache.sedona.core.formatMapper.shapefileParser.shapes.ShxFileReader -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.BoundReference -import org.apache.spark.sql.catalyst.expressions.Cast -import org.apache.spark.sql.catalyst.expressions.Literal -import org.apache.spark.sql.catalyst.expressions.UnsafeProjection -import org.apache.spark.sql.connector.read.PartitionReader -import org.apache.spark.sql.execution.datasources.PartitionedFile -import org.apache.sedona.sql.datasources.shapefile.ShapefilePartitionReader.logger -import org.apache.sedona.sql.datasources.shapefile.ShapefilePartitionReader.openStream -import org.apache.sedona.sql.datasources.shapefile.ShapefilePartitionReader.tryOpenStream -import org.apache.sedona.sql.datasources.shapefile.ShapefileUtils.baseSchema -import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT -import org.apache.spark.sql.types.StructType -import org.locationtech.jts.geom.GeometryFactory -import org.locationtech.jts.geom.PrecisionModel -import org.slf4j.Logger -import org.slf4j.LoggerFactory - -import java.nio.charset.StandardCharsets -import scala.collection.JavaConverters._ -import java.util.Locale -import scala.util.Try - -class ShapefilePartitionReader( - configuration: Configuration, - partitionedFiles: Array[PartitionedFile], - readDataSchema: StructType, - options: ShapefileReadOptions) - extends PartitionReader[InternalRow] { - - private val partitionedFilesMap: Map[String, Path] = partitionedFiles.map { file => - val fileName = file.filePath.toPath.getName - val extension = FilenameUtils.getExtension(fileName).toLowerCase(Locale.ROOT) - extension -> file.filePath.toPath - }.toMap - - private val cpg = options.charset.orElse { - // No charset option or sedona.global.charset system property specified, infer charset - // from the cpg file. - tryOpenStream(partitionedFilesMap, "cpg", configuration) - .flatMap { stream => - try { - val lineIter = IOUtils.lineIterator(stream, StandardCharsets.UTF_8) - if (lineIter.hasNext) { - Some(lineIter.next().trim()) - } else { - None - } - } finally { - stream.close() - } - } - .orElse { - // Cannot infer charset from cpg file. If sedona.global.charset is set to "utf8", use UTF-8 as - // the default charset. This is for compatibility with the behavior of the RDD API. - val charset = System.getProperty("sedona.global.charset", "default") - val utf8flag = charset.equalsIgnoreCase("utf8") - if (utf8flag) Some("UTF-8") else None - } - } - - private val prj = tryOpenStream(partitionedFilesMap, "prj", configuration).map { stream => - try { - IOUtils.toString(stream, StandardCharsets.UTF_8) - } finally { - stream.close() - } - } - - private val shpReader: ShapeFileReader = { - val reader = tryOpenStream(partitionedFilesMap, "shx", configuration) match { - case Some(shxStream) => - try { - val index = ShxFileReader.readAll(shxStream) - new ShapeFileReader(index) - } finally { - shxStream.close() - } - case None => new ShapeFileReader() - } - val stream = openStream(partitionedFilesMap, "shp", configuration) - reader.initialize(stream) - reader - } - - private val dbfReader = - tryOpenStream(partitionedFilesMap, "dbf", configuration).map { stream => - val reader = new DbfFileReader() - reader.initialize(stream) - reader - } - - private val geometryField = readDataSchema.filter(_.dataType.isInstanceOf[GeometryUDT]) match { - case Seq(geoField) => Some(geoField) - case Seq() => None - case _ => throw new IllegalArgumentException("Only one geometry field is allowed") - } - - private val shpSchema: StructType = { - val dbfFields = dbfReader - .map { reader => - ShapefileUtils.fieldDescriptorsToStructFields(reader.getFieldDescriptors.asScala.toSeq) - } - .getOrElse(Seq.empty) - StructType(baseSchema(options).fields ++ dbfFields) - } - - // projection from shpSchema to readDataSchema - private val projection = { - val expressions = readDataSchema.map { field => - val index = Try(shpSchema.fieldIndex(field.name)).getOrElse(-1) - if (index >= 0) { - val sourceField = shpSchema.fields(index) - val refExpr = BoundReference(index, sourceField.dataType, sourceField.nullable) - if (sourceField.dataType == field.dataType) refExpr - else { - Cast(refExpr, field.dataType) - } - } else { - if (field.nullable) { - Literal(null) - } else { - // This usually won't happen, since all fields of readDataSchema are nullable for most - // of the time. See org.apache.spark.sql.execution.datasources.v2.FileTable#dataSchema - // for more details. - val dbfPath = partitionedFilesMap.get("dbf").orNull - throw new IllegalArgumentException( - s"Field ${field.name} not found in shapefile $dbfPath") - } - } - } - UnsafeProjection.create(expressions) - } - - // Convert DBF field values to SQL values - private val fieldValueConverters: Seq[Array[Byte] => Any] = dbfReader - .map { reader => - reader.getFieldDescriptors.asScala.map { field => - val index = Try(readDataSchema.fieldIndex(field.getFieldName)).getOrElse(-1) - if (index >= 0) { - ShapefileUtils.fieldValueConverter(field, cpg) - } else { (_: Array[Byte]) => - null - } - }.toSeq - } - .getOrElse(Seq.empty) - - private val geometryFactory = prj match { - case Some(wkt) => - val srid = - try { - FunctionsGeoTools.wktCRSToSRID(wkt) - } catch { - case e: Throwable => - val prjPath = partitionedFilesMap.get("prj").orNull - logger.warn(s"Failed to parse SRID from .prj file $prjPath", e) - 0 - } - new GeometryFactory(new PrecisionModel, srid) - case None => new GeometryFactory() - } - - private var currentRow: InternalRow = _ - - override def next(): Boolean = { - if (shpReader.nextKeyValue()) { - val key = shpReader.getCurrentKey - val id = key.getIndex - - val attributesOpt = dbfReader.flatMap { reader => - if (reader.nextKeyValue()) { - val value = reader.getCurrentFieldBytes - Option(value) - } else { - val dbfPath = partitionedFilesMap.get("dbf").orNull - logger.warn("Shape record loses attributes in .dbf file {} at ID={}", dbfPath, id) - None - } - } - - val value = shpReader.getCurrentValue - val geometry = geometryField.flatMap { _ => - if (value.getType.isSupported) { - val shape = new PrimitiveShape(value) - Some(shape.getShape(geometryFactory)) - } else { - logger.warn( - "Shape type {} is not supported, geometry value will be null", - value.getType.name()) - None - } - } - - val attrValues = attributesOpt match { - case Some(fieldBytesList) => - // Convert attributes to SQL values - fieldBytesList.asScala.zip(fieldValueConverters).map { case (fieldBytes, converter) => - converter(fieldBytes) - } - case None => - // No attributes, fill with nulls - Seq.fill(fieldValueConverters.length)(null) - } - - val serializedGeom = geometry.map(GeometryUDT.serialize).orNull - val shpRow = if (options.keyFieldName.isDefined) { - InternalRow.fromSeq(serializedGeom +: key.getIndex +: attrValues.toSeq) - } else { - InternalRow.fromSeq(serializedGeom +: attrValues.toSeq) - } - currentRow = projection(shpRow) - true - } else { - dbfReader.foreach { reader => - if (reader.nextKeyValue()) { - val dbfPath = partitionedFilesMap.get("dbf").orNull - logger.warn("Redundant attributes in {} exists", dbfPath) - } - } - false - } - } - - override def get(): InternalRow = currentRow - - override def close(): Unit = { - dbfReader.foreach(_.close()) - shpReader.close() - } -} - -object ShapefilePartitionReader { - val logger: Logger = LoggerFactory.getLogger(classOf[ShapefilePartitionReader]) - - private def openStream( - partitionedFilesMap: Map[String, Path], - extension: String, - configuration: Configuration): FSDataInputStream = { - tryOpenStream(partitionedFilesMap, extension, configuration).getOrElse { - val path = partitionedFilesMap.head._2 - val baseName = FilenameUtils.getBaseName(path.getName) - throw new IllegalArgumentException( - s"No $extension file found for shapefile $baseName in ${path.getParent}") - } - } - - private def tryOpenStream( - partitionedFilesMap: Map[String, Path], - extension: String, - configuration: Configuration): Option[FSDataInputStream] = { - partitionedFilesMap.get(extension).map { path => - val fs = path.getFileSystem(configuration) - fs.open(path) - } - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReaderFactory.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReaderFactory.scala deleted file mode 100644 index 79c0638bd6e..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefilePartitionReaderFactory.scala +++ /dev/null @@ -1,154 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.hadoop.fs.Path -import org.apache.spark.broadcast.Broadcast -import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{BoundReference, JoinedRow} -import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection -import org.apache.spark.sql.connector.read.InputPartition -import org.apache.spark.sql.connector.read.PartitionReader -import org.apache.spark.sql.connector.read.PartitionReaderFactory -import org.apache.spark.sql.execution.datasources.PartitionedFile -import org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues -import org.apache.spark.sql.internal.SQLConf -import org.apache.spark.sql.sources.Filter -import org.apache.spark.sql.types.StructType -import org.apache.spark.unsafe.types.UTF8String -import org.apache.spark.util.SerializableConfiguration - -import java.util.Locale - -case class ShapefilePartitionReaderFactory( - sqlConf: SQLConf, - broadcastedConf: Broadcast[SerializableConfiguration], - dataSchema: StructType, - readDataSchema: StructType, - partitionSchema: StructType, - /** The metadata fields requested by the query (e.g., fields from `_metadata`). */ - metadataSchema: StructType, - options: ShapefileReadOptions, - filters: Seq[Filter]) - extends PartitionReaderFactory { - - private def buildReader( - partitionedFiles: Array[PartitionedFile]): PartitionReader[InternalRow] = { - val fileReader = - new ShapefilePartitionReader( - broadcastedConf.value.value, - partitionedFiles, - readDataSchema, - options) - val withPartitionValues = new PartitionReaderWithPartitionValues( - fileReader, - readDataSchema, - partitionSchema, - partitionedFiles.head.partitionValues) - - if (metadataSchema.nonEmpty) { - // Build metadata values from the .shp file's partition information. - // We use the .shp file because it is the primary shapefile component and its path - // is what users would expect to see in _metadata.file_path / _metadata.file_name. - val shpFile = partitionedFiles - .find(_.filePath.toPath.getName.toLowerCase(Locale.ROOT).endsWith(".shp")) - .getOrElse(partitionedFiles.head) - val filePath = shpFile.filePath.toString - val fileName = new Path(filePath).getName - - // Complete map of all metadata field values keyed by field name. - // The modificationTime from PartitionedFile is in milliseconds but Spark's - // TimestampType uses microseconds, so we multiply by 1000. - val allMetadataValues: Map[String, Any] = Map( - "file_path" -> UTF8String.fromString(filePath), - "file_name" -> UTF8String.fromString(fileName), - "file_size" -> shpFile.fileSize, - "file_block_start" -> shpFile.start, - "file_block_length" -> shpFile.length, - "file_modification_time" -> (shpFile.modificationTime * 1000L)) - - // The metadataSchema may be pruned by Spark's column pruning (e.g., when the query - // only selects `_metadata.file_name`). We must construct the inner struct to match - // the pruned schema exactly, otherwise field ordinals will be misaligned. - val innerStructType = metadataSchema.fields.head.dataType.asInstanceOf[StructType] - val prunedValues = innerStructType.fields.map(f => allMetadataValues(f.name)) - val metadataStruct = InternalRow.fromSeq(prunedValues.toSeq) - - // Wrap the struct in an outer row since _metadata is a single StructType column - val metadataRow = InternalRow.fromSeq(Seq(metadataStruct)) - val baseSchema = StructType(readDataSchema.fields ++ partitionSchema.fields) - new PartitionReaderWithMetadata( - withPartitionValues, - baseSchema, - metadataSchema, - metadataRow) - } else { - withPartitionValues - } - } - - override def createReader(partition: InputPartition): PartitionReader[InternalRow] = { - partition match { - case filePartition: ShapefilePartition => buildReader(filePartition.files) - case _ => - throw new IllegalArgumentException( - s"Unexpected partition type: ${partition.getClass.getCanonicalName}") - } - } -} - -/** - * Wraps a partition reader to append metadata column values to each row. This follows the same - * pattern as [[PartitionReaderWithPartitionValues]] but for metadata columns: it uses a - * [[JoinedRow]] to concatenate the base row (data + partition values) with the metadata row, then - * projects the combined row through an - * [[org.apache.spark.sql.catalyst.expressions.UnsafeProjection]] to produce a compact unsafe row. - * - * @param reader - * the underlying reader that produces data + partition value rows - * @param baseSchema - * the combined schema of data columns and partition columns - * @param metadataSchema - * the schema of the metadata columns being appended - * @param metadataValues - * the constant metadata values to append to every row - */ -private[shapefile] class PartitionReaderWithMetadata( - reader: PartitionReader[InternalRow], - baseSchema: StructType, - metadataSchema: StructType, - metadataValues: InternalRow) - extends PartitionReader[InternalRow] { - - private val joinedRow = new JoinedRow() - private val unsafeProjection = - GenerateUnsafeProjection.generate(baseSchema.fields.zipWithIndex.map { case (f, i) => - BoundReference(i, f.dataType, f.nullable) - } ++ metadataSchema.fields.zipWithIndex.map { case (f, i) => - BoundReference(baseSchema.length + i, f.dataType, f.nullable) - }) - - override def next(): Boolean = reader.next() - - override def get(): InternalRow = { - unsafeProjection(joinedRow(reader.get(), metadataValues)) - } - - override def close(): Unit = reader.close() -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileReadOptions.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileReadOptions.scala deleted file mode 100644 index ebc02fae85a..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileReadOptions.scala +++ /dev/null @@ -1,45 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -/** - * Options for reading Shapefiles. - * @param geometryFieldName - * The name of the geometry field. - * @param keyFieldName - * The name of the shape key field. - * @param charset - * The charset of non-spatial attributes. - */ -case class ShapefileReadOptions( - geometryFieldName: String, - keyFieldName: Option[String], - charset: Option[String]) - -object ShapefileReadOptions { - def parse(options: CaseInsensitiveStringMap): ShapefileReadOptions = { - val geometryFieldName = options.getOrDefault("geometry.name", "geometry") - val keyFieldName = - if (options.containsKey("key.name")) Some(options.get("key.name")) else None - val charset = if (options.containsKey("charset")) Some(options.get("charset")) else None - ShapefileReadOptions(geometryFieldName, keyFieldName, charset) - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScan.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScan.scala deleted file mode 100644 index 3f6a9224aa2..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScan.scala +++ /dev/null @@ -1,129 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.hadoop.fs.Path -import org.apache.spark.sql.catalyst.expressions.Expression -import org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex -import org.apache.spark.sql.sources.Filter -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.connector.read.InputPartition -import org.apache.spark.sql.connector.read.PartitionReaderFactory -import org.apache.spark.sql.execution.datasources.v2.FileScan -import org.apache.spark.sql.execution.datasources.FilePartition -import org.apache.spark.sql.execution.datasources.PartitionedFile -import org.apache.sedona.sql.datasources.shapefile.ShapefileScan.logger -import org.apache.spark.util.SerializableConfiguration -import org.slf4j.{Logger, LoggerFactory} - -import java.util.Locale -import scala.collection.JavaConverters._ -import scala.collection.mutable - -case class ShapefileScan( - sparkSession: SparkSession, - fileIndex: PartitioningAwareFileIndex, - dataSchema: StructType, - readDataSchema: StructType, - readPartitionSchema: StructType, - /** The metadata fields requested by the query (e.g., fields from `_metadata`). */ - metadataSchema: StructType, - options: CaseInsensitiveStringMap, - pushedFilters: Array[Filter], - partitionFilters: Seq[Expression] = Seq.empty, - dataFilters: Seq[Expression] = Seq.empty) - extends FileScan { - - /** - * Returns the complete read schema including data columns, partition columns, and any requested - * metadata columns. Metadata columns are appended last so the reader factory can construct a - * [[JoinedRow]] that appends metadata values after data and partition values. - */ - override def readSchema(): StructType = - StructType(readDataSchema.fields ++ readPartitionSchema.fields ++ metadataSchema.fields) - - override def createReaderFactory(): PartitionReaderFactory = { - val caseSensitiveMap = options.asScala.toMap - val hadoopConf = sparkSession.sessionState.newHadoopConfWithOptions(caseSensitiveMap) - val broadcastedConf = - sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf)) - ShapefilePartitionReaderFactory( - sparkSession.sessionState.conf, - broadcastedConf, - dataSchema, - readDataSchema, - readPartitionSchema, - metadataSchema, - ShapefileReadOptions.parse(options), - pushedFilters) - } - - override def planInputPartitions(): Array[InputPartition] = { - // Simply use the default implementation to compute input partitions for all files - val allFilePartitions = super.planInputPartitions().flatMap { - case filePartition: FilePartition => - filePartition.files - case partition => - throw new IllegalArgumentException( - s"Unexpected partition type: ${partition.getClass.getCanonicalName}") - } - - // Group shapefiles by their main path (without the extension) - val shapefileGroups: mutable.Map[String, mutable.Map[String, PartitionedFile]] = - mutable.Map.empty - allFilePartitions.foreach { partitionedFile => - val path = partitionedFile.filePath.toPath - val fileName = path.getName - val pos = fileName.lastIndexOf('.') - if (pos == -1) None - else { - val mainName = fileName.substring(0, pos) - val extension = fileName.substring(pos + 1).toLowerCase(Locale.ROOT) - if (ShapefileUtils.shapeFileExtensions.contains(extension)) { - val key = new Path(path.getParent, mainName).toString - val group = shapefileGroups.getOrElseUpdate(key, mutable.Map.empty) - group += (extension -> partitionedFile) - } - } - } - - // Create a partition for each group - shapefileGroups.zipWithIndex.flatMap { case ((key, group), index) => - // Check if the group has all the necessary files - val suffixes = group.keys.toSet - val hasMissingFiles = ShapefileUtils.mandatoryFileExtensions.exists { suffix => - if (!suffixes.contains(suffix)) { - logger.warn(s"Shapefile $key is missing a $suffix file") - true - } else false - } - if (!hasMissingFiles) { - Some(ShapefilePartition(index, group.values.toArray)) - } else { - None - } - }.toArray - } -} - -object ShapefileScan { - val logger: Logger = LoggerFactory.getLogger(classOf[ShapefileScan]) -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScanBuilder.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScanBuilder.scala deleted file mode 100644 index 48b5e45d535..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileScanBuilder.scala +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.spark.sql.connector.read.Scan -import org.apache.spark.sql.execution.datasources.v2.FileScanBuilder -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -case class ShapefileScanBuilder( - sparkSession: SparkSession, - fileIndex: PartitioningAwareFileIndex, - schema: StructType, - dataSchema: StructType, - options: CaseInsensitiveStringMap) - extends FileScanBuilder(sparkSession, fileIndex, dataSchema) { - - /** - * Tracks any metadata fields (e.g., from `_metadata`) requested in the query. Populated by - * [[pruneColumns]] when Spark pushes down column projections. - */ - private var _requiredMetadataSchema: StructType = StructType(Seq.empty) - - /** - * Intercepts Spark's column pruning to separate metadata columns from data/partition columns. - * Fields in [[requiredSchema]] that do not belong to the data schema or partition schema are - * assumed to be metadata fields (e.g., `_metadata`). These are captured in - * [[_requiredMetadataSchema]] so the scan can include them in the output. - */ - override def pruneColumns(requiredSchema: StructType): Unit = { - val resolver = sparkSession.sessionState.conf.resolver - val metaFields = requiredSchema.fields.filter { field => - !dataSchema.fields.exists(df => resolver(df.name, field.name)) && - !fileIndex.partitionSchema.fields.exists(pf => resolver(pf.name, field.name)) - } - _requiredMetadataSchema = StructType(metaFields) - super.pruneColumns(requiredSchema) - } - - override def build(): Scan = { - ShapefileScan( - sparkSession, - fileIndex, - dataSchema, - readDataSchema(), - readPartitionSchema(), - _requiredMetadataSchema, - options, - pushedDataFilters, - partitionFilters, - dataFilters) - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileTable.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileTable.scala deleted file mode 100644 index 1903623a6ea..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileTable.scala +++ /dev/null @@ -1,159 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.hadoop.fs.FileStatus -import org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.dbf.DbfParseUtil -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.connector.catalog.{MetadataColumn, SupportsMetadataColumns, TableCapability} -import org.apache.spark.sql.connector.read.ScanBuilder -import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder} -import org.apache.spark.sql.execution.datasources.{FileFormat, PartitioningAwareFileIndex, SedonaFileIndexHelper} -import org.apache.sedona.sql.datasources.shapefile.ShapefileUtils.{baseSchema, fieldDescriptorsToSchema, mergeSchemas} -import org.apache.spark.sql.execution.datasources.v2.FileTable -import org.apache.spark.sql.types.{DataType, LongType, StringType, StructField, StructType, TimestampType} -import org.apache.spark.sql.util.CaseInsensitiveStringMap -import org.apache.spark.util.SerializableConfiguration - -import java.util.Locale -import scala.collection.JavaConverters._ - -/** - * A Spark DataSource V2 table implementation for reading Shapefiles. - * - * Extends [[FileTable]] to leverage Spark's file-based scan infrastructure and implements - * [[SupportsMetadataColumns]] to expose hidden metadata columns (e.g., `_metadata`) that provide - * file-level information such as path, name, size, and modification time. These metadata columns - * are not part of the user-visible schema but can be explicitly selected in queries. - */ -case class ShapefileTable( - name: String, - sparkSession: SparkSession, - options: CaseInsensitiveStringMap, - paths: Seq[String], - userSpecifiedSchema: Option[StructType], - fallbackFileFormat: Class[_ <: FileFormat]) - extends FileTable(sparkSession, options, paths, userSpecifiedSchema) - with SupportsMetadataColumns { - - // Override fileIndex to skip the FileStreamSink.hasMetadata check that causes - // spurious FileNotFoundException warnings when reading from cloud storage (e.g., S3). - // Shapefile tables are always non-streaming batch sources, so the streaming - // metadata check is unnecessary. - override lazy val fileIndex: PartitioningAwareFileIndex = - SedonaFileIndexHelper.createFileIndex(sparkSession, options, paths, userSpecifiedSchema) - - override def formatName: String = "Shapefile" - - override def capabilities: java.util.Set[TableCapability] = - java.util.EnumSet.of(TableCapability.BATCH_READ) - - override def inferSchema(files: Seq[FileStatus]): Option[StructType] = { - if (files.isEmpty) None - else { - def isDbfFile(file: FileStatus): Boolean = { - val name = file.getPath.getName.toLowerCase(Locale.ROOT) - name.endsWith(".dbf") - } - - def isShpFile(file: FileStatus): Boolean = { - val name = file.getPath.getName.toLowerCase(Locale.ROOT) - name.endsWith(".shp") - } - - if (!files.exists(isShpFile)) None - else { - val readOptions = ShapefileReadOptions.parse(options) - val resolver = sparkSession.sessionState.conf.resolver - val dbfFiles = files.filter(isDbfFile) - if (dbfFiles.isEmpty) { - Some(baseSchema(readOptions, Some(resolver))) - } else { - val serializableConf = new SerializableConfiguration( - sparkSession.sessionState.newHadoopConfWithOptions(options.asScala.toMap)) - val partiallyMergedSchemas = sparkSession.sparkContext - .parallelize(dbfFiles) - .mapPartitions { iter => - val schemas = iter.map { stat => - val fs = stat.getPath.getFileSystem(serializableConf.value) - val stream = fs.open(stat.getPath) - try { - val dbfParser = new DbfParseUtil() - dbfParser.parseFileHead(stream) - val fieldDescriptors = dbfParser.getFieldDescriptors - fieldDescriptorsToSchema(fieldDescriptors.asScala.toSeq, readOptions, resolver) - } finally { - stream.close() - } - }.toSeq - mergeSchemas(schemas).iterator - } - .collect() - mergeSchemas(partiallyMergedSchemas) - } - } - } - } - - /** Returns the metadata columns that this table exposes as hidden columns. */ - override def metadataColumns(): Array[MetadataColumn] = ShapefileTable.fileMetadataColumns - - override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = { - ShapefileScanBuilder(sparkSession, fileIndex, schema, dataSchema, options) - } - - override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = null -} - -object ShapefileTable { - - /** - * Schema of the `_metadata` struct column exposed by [[SupportsMetadataColumns]]. Each field - * provides file-level information about the source shapefile: - * - * - `file_path`: The fully qualified path of the `.shp` file (e.g., - * `hdfs://host/data/file.shp`). - * - `file_name`: The name of the `.shp` file without directory components (e.g., `file.shp`). - * - `file_size`: The total size of the `.shp` file in bytes. - * - `file_block_start`: The byte offset within the file where this partition's data begins. - * For non-splittable formats this is typically 0. - * - `file_block_length`: The number of bytes in this partition's data block. For - * non-splittable formats this equals the file size. - * - `file_modification_time`: The last modification timestamp of the `.shp` file. - */ - private val FILE_METADATA_STRUCT_TYPE: StructType = StructType( - Seq( - StructField("file_path", StringType, nullable = false), - StructField("file_name", StringType, nullable = false), - StructField("file_size", LongType, nullable = false), - StructField("file_block_start", LongType, nullable = false), - StructField("file_block_length", LongType, nullable = false), - StructField("file_modification_time", TimestampType, nullable = false))) - - /** - * The single metadata column `_metadata` exposed to Spark's catalog. This hidden column can be - * selected in queries (e.g., `SELECT _metadata.file_name FROM shapefile.`...``) but does not - * appear in `SELECT *`. - */ - private[shapefile] val fileMetadataColumns: Array[MetadataColumn] = Array(new MetadataColumn { - override def name: String = "_metadata" - override def dataType: DataType = FILE_METADATA_STRUCT_TYPE - override def isNullable: Boolean = false - }) -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileUtils.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileUtils.scala deleted file mode 100644 index efd6098d518..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/datasources/shapefile/ShapefileUtils.scala +++ /dev/null @@ -1,202 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.datasources.shapefile - -import org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.dbf.FieldDescriptor -import org.apache.spark.sql.catalyst.analysis.Resolver -import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT -import org.apache.spark.sql.types.BooleanType -import org.apache.spark.sql.types.DateType -import org.apache.spark.sql.types.Decimal -import org.apache.spark.sql.types.DecimalType -import org.apache.spark.sql.types.LongType -import org.apache.spark.sql.types.StringType -import org.apache.spark.sql.types.StructField -import org.apache.spark.sql.types.StructType -import org.apache.spark.unsafe.types.UTF8String - -import java.nio.charset.StandardCharsets -import java.time.LocalDate -import java.time.format.DateTimeFormatter -import java.util.Locale - -object ShapefileUtils { - - /** - * shp: main file for storing shapes shx: index file for the main file dbf: attribute file cpg: - * code page file prj: projection file - */ - val shapeFileExtensions: Set[String] = Set("shp", "shx", "dbf", "cpg", "prj") - - /** - * The mandatory file extensions for a shapefile. We don't require the dbf file and shx file for - * being consistent with the behavior of the RDD API ShapefileReader.readToGeometryRDD - */ - val mandatoryFileExtensions: Set[String] = Set("shp") - - def mergeSchemas(schemas: Seq[StructType]): Option[StructType] = { - if (schemas.isEmpty) { - None - } else { - var mergedSchema = schemas.head - schemas.tail.foreach { schema => - try { - mergedSchema = mergeSchema(mergedSchema, schema) - } catch { - case cause: IllegalArgumentException => - throw new IllegalArgumentException( - s"Failed to merge schema $mergedSchema with $schema", - cause) - } - } - Some(mergedSchema) - } - } - - private def mergeSchema(schema1: StructType, schema2: StructType): StructType = { - // The field names are case insensitive when performing schema merging - val fieldMap = schema1.fields.map(f => f.name.toLowerCase(Locale.ROOT) -> f).toMap - var newFields = schema1.fields - schema2.fields.foreach { f => - fieldMap.get(f.name.toLowerCase(Locale.ROOT)) match { - case Some(existingField) => - if (existingField.dataType != f.dataType) { - throw new IllegalArgumentException( - s"Failed to merge fields ${existingField.name} and ${f.name} because they have different data types: ${existingField.dataType} and ${f.dataType}") - } - case _ => - newFields :+= f - } - } - StructType(newFields) - } - - def fieldDescriptorsToStructFields(fieldDescriptors: Seq[FieldDescriptor]): Seq[StructField] = { - fieldDescriptors.map { desc => - val name = desc.getFieldName - val dataType = desc.getFieldType match { - case 'C' => StringType - case 'N' | 'F' => - val scale = desc.getFieldDecimalCount - if (scale == 0) LongType - else { - val precision = desc.getFieldLength - DecimalType(precision, scale) - } - case 'L' => BooleanType - case 'D' => DateType - case _ => - throw new IllegalArgumentException(s"Unsupported field type ${desc.getFieldType}") - } - StructField(name, dataType, nullable = true) - } - } - - def fieldDescriptorsToSchema(fieldDescriptors: Seq[FieldDescriptor]): StructType = { - val structFields = fieldDescriptorsToStructFields(fieldDescriptors) - StructType(structFields) - } - - def fieldDescriptorsToSchema( - fieldDescriptors: Seq[FieldDescriptor], - options: ShapefileReadOptions, - resolver: Resolver): StructType = { - val structFields = fieldDescriptorsToStructFields(fieldDescriptors) - val geometryFieldName = options.geometryFieldName - if (structFields.exists(f => resolver(f.name, geometryFieldName))) { - throw new IllegalArgumentException( - s"Field name $geometryFieldName is reserved for geometry but appears in non-spatial attributes. " + - "Please specify a different field name for geometry using the 'geometry.name' option.") - } - options.keyFieldName.foreach { name => - if (structFields.exists(f => resolver(f.name, name))) { - throw new IllegalArgumentException( - s"Field name $name is reserved for shape key but appears in non-spatial attributes. " + - "Please specify a different field name for shape key using the 'key.name' option.") - } - } - StructType(baseSchema(options, Some(resolver)).fields ++ structFields) - } - - def baseSchema(options: ShapefileReadOptions, resolver: Option[Resolver] = None): StructType = { - options.keyFieldName match { - case Some(name) => - if (resolver.exists(_(name, options.geometryFieldName))) { - throw new IllegalArgumentException(s"geometry.name and key.name cannot be the same") - } - StructType( - Seq(StructField(options.geometryFieldName, GeometryUDT()), StructField(name, LongType))) - case _ => - StructType(StructField(options.geometryFieldName, GeometryUDT()) :: Nil) - } - } - - def fieldValueConverter(desc: FieldDescriptor, cpg: Option[String]): Array[Byte] => Any = { - desc.getFieldType match { - case 'C' => - val encoding = cpg.getOrElse("ISO-8859-1") - if (encoding.toLowerCase(Locale.ROOT) == "utf-8") { (bytes: Array[Byte]) => - UTF8String.fromBytes(bytes).trimRight() - } else { (bytes: Array[Byte]) => - { - val str = new String(bytes, encoding) - UTF8String.fromString(str).trimRight() - } - } - case 'N' | 'F' => - val scale = desc.getFieldDecimalCount - if (scale == 0) { (bytes: Array[Byte]) => - try { - new String(bytes, StandardCharsets.ISO_8859_1).trim.toLong - } catch { - case _: Exception => null - } - } else { (bytes: Array[Byte]) => - try { - Decimal.fromString(UTF8String.fromBytes(bytes)) - } catch { - case _: Exception => null - } - } - case 'L' => - (bytes: Array[Byte]) => - if (bytes.isEmpty) null - else { - bytes.head match { - case 'T' | 't' | 'Y' | 'y' => true - case 'F' | 'f' | 'N' | 'n' => false - case _ => null - } - } - case 'D' => - (bytes: Array[Byte]) => { - try { - val dateString = new String(bytes, StandardCharsets.ISO_8859_1) - val formatter = DateTimeFormatter.BASIC_ISO_DATE - val date = LocalDate.parse(dateString, formatter) - date.toEpochDay.toInt - } catch { - case _: Exception => null - } - } - case _ => - throw new IllegalArgumentException(s"Unsupported field type ${desc.getFieldType}") - } - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlAstBuilder.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlAstBuilder.scala deleted file mode 100644 index b6a87dfa4f7..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlAstBuilder.scala +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.parser - -import org.apache.spark.sql.catalyst.parser.SqlBaseParser._ -import org.apache.spark.sql.execution.SparkSqlAstBuilder -import org.apache.spark.sql.sedona_sql.UDT.{Box2DUDT, Box3DUDT, GeometryUDT} -import org.apache.spark.sql.types.DataType - -class SedonaSqlAstBuilder extends SparkSqlAstBuilder { - - /** - * Recognize Sedona UDT names (GEOMETRY, BOX2D, BOX3D) as primitive data types so SQL `CAST(... - * AS geometry)` / `CAST(... AS box2d)` / `CAST(... AS box3d)` parse to the matching UDT. - */ - override def visitPrimitiveDataType(ctx: PrimitiveDataTypeContext): DataType = { - ctx.getText.toUpperCase() match { - case "GEOMETRY" => GeometryUDT() - case "BOX2D" => Box2DUDT - case "BOX3D" => Box3DUDT - case _ => super.visitPrimitiveDataType(ctx) - } - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlParser.scala b/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlParser.scala deleted file mode 100644 index 54fb074eb05..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/sedona/sql/parser/SedonaSqlParser.scala +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql.parser - -import org.apache.spark.sql.catalyst.parser.ParserInterface -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan -import org.apache.spark.sql.execution.SparkSqlParser - -class SedonaSqlParser(delegate: ParserInterface) extends SparkSqlParser { - - // The parser builder for the Sedona SQL AST - val parserBuilder = new SedonaSqlAstBuilder - - /** - * Parse the SQL text and return the logical plan. This method first attempts to use the - * delegate parser to parse the SQL text. If the delegate parser fails (throws an exception), it - * falls back to using the Sedona SQL parser. - * - * @param sqlText - * The SQL text to be parsed. - * @return - * The parsed logical plan. - */ - override def parsePlan(sqlText: String): LogicalPlan = - try { - delegate.parsePlan(sqlText) - } catch { - case _: Exception => - parse(sqlText) { parser => - parserBuilder.visit(parser.singleStatement()) - }.asInstanceOf[LogicalPlan] - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataDataSource.scala b/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataDataSource.scala deleted file mode 100644 index 43e1ababb7d..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataDataSource.scala +++ /dev/null @@ -1,65 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.spark.sql.execution.datasources.v2.geoparquet.metadata - -import org.apache.spark.sql.connector.catalog.Table -import org.apache.spark.sql.execution.datasources.FileFormat -import org.apache.spark.sql.execution.datasources.v2.FileDataSourceV2 -import org.apache.spark.sql.sources.DataSourceRegister -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -/** - * Data source for reading GeoParquet metadata. This could be accessed using the `spark.read` - * interface: - * {{{ - * val df = spark.read.format("geoparquet.metadata").load("path/to/geoparquet") - * }}} - */ -class GeoParquetMetadataDataSource extends FileDataSourceV2 with DataSourceRegister { - override val shortName: String = "geoparquet.metadata" - - override def fallbackFileFormat: Class[_ <: FileFormat] = null - - override def getTable(options: CaseInsensitiveStringMap): Table = { - val paths = getPaths(options) - val tableName = getTableName(options, paths) - val optionsWithoutPaths = getOptionsWithoutPaths(options) - GeoParquetMetadataTable( - tableName, - sparkSession, - optionsWithoutPaths, - paths, - None, - fallbackFileFormat) - } - - override def getTable(options: CaseInsensitiveStringMap, schema: StructType): Table = { - val paths = getPaths(options) - val tableName = getTableName(options, paths) - val optionsWithoutPaths = getOptionsWithoutPaths(options) - GeoParquetMetadataTable( - tableName, - sparkSession, - optionsWithoutPaths, - paths, - Some(schema), - fallbackFileFormat) - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataPartitionReaderFactory.scala b/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataPartitionReaderFactory.scala deleted file mode 100644 index d60ebcb07a2..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataPartitionReaderFactory.scala +++ /dev/null @@ -1,121 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.spark.sql.execution.datasources.v2.geoparquet.metadata - -import org.apache.hadoop.conf.Configuration -import org.apache.parquet.ParquetReadOptions -import org.apache.parquet.hadoop.ParquetFileReader -import org.apache.parquet.hadoop.util.HadoopInputFile -import org.apache.spark.broadcast.Broadcast -import org.apache.spark.sql.catalyst.{FileSourceOptions, InternalRow} -import org.apache.spark.sql.catalyst.expressions.GenericInternalRow -import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData} -import org.apache.spark.sql.connector.read.PartitionReader -import org.apache.spark.sql.execution.datasources.PartitionedFile -import org.apache.spark.sql.execution.datasources.geoparquet.GeoParquetMetaData -import org.apache.spark.sql.execution.datasources.v2._ -import org.apache.spark.sql.internal.SQLConf -import org.apache.spark.sql.sources.Filter -import org.apache.spark.sql.types.StructType -import org.apache.spark.unsafe.types.UTF8String -import org.apache.spark.util.SerializableConfiguration -import org.json4s.DefaultFormats -import org.json4s.jackson.JsonMethods.{compact, render} - -case class GeoParquetMetadataPartitionReaderFactory( - sqlConf: SQLConf, - broadcastedConf: Broadcast[SerializableConfiguration], - dataSchema: StructType, - readDataSchema: StructType, - partitionSchema: StructType, - options: FileSourceOptions, - filters: Seq[Filter]) - extends FilePartitionReaderFactory { - - override def buildReader(partitionedFile: PartitionedFile): PartitionReader[InternalRow] = { - val iter = GeoParquetMetadataPartitionReaderFactory.readFile( - broadcastedConf.value.value, - partitionedFile, - readDataSchema) - val fileReader = new PartitionReaderFromIterator[InternalRow](iter) - new PartitionReaderWithPartitionValues( - fileReader, - readDataSchema, - partitionSchema, - partitionedFile.partitionValues) - } -} - -object GeoParquetMetadataPartitionReaderFactory { - private def readFile( - configuration: Configuration, - partitionedFile: PartitionedFile, - readDataSchema: StructType): Iterator[InternalRow] = { - val inputFile = HadoopInputFile.fromPath(partitionedFile.toPath, configuration) - val inputStream = inputFile.newStream() - - val footer = ParquetFileReader - .readFooter(inputFile, ParquetReadOptions.builder().build(), inputStream) - - val filePath = partitionedFile.toPath.toString - val metadata = footer.getFileMetaData.getKeyValueMetaData - val row = GeoParquetMetaData.parseKeyValueMetaData(metadata) match { - case Some(geo) => - val geoColumnsMap = geo.columns.map { case (columnName, columnMetadata) => - implicit val formats: org.json4s.Formats = DefaultFormats - import org.json4s.jackson.Serialization - val columnMetadataFields: Array[Any] = Array( - UTF8String.fromString(columnMetadata.encoding), - new GenericArrayData(columnMetadata.geometryTypes.map(UTF8String.fromString).toArray), - columnMetadata.bbox.map(b => new GenericArrayData(b.toArray)).orNull, - columnMetadata.crs - .map(projjson => UTF8String.fromString(compact(render(projjson)))) - .getOrElse(UTF8String.fromString("")), - columnMetadata.covering - .map(covering => UTF8String.fromString(Serialization.write(covering))) - .orNull) - val columnMetadataStruct = new GenericInternalRow(columnMetadataFields) - UTF8String.fromString(columnName) -> columnMetadataStruct - } - val fields: Array[Any] = Array( - UTF8String.fromString(filePath), - UTF8String.fromString(geo.version.orNull), - UTF8String.fromString(geo.primaryColumn), - ArrayBasedMapData(geoColumnsMap)) - new GenericInternalRow(fields) - case None => - // Not a GeoParquet file, return a row with null metadata values. - val fields: Array[Any] = Array(UTF8String.fromString(filePath), null, null, null) - new GenericInternalRow(fields) - } - Iterator(pruneBySchema(row, GeoParquetMetadataTable.schema, readDataSchema)) - } - - private def pruneBySchema( - row: InternalRow, - schema: StructType, - readDataSchema: StructType): InternalRow = { - // Projection push down for nested fields is not enabled, so this very simple implementation is enough. - val values: Array[Any] = readDataSchema.fields.map { field => - val index = schema.fieldIndex(field.name) - row.get(index, field.dataType) - } - new GenericInternalRow(values) - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScan.scala b/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScan.scala deleted file mode 100644 index b6a2cc49f64..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScan.scala +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.spark.sql.execution.datasources.v2.geoparquet.metadata - -import org.apache.hadoop.fs.Path -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.catalyst.FileSourceOptions -import org.apache.spark.sql.catalyst.expressions.Expression -import org.apache.spark.sql.connector.read.PartitionReaderFactory -import org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex -import org.apache.spark.sql.execution.datasources.v2.FileScan -import org.apache.spark.sql.sources.Filter -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap -import org.apache.spark.util.SerializableConfiguration - -import scala.collection.JavaConverters._ - -case class GeoParquetMetadataScan( - sparkSession: SparkSession, - fileIndex: PartitioningAwareFileIndex, - dataSchema: StructType, - readDataSchema: StructType, - readPartitionSchema: StructType, - options: CaseInsensitiveStringMap, - pushedFilters: Array[Filter], - partitionFilters: Seq[Expression] = Seq.empty, - dataFilters: Seq[Expression] = Seq.empty) - extends FileScan { - override def createReaderFactory(): PartitionReaderFactory = { - val caseSensitiveMap = options.asCaseSensitiveMap.asScala.toMap - // Hadoop Configurations are case sensitive. - val hadoopConf = sparkSession.sessionState.newHadoopConfWithOptions(caseSensitiveMap) - val broadcastedConf = - sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf)) - // The partition values are already truncated in `FileScan.partitions`. - // We should use `readPartitionSchema` as the partition schema here. - val fileSourceOptions = new FileSourceOptions(caseSensitiveMap) - GeoParquetMetadataPartitionReaderFactory( - sparkSession.sessionState.conf, - broadcastedConf, - dataSchema, - readDataSchema, - readPartitionSchema, - fileSourceOptions, - pushedFilters) - } - - override def getFileUnSplittableReason(path: Path): String = - "Reading parquet file metadata does not require splitting the file" -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScanBuilder.scala b/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScanBuilder.scala deleted file mode 100644 index c60369e1087..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataScanBuilder.scala +++ /dev/null @@ -1,47 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.spark.sql.execution.datasources.v2.geoparquet.metadata - -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.connector.read.Scan -import org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex -import org.apache.spark.sql.execution.datasources.v2.FileScanBuilder -import org.apache.spark.sql.types.StructType -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -class GeoParquetMetadataScanBuilder( - sparkSession: SparkSession, - fileIndex: PartitioningAwareFileIndex, - schema: StructType, - dataSchema: StructType, - options: CaseInsensitiveStringMap) - extends FileScanBuilder(sparkSession, fileIndex, dataSchema) { - override def build(): Scan = { - GeoParquetMetadataScan( - sparkSession, - fileIndex, - dataSchema, - readDataSchema(), - readPartitionSchema(), - options, - pushedDataFilters, - partitionFilters, - dataFilters) - } -} diff --git a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataTable.scala b/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataTable.scala deleted file mode 100644 index abb18a9dddf..00000000000 --- a/spark/spark-3.4/src/main/scala/org/apache/spark/sql/execution/datasources/v2/geoparquet/metadata/GeoParquetMetadataTable.scala +++ /dev/null @@ -1,78 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.spark.sql.execution.datasources.v2.geoparquet.metadata - -import org.apache.hadoop.fs.FileStatus -import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.connector.catalog.TableCapability -import org.apache.spark.sql.connector.read.ScanBuilder -import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder} -import org.apache.spark.sql.execution.datasources.{FileFormat, PartitioningAwareFileIndex, SedonaFileIndexHelper} -import org.apache.spark.sql.execution.datasources.v2.FileTable -import org.apache.spark.sql.types._ -import org.apache.spark.sql.util.CaseInsensitiveStringMap - -case class GeoParquetMetadataTable( - name: String, - sparkSession: SparkSession, - options: CaseInsensitiveStringMap, - paths: Seq[String], - userSpecifiedSchema: Option[StructType], - fallbackFileFormat: Class[_ <: FileFormat]) - extends FileTable(sparkSession, options, paths, userSpecifiedSchema) { - - // Override fileIndex to skip the FileStreamSink.hasMetadata check that causes - // spurious FileNotFoundException warnings when reading from cloud storage (e.g., S3). - // GeoParquet metadata tables are always non-streaming batch sources, so the streaming - // metadata check is unnecessary. - override lazy val fileIndex: PartitioningAwareFileIndex = - SedonaFileIndexHelper.createFileIndex(sparkSession, options, paths, userSpecifiedSchema) - - override def formatName: String = "GeoParquet Metadata" - - override def inferSchema(files: Seq[FileStatus]): Option[StructType] = - Some(GeoParquetMetadataTable.schema) - - override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = - new GeoParquetMetadataScanBuilder(sparkSession, fileIndex, schema, dataSchema, options) - - override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = null - - override def capabilities: java.util.Set[TableCapability] = - java.util.EnumSet.of(TableCapability.BATCH_READ) -} - -object GeoParquetMetadataTable { - private val columnMetadataType = StructType( - Seq( - StructField("encoding", StringType, nullable = true), - StructField("geometry_types", ArrayType(StringType), nullable = true), - StructField("bbox", ArrayType(DoubleType), nullable = true), - StructField("crs", StringType, nullable = true), - StructField("covering", StringType, nullable = true))) - - private val columnsType = MapType(StringType, columnMetadataType, valueContainsNull = false) - - val schema: StructType = StructType( - Seq( - StructField("path", StringType, nullable = false), - StructField("version", StringType, nullable = true), - StructField("primary_column", StringType, nullable = true), - StructField("columns", columnsType, nullable = true))) -} diff --git a/spark/spark-3.4/src/test/resources/log4j2.properties b/spark/spark-3.4/src/test/resources/log4j2.properties deleted file mode 100644 index 1bd1963e7be..00000000000 --- a/spark/spark-3.4/src/test/resources/log4j2.properties +++ /dev/null @@ -1,31 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# Set everything to be logged to the file target/unit-tests.log -rootLogger.level = warn -rootLogger.appenderRef.file.ref = File - -appender.file.type = File -appender.file.name = File -appender.file.fileName = target/unit-tests.log -appender.file.append = true -appender.file.layout.type = PatternLayout -appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex - -# Ignore messages below warning level from Jetty, because it's a bit verbose -logger.jetty.name = org.sparkproject.jetty -logger.jetty.level = warn diff --git a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box2DCastSuite.scala b/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box2DCastSuite.scala deleted file mode 100644 index d9a62dabbe5..00000000000 --- a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box2DCastSuite.scala +++ /dev/null @@ -1,144 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql - -import org.apache.sedona.common.geometryObjects.Box2D -import org.apache.spark.sql.functions.{col, expr} -import org.apache.spark.sql.sedona_sql.UDT.{Box2DUDT, GeometryUDT} - -class Box2DCastSuite extends TestBaseScala { - - /** - * SQL `CAST(... AS box2d)` / `CAST(... AS geometry)` parsing requires Sedona's - * `SedonaSqlAstBuilder` to be active. The test base randomizes - * `spark.sedona.enableParserExtension` across CI runs, and `SparkContext` is JVM-singleton so - * the active value can differ from this suite's session-level config. Probe directly by parsing - * a tiny CAST: this matches the behavior the SQL tests actually depend on, and caches the - * answer for the rest of the suite. DataFrame `.cast(...)` tests run unconditionally because - * the resolution rule is always injected. - */ - private lazy val sqlCastSupported: Boolean = { - try { - sparkSession - .sql("SELECT CAST(ST_GeomFromText('POINT (0 0)') AS box2d) AS b") - .collect() - true - } catch { - case _: org.apache.spark.sql.catalyst.parser.ParseException => false - } - } - - describe("Geometry ↔ Box2D Catalyst cast") { - - it("DataFrame .cast(Box2DUDT) rewrites to ST_Box2D") { - import sparkSession.implicits._ - val df = Seq("LINESTRING (0 0, 10 20)").toDF("wkt") - val box = df - .select(expr("ST_GeomFromText(wkt)").alias("g")) - .select(col("g").cast(Box2DUDT).alias("b")) - .collect() - .head - .getAs[Box2D]("b") - assert(box == new Box2D(0.0, 0.0, 10.0, 20.0)) - } - - it("DataFrame .cast(GeometryUDT) rewrites to ST_GeomFromBox2D") { - val df = - sparkSession.sql("SELECT ST_MakeBox2D(ST_Point(0.0, 0.0), ST_Point(2.0, 4.0)) AS b") - val wkt = df - .select(col("b").cast(GeometryUDT()).alias("g")) - .selectExpr("ST_AsText(g) AS wkt") - .collect() - .head - .getString(0) - assert(wkt == "POLYGON ((0 0, 0 4, 2 4, 2 0, 0 0))") - } - - it("DataFrame round-trip Geometry → Box2D → Geometry yields the envelope polygon") { - import sparkSession.implicits._ - val df = Seq("LINESTRING (0 0, 5 10)").toDF("wkt") - val wkt = df - .select(expr("ST_GeomFromText(wkt)").alias("g")) - .select(col("g").cast(Box2DUDT).cast(GeometryUDT()).alias("env")) - .selectExpr("ST_AsText(env) AS wkt") - .collect() - .head - .getString(0) - assert(wkt == "POLYGON ((0 0, 0 10, 5 10, 5 0, 0 0))") - } - - it("DataFrame .cast(Box2DUDT) on NULL geometry returns null") { - val box = sparkSession - .sql("SELECT ST_GeomFromText(NULL) AS g") - .select(col("g").cast(Box2DUDT).alias("b")) - .collect() - .head - .getAs[Box2D]("b") - assert(box == null) - } - - it("SQL CAST(geom AS box2d) returns the planar bbox") { - assume( - sqlCastSupported, - "Sedona SQL parser extension is required for `CAST(... AS box2d)` syntax") - val box = sparkSession - .sql("SELECT CAST(ST_GeomFromText('LINESTRING (0 0, 10 20)') AS box2d) AS b") - .collect() - .head - .getAs[Box2D]("b") - assert(box == new Box2D(0.0, 0.0, 10.0, 20.0)) - } - - it("SQL CAST(box AS geometry) returns the rectangular polygon") { - assume( - sqlCastSupported, - "Sedona SQL parser extension is required for `CAST(... AS geometry)` syntax") - val wkt = sparkSession - .sql("SELECT ST_AsText(CAST(ST_MakeBox2D(ST_Point(0.0, 0.0), ST_Point(2.0, 4.0)) AS geometry)) AS w") - .collect() - .head - .getString(0) - assert(wkt == "POLYGON ((0 0, 0 4, 2 4, 2 0, 0 0))") - } - - it("SQL round-trip Geometry → Box2D → Geometry yields the envelope polygon") { - assume( - sqlCastSupported, - "Sedona SQL parser extension is required for `CAST(... AS ...)` between UDTs") - val wkt = sparkSession - .sql("SELECT ST_AsText(CAST(CAST(ST_GeomFromText('LINESTRING (0 0, 5 10)') AS box2d) AS geometry)) AS w") - .collect() - .head - .getString(0) - assert(wkt == "POLYGON ((0 0, 0 10, 5 10, 5 0, 0 0))") - } - - it("SQL CAST(NULL geometry AS box2d) returns null") { - assume( - sqlCastSupported, - "Sedona SQL parser extension is required for `CAST(... AS box2d)` syntax") - val box = sparkSession - .sql("SELECT CAST(ST_GeomFromText(NULL) AS box2d) AS b") - .collect() - .head - .getAs[Box2D]("b") - assert(box == null) - } - } -} diff --git a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box3DCastSuite.scala b/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box3DCastSuite.scala deleted file mode 100644 index cc6cc598081..00000000000 --- a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/Box3DCastSuite.scala +++ /dev/null @@ -1,118 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql - -import org.apache.sedona.common.geometryObjects.Box3D -import org.apache.spark.sql.functions.{col, expr} -import org.apache.spark.sql.sedona_sql.UDT.Box3DUDT - -class Box3DCastSuite extends TestBaseScala { - - /** - * SQL `CAST(... AS box3d)` parsing requires Sedona's `SedonaSqlAstBuilder` to be active. The - * test base randomizes `spark.sedona.enableParserExtension` across CI runs, and `SparkContext` - * is JVM-singleton so the active value can differ from this suite's session-level config. Probe - * directly by parsing a tiny CAST: this matches the behavior the SQL tests actually depend on, - * and caches the answer for the rest of the suite. DataFrame `.cast(...)` tests run - * unconditionally because the resolution rule is always injected. - */ - private lazy val sqlCastSupported: Boolean = { - try { - sparkSession - .sql("SELECT CAST(ST_GeomFromText('POINT (0 0)') AS box3d) AS b") - .collect() - true - } catch { - case _: org.apache.spark.sql.catalyst.parser.ParseException => false - } - } - - describe("Geometry → Box3D Catalyst cast") { - - it("DataFrame .cast(Box3DUDT) rewrites to ST_Box3D") { - import sparkSession.implicits._ - val df = Seq("LINESTRING Z(0 0 -3, 5 10 7)").toDF("wkt") - val box = df - .select(expr("ST_GeomFromText(wkt)").alias("g")) - .select(col("g").cast(Box3DUDT).alias("b")) - .collect() - .head - .getAs[Box3D]("b") - assert(box == new Box3D(0.0, 0.0, -3.0, 5.0, 10.0, 7.0)) - } - - it("DataFrame .cast(Box3DUDT) on XY geometry folds Z = 0") { - import sparkSession.implicits._ - val df = Seq("LINESTRING (0 0, 5 10)").toDF("wkt") - val box = df - .select(expr("ST_GeomFromText(wkt)").alias("g")) - .select(col("g").cast(Box3DUDT).alias("b")) - .collect() - .head - .getAs[Box3D]("b") - assert(box == new Box3D(0.0, 0.0, 0.0, 5.0, 10.0, 0.0)) - } - - it("DataFrame .cast(Box3DUDT) on NULL geometry returns null") { - val box = sparkSession - .sql("SELECT ST_GeomFromText(NULL) AS g") - .select(col("g").cast(Box3DUDT).alias("b")) - .collect() - .head - .getAs[Box3D]("b") - assert(box == null) - } - - it("SQL CAST(geom AS box3d) returns the 3D bbox") { - assume( - sqlCastSupported, - "Sedona SQL parser extension is required for `CAST(... AS box3d)` syntax") - val box = sparkSession - .sql("SELECT CAST(ST_GeomFromText('LINESTRING Z(0 0 -3, 5 10 7)') AS box3d) AS b") - .collect() - .head - .getAs[Box3D]("b") - assert(box == new Box3D(0.0, 0.0, -3.0, 5.0, 10.0, 7.0)) - } - - it("SQL CAST(geom AS box3d) on XY geometry folds Z = 0") { - assume( - sqlCastSupported, - "Sedona SQL parser extension is required for `CAST(... AS box3d)` syntax") - val box = sparkSession - .sql("SELECT CAST(ST_GeomFromText('LINESTRING (0 0, 5 10)') AS box3d) AS b") - .collect() - .head - .getAs[Box3D]("b") - assert(box == new Box3D(0.0, 0.0, 0.0, 5.0, 10.0, 0.0)) - } - - it("SQL CAST(NULL geometry AS box3d) returns null") { - assume( - sqlCastSupported, - "Sedona SQL parser extension is required for `CAST(... AS box3d)` syntax") - val box = sparkSession - .sql("SELECT CAST(ST_GeomFromText(NULL) AS box3d) AS b") - .collect() - .head - .getAs[Box3D]("b") - assert(box == null) - } - } -} diff --git a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoPackageReaderTest.scala b/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoPackageReaderTest.scala deleted file mode 100644 index 0443553a863..00000000000 --- a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoPackageReaderTest.scala +++ /dev/null @@ -1,441 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql - -import io.minio.{MakeBucketArgs, MinioClient} -import org.apache.spark.sql.{DataFrame, Row} -import org.apache.spark.sql.functions.expr -import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT -import org.apache.spark.sql.types._ -import org.scalatest.matchers.should.Matchers -import org.scalatest.prop.TableDrivenPropertyChecks._ -import org.testcontainers.containers.MinIOContainer - -import java.io.FileInputStream -import java.sql.{Date, Timestamp} -import java.util.TimeZone - -class GeoPackageReaderTest extends TestBaseScala with Matchers { - TimeZone.setDefault(TimeZone.getTimeZone("UTC")) - import sparkSession.implicits._ - - val path: String = resourceFolder + "geopackage/example.gpkg" - val polygonsPath: String = resourceFolder + "geopackage/features.gpkg" - val rasterPath: String = resourceFolder + "geopackage/raster.gpkg" - val wktReader = new org.locationtech.jts.io.WKTReader() - val wktWriter = new org.locationtech.jts.io.WKTWriter() - - val expectedFeatureSchema = StructType( - Seq( - StructField("id", IntegerType, true), - StructField("geometry", GeometryUDT(), true), - StructField("text", StringType, true), - StructField("real", DoubleType, true), - StructField("boolean", BooleanType, true), - StructField("blob", BinaryType, true), - StructField("integer", IntegerType, true), - StructField("text_limited", StringType, true), - StructField("blob_limited", BinaryType, true), - StructField("date", DateType, true), - StructField("datetime", TimestampType, true))) - - describe("Reading GeoPackage metadata") { - it("should read GeoPackage metadata") { - val df = sparkSession.read - .format("geopackage") - .option("showMetadata", "true") - .load(path) - - df.where("data_type = 'tiles'").show(false) - - df.count shouldEqual 34 - } - } - - describe("Reading Vector data") { - it("should read GeoPackage - point1") { - val df = readFeatureData("point1") - df.schema shouldEqual expectedFeatureSchema - - df.count() shouldEqual 4 - - val firstElement = df.collectAsList().get(0).toSeq - - val expectedValues = Seq( - 1, - wktReader.read(POINT_1), - "BIT Systems", - 4519.866024037493, - true, - Array(48, 99, 57, 54, 49, 56, 55, 54, 45, 98, 102, 100, 52, 45, 52, 102, 52, 48, 45, 97, - 49, 102, 101, 45, 55, 49, 55, 101, 57, 100, 50, 98, 48, 55, 98, 101), - 3, - "bcd5a36f-16dc-4385-87be-b40353848597", - Array(49, 50, 53, 50, 97, 99, 98, 52, 45, 57, 54, 54, 52, 45, 52, 101, 51, 50, 45, 57, 54, - 100, 101, 45, 56, 48, 54, 101, 101, 48, 101, 101, 49, 102, 57, 48), - Date.valueOf("2023-09-19"), - Timestamp.valueOf("2023-09-19 11:24:15.695")) - - firstElement should contain theSameElementsAs expectedValues - } - - it("should read GeoPackage - line1") { - val df = readFeatureData("line1") - .withColumn("datetime", expr("from_utc_timestamp(datetime, 'UTC')")) - - df.schema shouldEqual expectedFeatureSchema - - df.count() shouldEqual 3 - - val firstElement = df.collectAsList().get(0).toSeq - - firstElement should contain theSameElementsAs Seq( - 1, - wktReader.read(LINESTRING_1), - "East Lockheed Drive", - 1990.5159635296877, - false, - Array(54, 97, 98, 100, 98, 51, 97, 56, 45, 54, 53, 101, 48, 45, 52, 55, 48, 54, 45, 56, - 50, 52, 48, 45, 51, 57, 48, 55, 99, 50, 102, 102, 57, 48, 99, 55), - 1, - "13dd91dc-3b7d-4d8d-a0ca-b3afb8e31c3d", - Array(57, 54, 98, 102, 56, 99, 101, 56, 45, 102, 48, 54, 49, 45, 52, 55, 99, 48, 45, 97, - 98, 48, 101, 45, 97, 99, 50, 52, 100, 98, 50, 97, 102, 50, 50, 54), - Date.valueOf("2023-09-19"), - Timestamp.valueOf("2023-09-19 11:24:15.716")) - } - - it("should read GeoPackage - polygon1") { - val df = readFeatureData("polygon1") - df.count shouldEqual 3 - df.schema shouldEqual expectedFeatureSchema - - df.select("geometry").collectAsList().get(0).toSeq should contain theSameElementsAs Seq( - wktReader.read(POLYGON_1)) - } - - it("should read GeoPackage - geometry1") { - val df = readFeatureData("geometry1") - df.count shouldEqual 10 - df.schema shouldEqual expectedFeatureSchema - - df.selectExpr("ST_ASTEXT(geometry)") - .as[String] - .collect() should contain theSameElementsAs Seq( - POINT_1, - POINT_2, - POINT_3, - POINT_4, - LINESTRING_1, - LINESTRING_2, - LINESTRING_3, - POLYGON_1, - POLYGON_2, - POLYGON_3) - } - - it("should read polygon with envelope data") { - val tables = Table( - ("tableName", "expectedCount"), - ("GB_Hex_5km_GS_CompressibleGround_v8", 4233), - ("GB_Hex_5km_GS_Landslides_v8", 4228), - ("GB_Hex_5km_GS_RunningSand_v8", 4233), - ("GB_Hex_5km_GS_ShrinkSwell_v8", 4233), - ("GB_Hex_5km_GS_SolubleRocks_v8", 4295)) - - forAll(tables) { (tableName: String, expectedCount: Int) => - val df = sparkSession.read - .format("geopackage") - .option("tableName", tableName) - .load(polygonsPath) - - df.count() shouldEqual expectedCount - } - } - - it("should handle datetime fields without timezone information") { - // This test verifies the fix for DateTimeParseException when reading - // GeoPackage files with datetime fields that don't include timezone info - val testFilePath = resourceFolder + "geopackage/test_datetime_issue.gpkg" - - // Test reading the test_features table with problematic datetime formats - val df = sparkSession.read - .format("geopackage") - .option("tableName", "test_features") - .load(testFilePath) - - // The test should not throw DateTimeParseException when reading datetime fields - noException should be thrownBy { - df.select("created_at", "updated_at").collect() - } - - // Verify that datetime fields are properly parsed as TimestampType - df.schema.fields.find(_.name == "created_at").get.dataType shouldEqual TimestampType - df.schema.fields.find(_.name == "updated_at").get.dataType shouldEqual TimestampType - - // Verify that we can read the datetime values - val datetimeValues = df.select("created_at", "updated_at").collect() - datetimeValues should not be empty - - // Verify that datetime values are valid timestamps - datetimeValues.foreach { row => - val createdTimestamp = row.getAs[Timestamp]("created_at") - val updatedTimestamp = row.getAs[Timestamp]("updated_at") - createdTimestamp should not be null - updatedTimestamp should not be null - createdTimestamp.getTime should be > 0L - updatedTimestamp.getTime should be > 0L - } - - // Test showMetadata option with the same file - noException should be thrownBy { - val metadataDf = sparkSession.read - .format("geopackage") - .option("showMetadata", "true") - .load(testFilePath) - metadataDf.select("last_change").collect() - } - } - } - - describe("GeoPackage Raster Data Test") { - it("should read") { - val fractions = - Table( - ("tableName", "channelNumber", "expectedSum"), - ("point1_tiles", 4, 466591.0), - ("line1_tiles", 4, 5775976.0), - ("polygon1_tiles", 4, 1.1269871e7), - ("geometry1_tiles", 4, 2.6328442e7), - ("point2_tiles", 4, 137456.0), - ("line2_tiles", 4, 6701101.0), - ("polygon2_tiles", 4, 5.1170714e7), - ("geometry2_tiles", 4, 1.6699823e7), - ("bit_systems", 1, 6.5561879e7), - ("nga", 1, 6.8078856e7), - ("bit_systems_wgs84", 1, 7.7276934e7), - ("nga_pc", 1, 2.90590616e8), - ("bit_systems_world", 1, 7.7276934e7), - ("nga_pc_world", 1, 2.90590616e8)) - - forAll(fractions) { (tableName: String, channelNumber: Int, expectedSum: Double) => - { - val df = readFeatureData(tableName) - val calculatedSum = df - .selectExpr(s"RS_SummaryStats(tile_data, 'sum', ${channelNumber}) as stats") - .selectExpr("sum(stats)") - .as[Double] - - calculatedSum.collect().head shouldEqual expectedSum - } - } - } - - it("should be able to read complex raster data") { - val df = sparkSession.read - .format("geopackage") - .option("tableName", "AuroraAirportNoise") - .load(rasterPath) - - df.show(5) - - val calculatedSum = df - .selectExpr(s"RS_SummaryStats(tile_data, 'sum', ${1}) as stats") - .selectExpr("sum(stats)") - .as[Double] - - calculatedSum.first() shouldEqual 2.027126e7 - - val df2 = sparkSession.read - .format("geopackage") - .option("tableName", "LiquorLicenseDensity") - .load(rasterPath) - - val calculatedSum2 = df2 - .selectExpr(s"RS_SummaryStats(tile_data, 'sum', ${1}) as stats") - .selectExpr("sum(stats)") - .as[Double] - - calculatedSum2.first() shouldEqual 2.882028e7 - } - - } - - describe("Reading from S3") { - it("should be able to read files from S3") { - val container = new MinIOContainer("minio/minio:latest") - - container.start() - - val minioClient = createMinioClient(container) - val makeBucketRequest = MakeBucketArgs - .builder() - .bucket("sedona") - .build() - - minioClient.makeBucket(makeBucketRequest) - - adjustSparkSession(sparkSessionMinio, container) - - val inputPath: String = prepareFile("example.geopackage", path, minioClient) - - sparkSessionMinio.read - .format("geopackage") - .option("showMetadata", "true") - .load(inputPath) - .count shouldEqual 34 - - val df = sparkSession.read - .format("geopackage") - .option("tableName", "point1") - .load(inputPath) - - df.count shouldEqual 4 - - val inputPathLarger: String = prepareFiles((1 to 300).map(_ => path).toArray, minioClient) - - val dfLarger = sparkSessionMinio.read - .format("geopackage") - .option("tableName", "point1") - .load(inputPathLarger) - - dfLarger.count shouldEqual 300 * 4 - - container.stop() - } - } - - describe("_metadata hidden column support") { - it("should expose _metadata struct with all expected fields") { - val df = readFeatureData("point1") - val metaDf = df.select("_metadata") - val metaSchema = metaDf.schema.fields.head.dataType.asInstanceOf[StructType] - val fieldNames = metaSchema.fieldNames.toSet - fieldNames should contain("file_path") - fieldNames should contain("file_name") - fieldNames should contain("file_size") - fieldNames should contain("file_block_start") - fieldNames should contain("file_block_length") - fieldNames should contain("file_modification_time") - } - - it("should not include _metadata in select(*)") { - val df = readFeatureData("point1") - val starCols = df.select("*").columns.toSet - starCols should not contain "_metadata" - } - - it("should return correct file_path and file_name in _metadata") { - val df = readFeatureData("point1") - val row = df.select("_metadata.file_path", "_metadata.file_name").head() - val filePath = row.getString(0) - val fileName = row.getString(1) - filePath should endWith("example.gpkg") - fileName shouldEqual "example.gpkg" - } - - it("should return actual file_size matching the .gpkg file on disk") { - val df = readFeatureData("point1") - val metaFileSize = df.select("_metadata.file_size").head().getLong(0) - val actualFile = new java.io.File(path) - metaFileSize shouldEqual actualFile.length() - } - - it("should return file_block_start=0 and file_block_length=file_size") { - val df = readFeatureData("point1") - val row = df - .select( - "_metadata.file_block_start", - "_metadata.file_block_length", - "_metadata.file_size") - .head() - row.getLong(0) shouldEqual 0L - row.getLong(1) shouldEqual row.getLong(2) - } - - it("should return file_modification_time matching the .gpkg file on disk") { - val df = readFeatureData("point1") - val metaModTime = df.select("_metadata.file_modification_time").head().getTimestamp(0) - val actualFile = new java.io.File(path) - val expectedModTime = new java.sql.Timestamp(actualFile.lastModified()) - metaModTime shouldEqual expectedModTime - } - - it("should allow filtering on _metadata fields") { - val df = readFeatureData("point1") - val filtered = df.filter(df("_metadata.file_name") === "example.gpkg") - filtered.count() shouldEqual df.count() - val empty = df.filter(df("_metadata.file_name") === "nonexistent.gpkg") - empty.count() shouldEqual 0 - } - - it("should select _metadata along with data columns") { - val df = readFeatureData("point1") - val result = df.select("id", "_metadata.file_name").head() - result.getInt(0) shouldEqual 1 - result.getString(1) shouldEqual "example.gpkg" - } - } - - private def readFeatureData(tableName: String): DataFrame = { - sparkSession.read - .format("geopackage") - .option("tableName", tableName) - .load(path) - } - - private def prepareFiles(paths: Array[String], minioClient: MinioClient): String = { - val key = "geopackage" - - paths.foreach(path => { - val fis = new FileInputStream(path); - putFileIntoBucket( - "sedona", - s"${key}/${scala.util.Random.nextInt(1000000000)}.geopackage", - fis, - minioClient) - }) - - s"s3a://sedona/$key" - } - - private def prepareFile(name: String, path: String, minioClient: MinioClient): String = { - val fis = new FileInputStream(path); - putFileIntoBucket("sedona", name, fis, minioClient) - - s"s3a://sedona/$name" - } - - private val POINT_1 = "POINT (-104.801918 39.720014)" - private val POINT_2 = "POINT (-104.802987 39.717703)" - private val POINT_3 = "POINT (-104.807496 39.714085)" - private val POINT_4 = "POINT (-104.79948 39.714729)" - private val LINESTRING_1 = - "LINESTRING (-104.800614 39.720721, -104.802174 39.720726, -104.802584 39.72066, -104.803088 39.720477, -104.803474 39.720209)" - private val LINESTRING_2 = - "LINESTRING (-104.809612 39.718379, -104.806638 39.718372, -104.806236 39.718439, -104.805939 39.718536, -104.805654 39.718677, -104.803652 39.720095)" - private val LINESTRING_3 = - "LINESTRING (-104.806344 39.722425, -104.805854 39.722634, -104.805656 39.722647, -104.803749 39.722641, -104.803769 39.721849, -104.803806 39.721725, -104.804382 39.720865)" - private val POLYGON_1 = - "POLYGON ((-104.802246 39.720343, -104.802246 39.719753, -104.802183 39.719754, -104.802184 39.719719, -104.802138 39.719694, -104.802097 39.719691, -104.802096 39.719648, -104.801646 39.719648, -104.801644 39.719722, -104.80155 39.719723, -104.801549 39.720207, -104.801648 39.720207, -104.801648 39.720341, -104.802246 39.720343))" - private val POLYGON_2 = - "POLYGON ((-104.802259 39.719604, -104.80226 39.71955, -104.802281 39.719416, -104.802332 39.719372, -104.802081 39.71924, -104.802044 39.71929, -104.802027 39.719278, -104.802044 39.719229, -104.801785 39.719129, -104.801639 39.719413, -104.801649 39.719472, -104.801694 39.719524, -104.801753 39.71955, -104.80175 39.719606, -104.80194 39.719606, -104.801939 39.719555, -104.801977 39.719556, -104.801979 39.719606, -104.802259 39.719604), (-104.80213 39.71944, -104.802133 39.71949, -104.802148 39.71949, -104.80218 39.719473, -104.802187 39.719456, -104.802182 39.719439, -104.802088 39.719387, -104.802047 39.719427, -104.801858 39.719342, -104.801883 39.719294, -104.801832 39.719284, -104.801787 39.719298, -104.801763 39.719331, -104.801823 39.719352, -104.80179 39.71942, -104.801722 39.719404, -104.801715 39.719445, -104.801748 39.719484, -104.801809 39.719494, -104.801816 39.719439, -104.80213 39.71944))" - private val POLYGON_3 = - "POLYGON ((-104.802867 39.718122, -104.802369 39.717845, -104.802571 39.71763, -104.803066 39.717909, -104.802867 39.718122))" -} diff --git a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoParquetMetadataTests.scala b/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoParquetMetadataTests.scala deleted file mode 100644 index 01306c1b452..00000000000 --- a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/GeoParquetMetadataTests.scala +++ /dev/null @@ -1,152 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql - -import org.apache.spark.sql.Row -import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT -import org.apache.spark.sql.types.{IntegerType, StructField, StructType} -import org.scalatest.BeforeAndAfterAll - -import java.util.Collections -import scala.collection.JavaConverters._ - -class GeoParquetMetadataTests extends TestBaseScala with BeforeAndAfterAll { - val geoparquetdatalocation: String = resourceFolder + "geoparquet/" - val geoparquetoutputlocation: String = resourceFolder + "geoparquet/geoparquet_output/" - - describe("GeoParquet Metadata tests") { - it("Reading GeoParquet Metadata") { - val df = sparkSession.read.format("geoparquet.metadata").load(geoparquetdatalocation) - val metadataArray = df.collect() - assert(metadataArray.length > 1) - assert(metadataArray.exists(_.getAs[String]("path").endsWith(".parquet"))) - assert(metadataArray.exists(_.getAs[String]("version") == "1.0.0-dev")) - assert(metadataArray.exists(_.getAs[String]("primary_column") == "geometry")) - assert(metadataArray.exists { row => - val columnsMap = row.getJavaMap(row.fieldIndex("columns")) - columnsMap != null && columnsMap - .containsKey("geometry") && columnsMap.get("geometry").isInstanceOf[Row] - }) - assert(metadataArray.forall { row => - val columnsMap = row.getJavaMap(row.fieldIndex("columns")) - if (columnsMap == null || !columnsMap.containsKey("geometry")) true - else { - val columnMetadata = columnsMap.get("geometry").asInstanceOf[Row] - columnMetadata.getAs[String]("encoding") == "WKB" && - columnMetadata - .getList[Any](columnMetadata.fieldIndex("bbox")) - .asScala - .forall(_.isInstanceOf[Double]) && - columnMetadata - .getList[Any](columnMetadata.fieldIndex("geometry_types")) - .asScala - .forall(_.isInstanceOf[String]) && - columnMetadata.getAs[String]("crs").nonEmpty && - columnMetadata.getAs[String]("crs") != "null" - } - }) - } - - it("Reading GeoParquet Metadata with column pruning") { - val df = sparkSession.read.format("geoparquet.metadata").load(geoparquetdatalocation) - val metadataArray = df - .selectExpr("path", "substring(primary_column, 1, 2) AS partial_primary_column") - .collect() - assert(metadataArray.length > 1) - assert(metadataArray.forall(_.length == 2)) - assert(metadataArray.exists(_.getAs[String]("path").endsWith(".parquet"))) - assert(metadataArray.exists(_.getAs[String]("partial_primary_column") == "ge")) - } - - it("Reading GeoParquet Metadata of plain parquet files") { - val df = sparkSession.read.format("geoparquet.metadata").load(geoparquetdatalocation) - val metadataArray = df.where("path LIKE '%plain.parquet'").collect() - assert(metadataArray.nonEmpty) - assert(metadataArray.forall(_.getAs[String]("path").endsWith("plain.parquet"))) - assert(metadataArray.forall(_.getAs[String]("version") == null)) - assert(metadataArray.forall(_.getAs[String]("primary_column") == null)) - assert(metadataArray.forall(_.getAs[String]("columns") == null)) - } - - it("Read GeoParquet without CRS") { - val df = sparkSession.read - .format("geoparquet") - .load(geoparquetdatalocation + "/example-1.0.0-beta.1.parquet") - val geoParquetSavePath = geoparquetoutputlocation + "/gp_crs_omit.parquet" - df.write - .format("geoparquet") - .option("geoparquet.crs", "") - .mode("overwrite") - .save(geoParquetSavePath) - val dfMeta = sparkSession.read.format("geoparquet.metadata").load(geoParquetSavePath) - val row = dfMeta.collect()(0) - val metadata = row.getJavaMap(row.fieldIndex("columns")).get("geometry").asInstanceOf[Row] - assert(metadata.getAs[String]("crs") == "") - } - - it("Read GeoParquet with null CRS") { - val df = sparkSession.read - .format("geoparquet") - .load(geoparquetdatalocation + "/example-1.0.0-beta.1.parquet") - val geoParquetSavePath = geoparquetoutputlocation + "/gp_crs_null.parquet" - df.write - .format("geoparquet") - .option("geoparquet.crs", "null") - .mode("overwrite") - .save(geoParquetSavePath) - val dfMeta = sparkSession.read.format("geoparquet.metadata").load(geoParquetSavePath) - val row = dfMeta.collect()(0) - val metadata = row.getJavaMap(row.fieldIndex("columns")).get("geometry").asInstanceOf[Row] - assert(metadata.getAs[String]("crs") == "null") - } - - it("Read GeoParquet with snake_case geometry column name and camelCase column name") { - val schema = StructType( - Seq( - StructField("id", IntegerType, nullable = false), - StructField("geom_column_1", GeometryUDT(), nullable = false), - StructField("geomColumn2", GeometryUDT(), nullable = false))) - val df = sparkSession.createDataFrame(Collections.emptyList[Row](), schema) - val geoParquetSavePath = geoparquetoutputlocation + "/gp_column_name_styles.parquet" - df.write.format("geoparquet").mode("overwrite").save(geoParquetSavePath) - - val dfMeta = sparkSession.read.format("geoparquet.metadata").load(geoParquetSavePath) - val row = dfMeta.collect()(0) - val metadata = row.getJavaMap(row.fieldIndex("columns")) - assert(metadata.containsKey("geom_column_1")) - assert(!metadata.containsKey("geoColumn1")) - assert(metadata.containsKey("geomColumn2")) - assert(!metadata.containsKey("geom_column2")) - assert(!metadata.containsKey("geom_column_2")) - } - - it("Read GeoParquet with covering metadata") { - val dfMeta = sparkSession.read - .format("geoparquet.metadata") - .load(geoparquetdatalocation + "/example-1.1.0.parquet") - val row = dfMeta.collect()(0) - val metadata = row.getJavaMap(row.fieldIndex("columns")).get("geometry").asInstanceOf[Row] - val covering = metadata.getAs[String]("covering") - assert(covering.nonEmpty) - Seq("bbox", "xmin", "ymin", "xmax", "ymax").foreach { key => - assert(covering contains key) - } - } - } -} diff --git a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/SQLSyntaxTestScala.scala b/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/SQLSyntaxTestScala.scala deleted file mode 100644 index 6f873d0a087..00000000000 --- a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/SQLSyntaxTestScala.scala +++ /dev/null @@ -1,72 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql - -import org.scalatest.matchers.must.Matchers.be -import org.scalatest.matchers.should.Matchers.convertToAnyShouldWrapper -import org.scalatest.prop.TableDrivenPropertyChecks - -/** - * Test suite for testing Sedona SQL support. - */ -class SQLSyntaxTestScala extends TestBaseScala with TableDrivenPropertyChecks { - - override def beforeAll(): Unit = { - super.beforeAll() - sparkSession.conf.set("spark.sql.legacy.createHiveTableByDefault", "false") - } - - describe("Table creation DDL tests") { - - it("should be able to create a regular table without geometry column should work") { - sparkSession.sql("DROP TABLE IF EXISTS T_TEST_REGULAR") - sparkSession.sql("CREATE TABLE IF NOT EXISTS T_TEST_REGULAR (INT_COL INT)") - sparkSession.catalog.tableExists("T_TEST_REGULAR") should be(true) - sparkSession.sql("DROP TABLE IF EXISTS T_TEST_REGULAR") - sparkSession.catalog.tableExists("T_TEST_REGULAR") should be(false) - } - - it( - "should be able to create a regular table with geometry column should work without a workaround") { - try { - sparkSession.sql("CREATE TABLE T_TEST_EXPLICIT_GEOMETRY (GEO_COL GEOMETRY)") - sparkSession.catalog.tableExists("T_TEST_EXPLICIT_GEOMETRY") should be(true) - sparkSession.sparkContext.getConf.get(keyParserExtension) should be("true") - } catch { - case ex: Exception => - ex.getClass.getName.endsWith("ParseException") should be(true) - sparkSession.sparkContext.getConf.get(keyParserExtension) should be("false") - } - } - - it( - "should be able to create a regular table with regular and geometry column should work without a workaround") { - try { - sparkSession.sql( - "CREATE TABLE T_TEST_EXPLICIT_GEOMETRY_2 (INT_COL INT, GEO_COL GEOMETRY)") - sparkSession.catalog.tableExists("T_TEST_EXPLICIT_GEOMETRY_2") should be(true) - sparkSession.sparkContext.getConf.get(keyParserExtension) should be("true") - } catch { - case ex: Exception => - ex.getClass.getName.endsWith("ParseException") should be(true) - sparkSession.sparkContext.getConf.get(keyParserExtension) should be("false") - } - } - } -} diff --git a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/ShapefileTests.scala b/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/ShapefileTests.scala deleted file mode 100644 index 47d5ea1db58..00000000000 --- a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/ShapefileTests.scala +++ /dev/null @@ -1,985 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql - -import org.apache.commons.io.FileUtils -import org.apache.log4j.{AppenderSkeleton, Level, Logger} -import org.apache.log4j.spi.LoggingEvent -import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT -import org.apache.spark.sql.types.{DateType, DecimalType, LongType, StringType, StructField, StructType, TimestampType} -import org.locationtech.jts.geom.{Geometry, MultiPolygon, Point, Polygon} -import org.locationtech.jts.io.{WKTReader, WKTWriter} -import org.scalatest.BeforeAndAfterAll - -import java.io.File -import java.nio.file.Files -import java.util.{ArrayList => JList} -import scala.collection.mutable - -class ShapefileTests extends TestBaseScala with BeforeAndAfterAll { - val temporaryLocation: String = resourceFolder + "shapefiles/tmp" - - override def beforeAll(): Unit = { - super.beforeAll() - FileUtils.deleteDirectory(new File(temporaryLocation)) - Files.createDirectory(new File(temporaryLocation).toPath) - } - - override def afterAll(): Unit = FileUtils.deleteDirectory(new File(temporaryLocation)) - - describe("Shapefile read tests") { - it("read gis_osm_pois_free_1") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "osm_id").get.dataType == StringType) - assert(schema.find(_.name == "code").get.dataType == LongType) - assert(schema.find(_.name == "fclass").get.dataType == StringType) - assert(schema.find(_.name == "name").get.dataType == StringType) - assert(schema.length == 5) - assert(shapefileDf.count == 12873) - - shapefileDf.collect().foreach { row => - val geom = row.getAs[Geometry]("geometry") - assert(geom.isInstanceOf[Point]) - assert(geom.getSRID == 4326) - assert(row.getAs[String]("osm_id").nonEmpty) - assert(row.getAs[Long]("code") > 0) - assert(row.getAs[String]("fclass").nonEmpty) - assert(row.getAs[String]("name") != null) - } - - // with projection, selecting geometry and attribute fields - shapefileDf.select("geometry", "code").take(10).foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - assert(row.getAs[Long]("code") > 0) - } - - // with projection, selecting geometry fields - shapefileDf.select("geometry").take(10).foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - } - - // with projection, selecting attribute fields - shapefileDf.select("code", "osm_id").take(10).foreach { row => - assert(row.getAs[Long]("code") > 0) - assert(row.getAs[String]("osm_id").nonEmpty) - } - - // with transformation - shapefileDf - .selectExpr("ST_Buffer(geometry, 0.001) AS geom", "code", "osm_id as id") - .take(10) - .foreach { row => - assert(row.getAs[Geometry]("geom").isInstanceOf[Polygon]) - assert(row.getAs[Long]("code") > 0) - assert(row.getAs[String]("id").nonEmpty) - } - } - - it("read dbf") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/dbf") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "STATEFP").get.dataType == StringType) - assert(schema.find(_.name == "COUNTYFP").get.dataType == StringType) - assert(schema.find(_.name == "COUNTYNS").get.dataType == StringType) - assert(schema.find(_.name == "AFFGEOID").get.dataType == StringType) - assert(schema.find(_.name == "GEOID").get.dataType == StringType) - assert(schema.find(_.name == "NAME").get.dataType == StringType) - assert(schema.find(_.name == "LSAD").get.dataType == StringType) - assert(schema.find(_.name == "ALAND").get.dataType == LongType) - assert(schema.find(_.name == "AWATER").get.dataType == LongType) - assert(schema.length == 10) - assert(shapefileDf.count() == 3220) - - shapefileDf.collect().foreach { row => - val geom = row.getAs[Geometry]("geometry") - assert(geom.getSRID == 0) - assert(geom.isInstanceOf[Polygon] || geom.isInstanceOf[MultiPolygon]) - assert(row.getAs[String]("STATEFP").nonEmpty) - assert(row.getAs[String]("COUNTYFP").nonEmpty) - assert(row.getAs[String]("COUNTYNS").nonEmpty) - assert(row.getAs[String]("AFFGEOID").nonEmpty) - assert(row.getAs[String]("GEOID").nonEmpty) - assert(row.getAs[String]("NAME").nonEmpty) - assert(row.getAs[String]("LSAD").nonEmpty) - assert(row.getAs[Long]("ALAND") > 0) - assert(row.getAs[Long]("AWATER") >= 0) - } - } - - it("read multipleshapefiles") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/multipleshapefiles") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "STATEFP").get.dataType == StringType) - assert(schema.find(_.name == "COUNTYFP").get.dataType == StringType) - assert(schema.find(_.name == "COUNTYNS").get.dataType == StringType) - assert(schema.find(_.name == "AFFGEOID").get.dataType == StringType) - assert(schema.find(_.name == "GEOID").get.dataType == StringType) - assert(schema.find(_.name == "NAME").get.dataType == StringType) - assert(schema.find(_.name == "LSAD").get.dataType == StringType) - assert(schema.find(_.name == "ALAND").get.dataType == LongType) - assert(schema.find(_.name == "AWATER").get.dataType == LongType) - assert(schema.length == 10) - assert(shapefileDf.count() == 3220) - } - - it("read missing") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/missing") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "id").get.dataType == LongType) - assert(schema.find(_.name == "a").get.dataType == StringType) - assert(schema.find(_.name == "b").get.dataType == StringType) - assert(schema.find(_.name == "c").get.dataType == StringType) - assert(schema.find(_.name == "d").get.dataType == StringType) - assert(schema.find(_.name == "e").get.dataType == StringType) - assert(schema.length == 7) - val rows = shapefileDf.collect() - assert(rows.length == 3) - rows.foreach { row => - val a = row.getAs[String]("a") - val b = row.getAs[String]("b") - val c = row.getAs[String]("c") - val d = row.getAs[String]("d") - val e = row.getAs[String]("e") - if (a.isEmpty) { - assert(b == "First") - assert(c == "field") - assert(d == "is") - assert(e == "empty") - } else if (e.isEmpty) { - assert(a == "Last") - assert(b == "field") - assert(c == "is") - assert(d == "empty") - } else { - assert(a == "Are") - assert(b == "fields") - assert(c == "are") - assert(d == "not") - assert(e == "empty") - } - } - } - - it("read unsupported") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/unsupported") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - val rows = shapefileDf.collect() - assert(rows.length == 10) - rows.foreach { row => - assert(row.getAs[Geometry]("geometry") == null) - assert(!row.isNullAt(row.fieldIndex("id"))) - } - } - - it("read bad_shx") { - var shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/bad_shx") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "field_1").get.dataType == LongType) - var rows = shapefileDf.collect() - assert(rows.length == 2) - rows.foreach { row => - val geom = row.getAs[Geometry]("geometry") - if (geom == null) { - assert(row.getAs[Long]("field_1") == 3) - } else { - assert(geom.isInstanceOf[Point]) - assert(row.getAs[Long]("field_1") == 2) - } - } - - // Copy the .shp and .dbf files to temporary location, and read the same shapefiles without .shx - FileUtils.cleanDirectory(new File(temporaryLocation)) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/bad_shx/bad_shx.shp"), - new File(temporaryLocation + "/bad_shx.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/bad_shx/bad_shx.dbf"), - new File(temporaryLocation + "/bad_shx.dbf")) - shapefileDf = sparkSession.read - .format("shapefile") - .load(temporaryLocation) - rows = shapefileDf.collect() - assert(rows.length == 2) - rows.foreach { row => - val geom = row.getAs[Geometry]("geometry") - if (geom == null) { - assert(row.getAs[Long]("field_1") == 3) - } else { - assert(geom.isInstanceOf[Point]) - assert(row.getAs[Long]("field_1") == 2) - } - } - } - - it("read contains_null_geom") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/contains_null_geom") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "fInt").get.dataType == LongType) - assert(schema.find(_.name == "fFloat").get.dataType.isInstanceOf[DecimalType]) - assert(schema.find(_.name == "fString").get.dataType == StringType) - assert(schema.length == 4) - val rows = shapefileDf.collect() - assert(rows.length == 10) - rows.foreach { row => - val fInt = row.getAs[Long]("fInt") - val fFloat = row.getAs[java.math.BigDecimal]("fFloat").doubleValue() - val fString = row.getAs[String]("fString") - val geom = row.getAs[Geometry]("geometry") - if (fInt == 2 || fInt == 5) { - assert(geom == null) - } else { - assert(geom.isInstanceOf[Point]) - assert(geom.getCoordinate.x == fInt) - assert(geom.getCoordinate.y == fInt) - } - assert(Math.abs(fFloat - 3.14159 * fInt) < 1e-4) - assert(fString == s"str_$fInt") - } - } - - it("read test_datatypes") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/datatypes") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "id").get.dataType == LongType) - assert(schema.find(_.name == "aInt").get.dataType == LongType) - assert(schema.find(_.name == "aUnicode").get.dataType == StringType) - assert(schema.find(_.name == "aDecimal").get.dataType.isInstanceOf[DecimalType]) - assert(schema.find(_.name == "aDecimal2").get.dataType.isInstanceOf[DecimalType]) - assert(schema.find(_.name == "aDate").get.dataType == DateType) - assert(schema.length == 7) - - val rows = shapefileDf.collect() - assert(rows.length == 9) - rows.foreach { row => - val geom = row.getAs[Geometry]("geometry") - assert(geom.isInstanceOf[Point]) - assert(geom.getSRID == 4269) - val idIndex = row.fieldIndex("id") - if (row.isNullAt(idIndex)) { - assert(row.isNullAt(row.fieldIndex("aInt"))) - assert(row.getAs[String]("aUnicode").isEmpty) - assert(row.isNullAt(row.fieldIndex("aDecimal"))) - assert(row.isNullAt(row.fieldIndex("aDecimal2"))) - assert(row.isNullAt(row.fieldIndex("aDate"))) - } else { - val id = row.getLong(idIndex) - assert(row.getAs[Long]("aInt") == id) - assert(row.getAs[String]("aUnicode") == s"测试$id") - if (id < 10) { - val decimal = row.getDecimal(row.fieldIndex("aDecimal")).doubleValue() - assert((decimal * 10).toInt == id * 10 + id) - assert(row.isNullAt(row.fieldIndex("aDecimal2"))) - assert(row.getAs[java.sql.Date]("aDate").toString == s"202$id-0$id-0$id") - } else { - assert(row.isNullAt(row.fieldIndex("aDecimal"))) - val decimal = row.getDecimal(row.fieldIndex("aDecimal2")).doubleValue() - assert((decimal * 100).toInt == id * 100 + id) - assert(row.isNullAt(row.fieldIndex("aDate"))) - } - } - } - } - - it("read with .shp path specified") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/datatypes/datatypes1.shp") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "id").get.dataType == LongType) - assert(schema.find(_.name == "aInt").get.dataType == LongType) - assert(schema.find(_.name == "aUnicode").get.dataType == StringType) - assert(schema.find(_.name == "aDecimal").get.dataType.isInstanceOf[DecimalType]) - assert(schema.find(_.name == "aDate").get.dataType == DateType) - assert(schema.length == 6) - - val rows = shapefileDf.collect() - assert(rows.length == 5) - rows.foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - val idIndex = row.fieldIndex("id") - if (row.isNullAt(idIndex)) { - assert(row.isNullAt(row.fieldIndex("aInt"))) - assert(row.getAs[String]("aUnicode").isEmpty) - assert(row.isNullAt(row.fieldIndex("aDecimal"))) - assert(row.isNullAt(row.fieldIndex("aDate"))) - } else { - val id = row.getLong(idIndex) - assert(row.getAs[Long]("aInt") == id) - assert(row.getAs[String]("aUnicode") == s"测试$id") - val decimal = row.getDecimal(row.fieldIndex("aDecimal")).doubleValue() - assert((decimal * 10).toInt == id * 10 + id) - assert(row.getAs[java.sql.Date]("aDate").toString == s"202$id-0$id-0$id") - } - } - } - - it("read with glob path specified") { - val shapefileDf = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/datatypes/datatypes2.*") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "id").get.dataType == LongType) - assert(schema.find(_.name == "aInt").get.dataType == LongType) - assert(schema.find(_.name == "aUnicode").get.dataType == StringType) - assert(schema.find(_.name == "aDecimal2").get.dataType.isInstanceOf[DecimalType]) - assert(schema.length == 5) - - val rows = shapefileDf.collect() - assert(rows.length == 4) - rows.foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - val id = row.getAs[Long]("id") - assert(row.getAs[Long]("aInt") == id) - assert(row.getAs[String]("aUnicode") == s"测试$id") - val decimal = row.getDecimal(row.fieldIndex("aDecimal2")).doubleValue() - assert((decimal * 100).toInt == id * 100 + id) - } - } - - it("read without shx") { - FileUtils.cleanDirectory(new File(temporaryLocation)) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.shp"), - new File(temporaryLocation + "/gis_osm_pois_free_1.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.dbf"), - new File(temporaryLocation + "/gis_osm_pois_free_1.dbf")) - - val shapefileDf = sparkSession.read - .format("shapefile") - .load(temporaryLocation) - val rows = shapefileDf.collect() - assert(rows.length == 12873) - rows.foreach { row => - val geom = row.getAs[Geometry]("geometry") - assert(geom.isInstanceOf[Point]) - assert(geom.getSRID == 0) - assert(row.getAs[String]("osm_id").nonEmpty) - assert(row.getAs[Long]("code") > 0) - assert(row.getAs[String]("fclass").nonEmpty) - assert(row.getAs[String]("name") != null) - } - } - - it("read without dbf") { - FileUtils.cleanDirectory(new File(temporaryLocation)) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.shp"), - new File(temporaryLocation + "/gis_osm_pois_free_1.shp")) - val shapefileDf = sparkSession.read - .format("shapefile") - .load(temporaryLocation) - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.length == 1) - - val rows = shapefileDf.collect() - assert(rows.length == 12873) - rows.foreach { row => - val geom = row.getAs[Geometry]("geometry") - assert(geom.isInstanceOf[Point]) - } - } - - it("read without shp") { - FileUtils.cleanDirectory(new File(temporaryLocation)) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.dbf"), - new File(temporaryLocation + "/gis_osm_pois_free_1.dbf")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.shx"), - new File(temporaryLocation + "/gis_osm_pois_free_1.shx")) - intercept[Exception] { - sparkSession.read - .format("shapefile") - .load(temporaryLocation) - .count() - } - - intercept[Exception] { - sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.shx") - .count() - } - } - - it("read directory containing missing .shp files") { - FileUtils.cleanDirectory(new File(temporaryLocation)) - // Missing .shp file for datatypes1 - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes1.dbf"), - new File(temporaryLocation + "/datatypes1.dbf")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.shp"), - new File(temporaryLocation + "/datatypes2.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.dbf"), - new File(temporaryLocation + "/datatypes2.dbf")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.cpg"), - new File(temporaryLocation + "/datatypes2.cpg")) - - val shapefileDf = sparkSession.read - .format("shapefile") - .load(temporaryLocation) - val rows = shapefileDf.collect() - assert(rows.length == 4) - rows.foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - val id = row.getAs[Long]("id") - assert(row.getAs[Long]("aInt") == id) - assert(row.getAs[String]("aUnicode") == s"测试$id") - val decimal = row.getDecimal(row.fieldIndex("aDecimal2")).doubleValue() - assert((decimal * 100).toInt == id * 100 + id) - } - } - - it("read partitioned directory") { - FileUtils.cleanDirectory(new File(temporaryLocation)) - Files.createDirectory(new File(temporaryLocation + "/part=1").toPath) - Files.createDirectory(new File(temporaryLocation + "/part=2").toPath) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes1.shp"), - new File(temporaryLocation + "/part=1/datatypes1.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes1.dbf"), - new File(temporaryLocation + "/part=1/datatypes1.dbf")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes1.cpg"), - new File(temporaryLocation + "/part=1/datatypes1.cpg")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.shp"), - new File(temporaryLocation + "/part=2/datatypes2.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.dbf"), - new File(temporaryLocation + "/part=2/datatypes2.dbf")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.cpg"), - new File(temporaryLocation + "/part=2/datatypes2.cpg")) - - val shapefileDf = sparkSession.read - .format("shapefile") - .load(temporaryLocation) - .select("part", "id", "aInt", "aUnicode", "geometry") - val rows = shapefileDf.collect() - assert(rows.length == 9) - rows.foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - val id = row.getAs[Long]("id") - assert(row.getAs[Long]("aInt") == id) - if (id < 10) { - assert(row.getAs[Int]("part") == 1) - } else { - assert(row.getAs[Int]("part") == 2) - } - if (id > 0) { - assert(row.getAs[String]("aUnicode") == s"测试$id") - } - } - } - - it("read with recursiveFileLookup") { - FileUtils.cleanDirectory(new File(temporaryLocation)) - Files.createDirectory(new File(temporaryLocation + "/part1").toPath) - Files.createDirectory(new File(temporaryLocation + "/part2").toPath) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes1.shp"), - new File(temporaryLocation + "/part1/datatypes1.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes1.dbf"), - new File(temporaryLocation + "/part1/datatypes1.dbf")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes1.cpg"), - new File(temporaryLocation + "/part1/datatypes1.cpg")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.shp"), - new File(temporaryLocation + "/part2/datatypes2.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.dbf"), - new File(temporaryLocation + "/part2/datatypes2.dbf")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.cpg"), - new File(temporaryLocation + "/part2/datatypes2.cpg")) - - val shapefileDf = sparkSession.read - .format("shapefile") - .option("recursiveFileLookup", "true") - .load(temporaryLocation) - .select("id", "aInt", "aUnicode", "geometry") - val rows = shapefileDf.collect() - assert(rows.length == 9) - rows.foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - val id = row.getAs[Long]("id") - assert(row.getAs[Long]("aInt") == id) - if (id > 0) { - assert(row.getAs[String]("aUnicode") == s"测试$id") - } - } - } - - it("read with custom geometry column name") { - val shapefileDf = sparkSession.read - .format("shapefile") - .option("geometry.name", "geom") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geom").get.dataType == GeometryUDT) - assert(schema.find(_.name == "osm_id").get.dataType == StringType) - assert(schema.find(_.name == "code").get.dataType == LongType) - assert(schema.find(_.name == "fclass").get.dataType == StringType) - assert(schema.find(_.name == "name").get.dataType == StringType) - assert(schema.length == 5) - val rows = shapefileDf.collect() - assert(rows.length == 12873) - rows.foreach { row => - val geom = row.getAs[Geometry]("geom") - assert(geom.isInstanceOf[Point]) - assert(row.getAs[String]("osm_id").nonEmpty) - assert(row.getAs[Long]("code") > 0) - assert(row.getAs[String]("fclass").nonEmpty) - assert(row.getAs[String]("name") != null) - } - - val exception = intercept[Exception] { - sparkSession.read - .format("shapefile") - .option("geometry.name", "osm_id") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - } - assert( - exception.getMessage.contains( - "osm_id is reserved for geometry but appears in non-spatial attributes")) - } - - it("read with shape key column") { - val shapefileDf = sparkSession.read - .format("shapefile") - .option("key.name", "fid") - .load(resourceFolder + "shapefiles/datatypes") - .select("id", "fid", "geometry", "aUnicode") - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - assert(schema.find(_.name == "id").get.dataType == LongType) - assert(schema.find(_.name == "fid").get.dataType == LongType) - assert(schema.find(_.name == "aUnicode").get.dataType == StringType) - val rows = shapefileDf.collect() - assert(rows.length == 9) - rows.foreach { row => - val geom = row.getAs[Geometry]("geometry") - assert(geom.isInstanceOf[Point]) - val id = row.getAs[Long]("id") - if (id > 0) { - assert(row.getAs[Long]("fid") == id % 10) - assert(row.getAs[String]("aUnicode") == s"测试$id") - } else { - assert(row.getAs[Long]("fid") == 5) - } - } - } - - it("read with both custom geometry column and shape key column") { - val shapefileDf = sparkSession.read - .format("shapefile") - .option("geometry.name", "g") - .option("key.name", "fid") - .load(resourceFolder + "shapefiles/datatypes") - .select("id", "fid", "g", "aUnicode") - val schema = shapefileDf.schema - assert(schema.find(_.name == "g").get.dataType == GeometryUDT) - assert(schema.find(_.name == "id").get.dataType == LongType) - assert(schema.find(_.name == "fid").get.dataType == LongType) - assert(schema.find(_.name == "aUnicode").get.dataType == StringType) - val rows = shapefileDf.collect() - assert(rows.length == 9) - rows.foreach { row => - val geom = row.getAs[Geometry]("g") - assert(geom.isInstanceOf[Point]) - val id = row.getAs[Long]("id") - if (id > 0) { - assert(row.getAs[Long]("fid") == id % 10) - assert(row.getAs[String]("aUnicode") == s"测试$id") - } else { - assert(row.getAs[Long]("fid") == 5) - } - } - } - - it("read with invalid shape key column") { - val exception = intercept[Exception] { - sparkSession.read - .format("shapefile") - .option("geometry.name", "g") - .option("key.name", "aDate") - .load(resourceFolder + "shapefiles/datatypes") - } - assert( - exception.getMessage.contains( - "aDate is reserved for shape key but appears in non-spatial attributes")) - - val exception2 = intercept[Exception] { - sparkSession.read - .format("shapefile") - .option("geometry.name", "g") - .option("key.name", "g") - .load(resourceFolder + "shapefiles/datatypes") - } - assert(exception2.getMessage.contains("geometry.name and key.name cannot be the same")) - } - - it("read with custom charset") { - FileUtils.cleanDirectory(new File(temporaryLocation)) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.shp"), - new File(temporaryLocation + "/datatypes2.shp")) - FileUtils.copyFile( - new File(resourceFolder + "shapefiles/datatypes/datatypes2.dbf"), - new File(temporaryLocation + "/datatypes2.dbf")) - - val shapefileDf = sparkSession.read - .format("shapefile") - .option("charset", "GB2312") - .load(temporaryLocation) - val rows = shapefileDf.collect() - assert(rows.length == 4) - rows.foreach { row => - assert(row.getAs[Geometry]("geometry").isInstanceOf[Point]) - val id = row.getAs[Long]("id") - assert(row.getAs[Long]("aInt") == id) - assert(row.getAs[String]("aUnicode") == s"测试$id") - val decimal = row.getDecimal(row.fieldIndex("aDecimal2")).doubleValue() - assert((decimal * 100).toInt == id * 100 + id) - } - } - - it("read with custom schema") { - val customSchema = StructType( - Seq( - StructField("osm_id", StringType), - StructField("code2", LongType), - StructField("geometry", GeometryUDT()))) - val shapefileDf = sparkSession.read - .format("shapefile") - .schema(customSchema) - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - assert(shapefileDf.schema == customSchema) - val rows = shapefileDf.collect() - assert(rows.length == 12873) - rows.foreach { row => - val geom = row.getAs[Geometry]("geometry") - assert(geom.isInstanceOf[Point]) - assert(row.getAs[String]("osm_id").nonEmpty) - assert(row.isNullAt(row.fieldIndex("code2"))) - } - } - - it("should read shapes of various types") { - // There are multiple directories under shapefiles/shapetypes, each containing a shapefile. - // We'll iterate over each directory and read the shapefile within it. - val shapeTypesDir = new File(resourceFolder + "shapefiles/shapetypes") - val shapeTypeDirs = shapeTypesDir.listFiles().filter(_.isDirectory) - shapeTypeDirs.foreach { shapeTypeDir => - val fileName = shapeTypeDir.getName - val hasZ = fileName.endsWith("zm") || fileName.endsWith("z") - val hasM = fileName.endsWith("zm") || fileName.endsWith("m") - val shapeType = - if (fileName.startsWith("point")) "POINT" - else if (fileName.startsWith("linestring")) "LINESTRING" - else if (fileName.startsWith("multipoint")) "MULTIPOINT" - else "POLYGON" - val expectedWktPrefix = - if (!hasZ && !hasM) shapeType - else { - shapeType + " " + (if (hasZ) "Z" else "") + (if (hasM) "M" else "") - } - - val shapefileDf = sparkSession.read - .format("shapefile") - .load(shapeTypeDir.getAbsolutePath) - val schema = shapefileDf.schema - assert(schema.find(_.name == "geometry").get.dataType == GeometryUDT) - val rows = shapefileDf.collect() - assert(rows.length > 0) - - // Validate the geometry type and WKT prefix - val wktWriter = new WKTWriter(4) - val rowsMap = mutable.Map[String, Geometry]() - rows.foreach { row => - val id = row.getAs[String]("id") - val geom = row.getAs[Geometry]("geometry") - val wkt = wktWriter.write(geom) - assert(wkt.startsWith(expectedWktPrefix)) - assert(geom != null) - rowsMap.put(id, geom) - } - - // Validate the geometry values by reading the CSV file containing the same data - val csvDf = sparkSession.read - .format("csv") - .option("header", "true") - .load(shapeTypeDir.getAbsolutePath + "/*.csv") - val wktReader = new WKTReader() - csvDf.collect().foreach { row => - val id = row.getAs[String]("id") - val wkt = row.getAs[String]("wkt") - val geom = wktReader.read(wkt) - assert(rowsMap(id).equals(geom)) - } - } - } - - it("should expose _metadata struct with all expected fields") { - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val metaDf = df.select("_metadata") - val metaSchema = metaDf.schema("_metadata").dataType.asInstanceOf[StructType] - val expectedFields = - Seq( - "file_path", - "file_name", - "file_size", - "file_block_start", - "file_block_length", - "file_modification_time") - assert(metaSchema.fieldNames.toSeq == expectedFields) - assert(metaSchema("file_path").dataType == StringType) - assert(metaSchema("file_name").dataType == StringType) - assert(metaSchema("file_size").dataType == LongType) - assert(metaSchema("file_block_start").dataType == LongType) - assert(metaSchema("file_block_length").dataType == LongType) - assert(metaSchema("file_modification_time").dataType == TimestampType) - } - - it("should not include _metadata in select(*)") { - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val columns = df.columns - assert(!columns.contains("_metadata")) - } - - it("should return correct file_path and file_name in _metadata") { - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val metaRows = df.select("_metadata.file_path", "_metadata.file_name").distinct().collect() - assert(metaRows.length == 1) - val filePath = metaRows.head.getString(0) - val fileName = metaRows.head.getString(1) - assert(filePath.endsWith("gis_osm_pois_free_1.shp")) - assert(fileName == "gis_osm_pois_free_1.shp") - } - - it("should return actual file_size matching the .shp file on disk") { - val shpFile = - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.shp") - val expectedSize = shpFile.length() - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val sizes = df.select("_metadata.file_size").distinct().collect() - assert(sizes.length == 1) - assert(sizes.head.getLong(0) == expectedSize) - } - - it( - "should return file_block_start=0 and file_block_length=file_size for non-splittable shapefiles") { - val shpFile = - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.shp") - val expectedSize = shpFile.length() - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val rows = df - .select("_metadata.file_block_start", "_metadata.file_block_length") - .distinct() - .collect() - assert(rows.length == 1) - assert(rows.head.getLong(0) == 0L) // file_block_start - assert(rows.head.getLong(1) == expectedSize) // file_block_length - } - - it("should return file_modification_time matching the .shp file on disk") { - val shpFile = - new File(resourceFolder + "shapefiles/gis_osm_pois_free_1/gis_osm_pois_free_1.shp") - // File.lastModified() returns milliseconds, Spark TimestampType stores microseconds - val expectedModTimeMs = shpFile.lastModified() - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val times = - df.select("_metadata.file_modification_time").distinct().collect() - assert(times.length == 1) - val modTime = times.head.getTimestamp(0) - assert(modTime != null) - // Timestamp.getTime() returns milliseconds - assert(modTime.getTime == expectedModTimeMs) - } - - it("should return correct metadata values per file when reading multiple shapefiles") { - val map1Shp = - new File(resourceFolder + "shapefiles/multipleshapefiles/map1.shp") - val map2Shp = - new File(resourceFolder + "shapefiles/multipleshapefiles/map2.shp") - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/multipleshapefiles") - val metaRows = df - .select( - "_metadata.file_name", - "_metadata.file_size", - "_metadata.file_block_start", - "_metadata.file_block_length") - .distinct() - .collect() - assert(metaRows.length == 2) - val byName = metaRows.map(r => r.getString(0) -> r).toMap - // map1.shp - assert(byName("map1.shp").getLong(1) == map1Shp.length()) - assert(byName("map1.shp").getLong(2) == 0L) - assert(byName("map1.shp").getLong(3) == map1Shp.length()) - // map2.shp - assert(byName("map2.shp").getLong(1) == map2Shp.length()) - assert(byName("map2.shp").getLong(2) == 0L) - assert(byName("map2.shp").getLong(3) == map2Shp.length()) - } - - it("should allow filtering on _metadata fields") { - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/multipleshapefiles") - val totalCount = df.count() - val map1Df = df.filter(df("_metadata.file_name") === "map1.shp") - val map2Df = df.filter(df("_metadata.file_name") === "map2.shp") - assert(map1Df.count() > 0) - assert(map2Df.count() > 0) - assert(map1Df.count() + map2Df.count() == totalCount) - } - - it("should select _metadata along with data columns") { - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/gis_osm_pois_free_1") - val result = df.select("osm_id", "_metadata.file_name").collect() - assert(result.length == 12873) - result.foreach { row => - assert(row.getString(0).nonEmpty) - assert(row.getString(1) == "gis_osm_pois_free_1.shp") - } - } - - it("should return correct metadata for each file in multi-shapefile directory") { - val dt1Shp = new File(resourceFolder + "shapefiles/datatypes/datatypes1.shp") - val dt2Shp = new File(resourceFolder + "shapefiles/datatypes/datatypes2.shp") - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/datatypes") - val result = df - .select( - "_metadata.file_path", - "_metadata.file_name", - "_metadata.file_size", - "_metadata.file_block_start", - "_metadata.file_block_length", - "_metadata.file_modification_time") - .distinct() - .collect() - assert(result.length == 2) - val byName = result.map(r => r.getString(1) -> r).toMap - // datatypes1.shp - val r1 = byName("datatypes1.shp") - assert(r1.getString(0).endsWith("datatypes1.shp")) - assert(r1.getLong(2) == dt1Shp.length()) - assert(r1.getLong(3) == 0L) - assert(r1.getLong(4) == dt1Shp.length()) - assert(r1.getTimestamp(5).getTime == dt1Shp.lastModified()) - // datatypes2.shp - val r2 = byName("datatypes2.shp") - assert(r2.getString(0).endsWith("datatypes2.shp")) - assert(r2.getLong(2) == dt2Shp.length()) - assert(r2.getLong(3) == 0L) - assert(r2.getLong(4) == dt2Shp.length()) - assert(r2.getTimestamp(5).getTime == dt2Shp.lastModified()) - } - - it("reading shapefile by .shp path should not produce FileStreamSink metadata warning") { - // GH-2650: When reading shapefiles by .shp path, ShapefileDataSource.transformPaths - // converts it to a glob pattern (e.g., "file.???"). Without the fix, Spark's - // FileTable.fileIndex calls FileStreamSink.hasMetadata which tries to stat the glob - // path as a directory, causing a FileNotFoundException and a spurious WARN log: - // "Assume no metadata directory. Error while looking for metadata directory..." - val capturedWarnings = new JList[String]() - val appender = new AppenderSkeleton { - override def append(event: LoggingEvent): Unit = { - val msg = event.getRenderedMessage - if (msg != null && msg.contains("Assume no metadata directory")) { - capturedWarnings.add(msg) - } - } - override def close(): Unit = {} - override def requiresLayout(): Boolean = false - } - appender.setThreshold(Level.WARN) - val rootLogger = Logger.getRootLogger - rootLogger.addAppender(appender) - try { - val df = sparkSession.read - .format("shapefile") - .load(resourceFolder + "shapefiles/datatypes/datatypes1.shp") - df.collect() - assert( - capturedWarnings.isEmpty, - "FileStreamSink metadata warning should not be emitted when reading shapefiles " + - "by .shp path. This warning is caused by FileStreamSink.hasMetadata trying to " + - "stat the glob path as a directory. Captured warnings: " + capturedWarnings) - } finally { - rootLogger.removeAppender(appender) - } - } - } -} diff --git a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/TestBaseScala.scala b/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/TestBaseScala.scala deleted file mode 100644 index ce56bc4fa56..00000000000 --- a/spark/spark-3.4/src/test/scala/org/apache/sedona/sql/TestBaseScala.scala +++ /dev/null @@ -1,128 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.sedona.sql - -import io.minio.{MinioClient, PutObjectArgs} -import org.apache.log4j.{Level, Logger} -import org.apache.sedona.spark.SedonaContext -import org.apache.spark.sql.{DataFrame, SparkSession} -import org.scalatest.{BeforeAndAfterAll, FunSpec} -import org.testcontainers.containers.MinIOContainer - -import java.io.FileInputStream -import java.util.concurrent.ThreadLocalRandom - -trait TestBaseScala extends FunSpec with BeforeAndAfterAll { - Logger.getRootLogger().setLevel(Level.WARN) - Logger.getLogger("org.apache").setLevel(Level.WARN) - Logger.getLogger("com").setLevel(Level.WARN) - Logger.getLogger("akka").setLevel(Level.WARN) - Logger.getLogger("org.apache.sedona.core").setLevel(Level.WARN) - - val keyParserExtension = "spark.sedona.enableParserExtension" - val warehouseLocation = System.getProperty("user.dir") + "/target/" - val sparkSession = SedonaContext - .builder() - .master("local[*]") - .appName("sedonasqlScalaTest") - .config("spark.sql.warehouse.dir", warehouseLocation) - // We need to be explicit about broadcasting in tests. - .config("sedona.join.autoBroadcastJoinThreshold", "-1") - .config("spark.sql.extensions", "org.apache.sedona.sql.SedonaSqlExtensions") - .config(keyParserExtension, ThreadLocalRandom.current().nextBoolean()) - .getOrCreate() - - val sparkSessionMinio = SedonaContext - .builder() - .master("local[*]") - .appName("sedonasqlScalaTest") - .config("spark.sql.warehouse.dir", warehouseLocation) - .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.0") - .config( - "spark.hadoop.fs.s3a.aws.credentials.provider", - "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider") - .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") - .config("sedona.join.autoBroadcastJoinThreshold", "-1") - .getOrCreate() - - val resourceFolder = System.getProperty("user.dir") + "/../common/src/test/resources/" - - override def beforeAll(): Unit = { - SedonaContext.create(sparkSession) - } - - override def afterAll(): Unit = { - // SedonaSQLRegistrator.dropAll(spark) - // spark.stop - } - - def loadCsv(path: String): DataFrame = { - sparkSession.read.format("csv").option("delimiter", ",").option("header", "false").load(path) - } - - def withConf[T](conf: Map[String, String])(f: => T): T = { - val oldConf = conf.keys.map(key => key -> sparkSession.conf.getOption(key)) - conf.foreach { case (key, value) => sparkSession.conf.set(key, value) } - try { - f - } finally { - oldConf.foreach { case (key, value) => - value match { - case Some(v) => sparkSession.conf.set(key, v) - case None => sparkSession.conf.unset(key) - } - } - } - } - - def putFileIntoBucket( - bucketName: String, - key: String, - stream: FileInputStream, - client: MinioClient): Unit = { - val objectArguments = PutObjectArgs - .builder() - .bucket(bucketName) - .`object`(key) - .stream(stream, stream.available(), -1) - .build() - - client.putObject(objectArguments) - } - - def createMinioClient(container: MinIOContainer): MinioClient = { - MinioClient - .builder() - .endpoint(container.getS3URL) - .credentials(container.getUserName, container.getPassword) - .build() - } - - def adjustSparkSession(sparkSession: SparkSession, container: MinIOContainer): Unit = { - sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.endpoint", container.getS3URL) - sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.access.key", container.getUserName) - sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key", container.getPassword) - sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.connection.timeout", "2000") - - sparkSession.sparkContext.hadoopConfiguration.set("spark.sql.debug.maxToStringFields", "100") - sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.path.style.access", "true") - sparkSession.sparkContext.hadoopConfiguration - .set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") - } -} From de44425f945f3dec20a0d7e6fb0d93b1761160e3 Mon Sep 17 00:00:00 2001 From: Jia Yu Date: Fri, 28 Aug 2026 18:44:54 -0700 Subject: [PATCH 2/6] [GH-3297] Fix Spark 4.0 Scala 2.12 coordinates and guard docker builds Two problems found in review of the Spark 3.4 removal. Spark 4.x has no Scala 2.12 build: spark-core_2.12:4.0.0 and 4.1.1 are both absent from Maven Central, and Sedona has only ever published sedona-spark-shaded-4.0_2.13. Building with -Dspark=4.0 -Dscala=2.12 fails resolving spark-core_2.12 and spark-sql_2.12, because the scala2.12 profile is declared after sedona-spark-4.0 and so wins on scala.compat.version. - Drop the Spark 4.0 / Scala 2.12 deploy step from the snapshot docs - Point the tutorial examples at sedona-spark-shaded-4.0_2.13 - Remove the Spark 4.0 / Scala 2.12 coordinate tabs, and unmark that combination in the compatibility tables. publish.md already skipped Spark 4.x with Scala 2.12 as unsupported, so the tables and tabs contradicted the release procedure. The docker docs built against Spark 3.4.1. build.sh derives -Dspark from the first two components of that version, and with the 3.4 profile gone Maven matched no profile and silently used the new 3.5 defaults, installing a 3.5 shaded jar into a Spark 3.4 image. The STAC reader now calls the eight-argument PartitionedFile constructor directly, which Spark 3.4 does not have, so that mismatch would fail at runtime. - Build the documented images against Spark 4.0.1, which is what the dockerfile already defaults to and what CI exercises - Make build.sh reject Spark versions outside 3.5, 4.0 and 4.1 rather than letting Maven fall back to its defaults --- docker/build.sh | 15 +++++++++++---- docs/community/snapshot.md | 3 --- docs/community/snapshot.zh.md | 3 --- docs/setup/docker.md | 6 +++--- docs/setup/docker.zh.md | 6 +++--- docs/setup/maven-coordinates.md | 30 ------------------------------ docs/setup/maven-coordinates.zh.md | 30 ------------------------------ docs/setup/platform.md | 4 ++-- docs/setup/platform.zh.md | 4 ++-- docs/tutorial/raster.md | 2 +- docs/tutorial/raster.zh.md | 2 +- docs/tutorial/sql.md | 2 +- docs/tutorial/sql.zh.md | 2 +- 13 files changed, 25 insertions(+), 84 deletions(-) diff --git a/docker/build.sh b/docker/build.sh index 77c8a54767a..69f5b9c53b9 100755 --- a/docker/build.sh +++ b/docker/build.sh @@ -23,10 +23,17 @@ BUILD_MODE=$3 GEOTOOLS_VERSION=${4:-auto} SEDONA_SPARK_VERSION=${SPARK_VERSION:0:3} -if [ "${SPARK_VERSION:0:1}" -eq "3" ] && [ "${SPARK_VERSION:2:1}" -le "3" ]; then - # 3.0, 3.1, 3.2, 3.3 - SEDONA_SPARK_VERSION=3.0 -fi +# Reject Spark versions Sedona no longer builds for. Without this check an +# unsupported -Dspark value simply matches no Maven profile, so the build +# silently falls back to the default Spark version and produces jars that do +# not match the Spark image they are installed into. +case "$SEDONA_SPARK_VERSION" in + 3.5 | 4.0 | 4.1) ;; + *) + echo "Unsupported Spark version: ${SPARK_VERSION}. Sedona supports Spark 3.5, 4.0 and 4.1." >&2 + exit 1 + ;; +esac # Function to compare two version numbers version_gt() { diff --git a/docs/community/snapshot.md b/docs/community/snapshot.md index 408515e98b3..173cfc14ba0 100644 --- a/docs/community/snapshot.md +++ b/docs/community/snapshot.md @@ -65,9 +65,6 @@ mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.12 # Spark 3.5 and Scala 2.13 mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.13 -# Spark 4.0 and Scala 2.12 -mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.12 - # Spark 4.0 and Scala 2.13 mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.13 diff --git a/docs/community/snapshot.zh.md b/docs/community/snapshot.zh.md index 8aa80ed114c..2b102d8d980 100644 --- a/docs/community/snapshot.zh.md +++ b/docs/community/snapshot.zh.md @@ -65,9 +65,6 @@ mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.12 # Spark 3.5 与 Scala 2.13 mvn -q deploy -DskipTests -Dspark=3.5 -Dscala=2.13 -# Spark 4.0 与 Scala 2.12 -mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.12 - # Spark 4.0 与 Scala 2.13 mvn -q deploy -DskipTests -Dspark=4.0 -Dscala=2.13 diff --git a/docs/setup/docker.md b/docs/setup/docker.md index b82fad0b592..9d3f664d088 100644 --- a/docs/setup/docker.md +++ b/docs/setup/docker.md @@ -115,7 +115,7 @@ Format: Example: ```bash -./docker/build.sh 3.4.1 {{ sedona.current_version }} +./docker/build.sh 4.0.1 {{ sedona.current_version }} ``` `build_mode` is optional. If its value is not given or is `local`, the script will build the image locally. Otherwise, it will start a cross-platform compilation and push images directly to DockerHub. @@ -133,7 +133,7 @@ Format: Example: ```bash -./docker/build.sh 3.4.1 latest +./docker/build.sh 4.0.1 latest ``` `build_mode` is optional. If its value is not given or is `local`, the script will build the image locally. Otherwise, it will start a cross-platform compilation and push images directly to DockerHub. @@ -175,5 +175,5 @@ Example: ```bash docker login -./docker/build.sh 3.4.1 {{ sedona.current_version }} release +./docker/build.sh 4.0.1 {{ sedona.current_version }} release ``` diff --git a/docs/setup/docker.zh.md b/docs/setup/docker.zh.md index cd5ce8847a7..53f0819c15b 100644 --- a/docs/setup/docker.zh.md +++ b/docs/setup/docker.zh.md @@ -115,7 +115,7 @@ docker run -d -e DRIVER_MEM=6g -e EXECUTOR_MEM=8g \ 示例： ```bash -./docker/build.sh 3.4.1 {{ sedona.current_version }} +./docker/build.sh 4.0.1 {{ sedona.current_version }} ``` `build_mode` 为可选项。如果未提供或值为 `local`，脚本会在本地构建镜像；否则会启动跨平台编译并直接将镜像推送到 DockerHub。 @@ -133,7 +133,7 @@ docker run -d -e DRIVER_MEM=6g -e EXECUTOR_MEM=8g \ 示例： ```bash -./docker/build.sh 3.4.1 latest +./docker/build.sh 4.0.1 latest ``` `build_mode` 为可选项，含义与上文相同。 @@ -175,5 +175,5 @@ docker login ```bash docker login -./docker/build.sh 3.4.1 {{ sedona.current_version }} release +./docker/build.sh 4.0.1 {{ sedona.current_version }} release ``` diff --git a/docs/setup/maven-coordinates.md b/docs/setup/maven-coordinates.md index 7c9b29b1270..8e3b39f80b8 100644 --- a/docs/setup/maven-coordinates.md +++ b/docs/setup/maven-coordinates.md @@ -53,22 +53,6 @@ The optional GeoTools library is required if you want to use raster operators. V ``` - === "Spark 4.0 and Scala 2.12" - - ```xml - - org.apache.sedona - sedona-spark-shaded-4.0_2.12 - {{ sedona.current_version }} - - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` - !!! abstract "Sedona with Apache Spark and Scala 2.13" === "Spark 3.5 and Scala 2.13" @@ -183,20 +167,6 @@ The optional GeoTools library is required if you want to use raster operators. V {{ sedona.current_geotools }} ``` - === "Spark 4.0 and Scala 2.12" - ```xml - - org.apache.sedona - sedona-spark-4.0_2.12 - {{ sedona.current_version }} - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` - !!! abstract "Sedona with Apache Spark and Scala 2.13" === "Spark 3.5 and Scala 2.13" diff --git a/docs/setup/maven-coordinates.zh.md b/docs/setup/maven-coordinates.zh.md index a02cff8f9ba..54bb60cd4f0 100644 --- a/docs/setup/maven-coordinates.zh.md +++ b/docs/setup/maven-coordinates.zh.md @@ -53,22 +53,6 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 ``` - === "Spark 4.0 与 Scala 2.12" - - ```xml - - org.apache.sedona - sedona-spark-shaded-4.0_2.12 - {{ sedona.current_version }} - - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` - !!! abstract "Sedona 与 Apache Spark + Scala 2.13" === "Spark 3.5 与 Scala 2.13" @@ -183,20 +167,6 @@ Apache Sedona 针对每个受支持的 Spark 版本提供了不同的包。 {{ sedona.current_geotools }} ``` - === "Spark 4.0 与 Scala 2.12" - ```xml - - org.apache.sedona - sedona-spark-4.0_2.12 - {{ sedona.current_version }} - - - org.datasyslab - geotools-wrapper - {{ sedona.current_geotools }} - - ``` - !!! abstract "Sedona 与 Apache Spark + Scala 2.13" === "Spark 3.5 与 Scala 2.13" diff --git a/docs/setup/platform.md b/docs/setup/platform.md index 25abf6a6ecf..f1387ec779e 100644 --- a/docs/setup/platform.md +++ b/docs/setup/platform.md @@ -30,7 +30,7 @@ Sedona binary releases are compiled by Java 11/17 and Scala 2.12/2.13 and tested | | Spark 3.5 | Spark 4.0 | Spark 4.1 | |:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 |✅ |✅ | | + | Scala 2.12 |✅ | | | | Scala 2.13 |✅ |✅ |✅ | === "Sedona Python" @@ -46,4 +46,4 @@ Sedona binary releases are compiled by Java 11/17 and Scala 2.12/2.13 and tested | | Spark 3.5 | Spark 4.0 | Spark 4.1 | |:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 | ✅ | ✅ | | + | Scala 2.12 | ✅ | | | diff --git a/docs/setup/platform.zh.md b/docs/setup/platform.zh.md index 5f3699382c5..67a6bd0c550 100644 --- a/docs/setup/platform.zh.md +++ b/docs/setup/platform.zh.md @@ -30,7 +30,7 @@ Sedona 二进制版本由 Java 11/17 与 Scala 2.12/2.13 编译，并在以下 | | Spark 3.5 | Spark 4.0 | Spark 4.1 | |:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 |✅ |✅ | | + | Scala 2.12 |✅ | | | | Scala 2.13 |✅ |✅ |✅ | === "Sedona Python" @@ -46,4 +46,4 @@ Sedona 二进制版本由 Java 11/17 与 Scala 2.12/2.13 编译，并在以下 | | Spark 3.5 | Spark 4.0 | Spark 4.1 | |:---------:|:---------:|:---------:|:---------:| - | Scala 2.12 | ✅ | ✅ | | + | Scala 2.12 | ✅ | | | diff --git a/docs/tutorial/raster.md b/docs/tutorial/raster.md index e8403a0ac09..e930922b493 100644 --- a/docs/tutorial/raster.md +++ b/docs/tutorial/raster.md @@ -85,7 +85,7 @@ If you already have a SparkSession (Wherobots, AWS EMR, Databricks), skip ahead ) sedona = SedonaContext.create(config) ``` - Replace `3.3` with the major.minor version of your Spark install (for example `sedona-spark-shaded-4.0_2.12`). + Replace `3.3` with the major.minor version of your Spark install (for example `sedona-spark-shaded-4.0_2.13`). You can also register Sedona by passing `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` to `spark-submit` or `spark-shell`. diff --git a/docs/tutorial/raster.zh.md b/docs/tutorial/raster.zh.md index 8eab3ed10b9..fa0ffa44971 100644 --- a/docs/tutorial/raster.zh.md +++ b/docs/tutorial/raster.zh.md @@ -85,7 +85,7 @@ Scala、Java、Python、R 等所有 Sedona 语言绑定都已支持栅格能力 ) sedona = SedonaContext.create(config) ``` - 请将 `sedona-spark-shaded-3.3` 中的 `3.3` 替换为对应的 Spark 主.次版本号，例如 `sedona-spark-shaded-4.0_2.12`。 + 请将 `sedona-spark-shaded-3.3` 中的 `3.3` 替换为对应的 Spark 主.次版本号，例如 `sedona-spark-shaded-4.0_2.13`。 你也可以通过给 `spark-submit` 或 `spark-shell` 传入 `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` 来注册 Sedona。 diff --git a/docs/tutorial/sql.md b/docs/tutorial/sql.md index eb0df79887e..26dfdb44903 100644 --- a/docs/tutorial/sql.md +++ b/docs/tutorial/sql.md @@ -107,7 +107,7 @@ You can add additional Spark runtime config to the config builder. For example, 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \ getOrCreate() ``` - If you are using a different Spark version, please replace the `3.3` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-4.0_2.12:{{ sedona.current_version }}`. + If you are using a different Spark version, please replace the `3.3` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-4.0_2.13:{{ sedona.current_version }}`. ## Initiate SedonaContext diff --git a/docs/tutorial/sql.zh.md b/docs/tutorial/sql.zh.md index 6c00808c451..339eb7328cc 100644 --- a/docs/tutorial/sql.zh.md +++ b/docs/tutorial/sql.zh.md @@ -107,7 +107,7 @@ SedonaSQL 详细 API 说明请参阅 [SedonaSQL API](../api/sql/Overview.md)。 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \ getOrCreate() ``` - 如使用其他 Spark 版本，请将 sedona-spark-shaded 包名中的 `3.3` 替换为对应的 Spark major.minor 版本，例如 `sedona-spark-shaded-4.0_2.12:{{ sedona.current_version }}`。 + 如使用其他 Spark 版本，请将 sedona-spark-shaded 包名中的 `3.3` 替换为对应的 Spark major.minor 版本，例如 `sedona-spark-shaded-4.0_2.13:{{ sedona.current_version }}`。 ## 初始化 SedonaContext From 2018574d71fcb8573d24967e28ef01d4f264cbbe Mon Sep 17 00:00:00 2001 From: Jia Yu Date: Fri, 28 Aug 2026 19:09:47 -0700 Subject: [PATCH 3/6] [GH-3297] Scope the Scala 2.12 build instructions to Spark 3.5 The compile tab read "Spark 3.5+ Scala 2.12" and told readers to substitute a higher Spark version, and the EMR note said "Spark 3.5+ and Scala 2.12". Neither holds: Spark 4.x publishes no Scala 2.12 artifacts, so following either instruction fails. mvn -pl spark/common -am -Dspark=4.0 -Dscala=2.12 -DskipTests compile => Could not resolve dependencies ... spark-core_2.12:jar:4.0.0, spark-sql_2.12:jar:4.0.0 Scope the Scala 2.12 tab to Spark 3.5 and say why. The Scala 2.13 tab keeps its "higher Spark versions" note, which is accurate for 3.5, 4.0 and 4.1. The EMR note now points Spark 4.0 and above at the 2.13 artifact. --- docs/setup/compile.md | 4 ++-- docs/setup/compile.zh.md | 4 ++-- docs/setup/emr.md | 2 +- docs/setup/emr.zh.md | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/setup/compile.md b/docs/setup/compile.md index cea28c5cb30..599d5088d8a 100644 --- a/docs/setup/compile.md +++ b/docs/setup/compile.md @@ -55,11 +55,11 @@ User can specify `-Dspark` and `-Dscala` command line options to compile with di * `-Dspark`: `{major}.{minor}`: For example, specify `-Dspark=3.5` to build for Spark 3.5. * `-Dscala`: `2.12` or `2.13` -=== "Spark 3.5+ Scala 2.12" +=== "Spark 3.5 Scala 2.12" ``` mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.12 ``` - Please replace `3.5` with Spark major.minor version when building for higher Spark versions. + Scala 2.12 is only available for Spark 3.5. Spark 4.0 and above are built with Scala 2.13 only. === "Spark 3.5+ Scala 2.13" ``` mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.13 diff --git a/docs/setup/compile.zh.md b/docs/setup/compile.zh.md index 9c1291ddb43..2a57da2a507 100644 --- a/docs/setup/compile.zh.md +++ b/docs/setup/compile.zh.md @@ -55,11 +55,11 @@ Sedona Scala/Java 代码是一个多模块项目，每个模块都是 Scala/Java * `-Dspark`：`{major}.{minor}`，例如 `-Dspark=3.5` 表示针对 Spark 3.5 编译。 * `-Dscala`：`2.12` 或 `2.13` -=== "Spark 3.5+ Scala 2.12" +=== "Spark 3.5 Scala 2.12" ``` mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.12 ``` - 若要针对更高 Spark 版本编译，请将 `3.5` 替换为对应的 Spark major.minor 版本。 + Scala 2.12 仅适用于 Spark 3.5。Spark 4.0 及以上仅提供 Scala 2.13 构建。 === "Spark 3.5+ Scala 2.13" ``` mvn clean install -DskipTests -Dspark=3.5 -Dscala=2.13 diff --git a/docs/setup/emr.md b/docs/setup/emr.md index 2d759b81495..2c99a2853e4 100644 --- a/docs/setup/emr.md +++ b/docs/setup/emr.md @@ -22,7 +22,7 @@ We recommend Sedona-1.3.1-incubating and above for EMR. In the tutorial, we use This tutorial is tested on EMR on EC2 with EMR Studio (notebooks). EMR on EC2 uses YARN to manage resources. !!!note - If you are using Spark 3.5+ and Scala 2.12, please use `sedona-spark-shaded-3.5_2.12`. Please pay attention to the Spark version postfix and Scala version postfix. + If you are using Spark 3.5 and Scala 2.12, please use `sedona-spark-shaded-3.5_2.12`. Spark 4.0 and above are built with Scala 2.13 only, so use `sedona-spark-shaded-4.0_2.13` there. Please pay attention to the Spark version postfix and Scala version postfix. ## Prepare initialization script diff --git a/docs/setup/emr.zh.md b/docs/setup/emr.zh.md index dc208ed59a6..ff0f8148566 100644 --- a/docs/setup/emr.zh.md +++ b/docs/setup/emr.zh.md @@ -22,7 +22,7 @@ 本教程在带 EMR Studio（notebook）的 EMR on EC2 上完成测试。EMR on EC2 使用 YARN 进行资源管理。 !!!note - 如果您使用 Spark 3.5+ 与 Scala 2.12，请使用 `sedona-spark-shaded-3.5_2.12`。请注意 Spark 与 Scala 的版本后缀。 + 如果您使用 Spark 3.5 与 Scala 2.12，请使用 `sedona-spark-shaded-3.5_2.12`。Spark 4.0 及以上仅提供 Scala 2.13 构建，请使用 `sedona-spark-shaded-4.0_2.13`。请注意 Spark 与 Scala 的版本后缀。 ## 准备初始化脚本 From 034099e78c46dce8f2074d98e4722103ca210450 Mon Sep 17 00:00:00 2001 From: Jia Yu Date: Fri, 28 Aug 2026 23:57:17 -0700 Subject: [PATCH 4/6] [GH-3297] Replace the stale Spark 3.3 artifacts in the setup docs Spark 3.3 was dropped in Sedona 1.8.0, but the setup and tutorial pages still told readers to pull sedona-spark-shaded-3.3_2.12 at the current Sedona version, an artifact that has not been published since 1.7.1. Dropping 3.4 made this worse in emr.md, whose note recommended 3.5_2.12 while the script below it downloaded 3.3_2.12. Point the 28 coordinates in emr, install-scala, install-python, sql-pure-sql, sql and raster (plus their Chinese mirrors) at 3.5_2.12, and update the surrounding "replace the 3.3" prose to match. Both artifacts resolve on Maven Central at 1.9.1. sql-pure-sql also carried a "Spark 3.3+ and Scala 2.12" tab with the same open-ended range that was corrected in compile.md and emr.md: Spark 4.x has no Scala 2.12 build, so the tab is now Spark 3.5 and points at 4.0_2.13 for later versions. The EMR walkthrough ran on EMR 6.9.0, which ships Spark 3.3. Move it to EMR 7.9.0; component versions are from the AWS EMR 7.x application versions table. The remaining 3.3 references are deliberate and kept: glue.md documents the Glue 4.0 fallback for users pinned to Sedona 1.7.1, and release-notes.md records a 1.4.1 snippet. --- docs/setup/emr.md | 6 +++--- docs/setup/emr.zh.md | 6 +++--- docs/setup/install-python.md | 4 ++-- docs/setup/install-python.zh.md | 4 ++-- docs/setup/install-scala.md | 10 +++++----- docs/setup/install-scala.zh.md | 10 +++++----- docs/tutorial/raster.md | 4 ++-- docs/tutorial/raster.zh.md | 4 ++-- docs/tutorial/sql-pure-sql.md | 6 +++--- docs/tutorial/sql-pure-sql.zh.md | 6 +++--- docs/tutorial/sql.md | 4 ++-- docs/tutorial/sql.zh.md | 4 ++-- 12 files changed, 34 insertions(+), 34 deletions(-) diff --git a/docs/setup/emr.md b/docs/setup/emr.md index 2c99a2853e4..854abf6a090 100644 --- a/docs/setup/emr.md +++ b/docs/setup/emr.md @@ -17,7 +17,7 @@ under the License. --> -We recommend Sedona-1.3.1-incubating and above for EMR. In the tutorial, we use AWS Elastic MapReduce (EMR) 6.9.0. It has the following applications installed: Hadoop 3.3.3, JupyterEnterpriseGateway 2.6.0, Livy 0.7.1, Spark 3.3.0. +In the tutorial, we use AWS Elastic MapReduce (EMR) 7.9.0. It has the following applications installed: Hadoop 3.4.1, JupyterEnterpriseGateway 2.6.0, Livy 0.8.0-incubating, Spark 3.5.5. Any EMR 7.x release works, since they all ship Spark 3.5. This tutorial is tested on EMR on EC2 with EMR Studio (notebooks). EMR on EC2 uses YARN to manage resources. @@ -35,7 +35,7 @@ In your S3 bucket, add a script that has the following content: sudo mkdir /jars # Download Sedona jar -sudo curl -o /jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.3_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar" +sudo curl -o /jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.5_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar" # Download GeoTools jar sudo curl -o /jars/geotools-wrapper-{{ sedona.current_geotools }}.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar" @@ -60,7 +60,7 @@ When you create an EMR cluster, in the software configuration, add the following { "Classification":"spark-defaults", "Properties":{ - "spark.yarn.dist.jars": "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar,/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", + "spark.yarn.dist.jars": "/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar,/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", "spark.serializer": "org.apache.spark.serializer.KryoSerializer", "spark.kryo.registrator": "org.apache.sedona.core.serde.SedonaKryoRegistrator", "spark.sql.extensions": "org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions" diff --git a/docs/setup/emr.zh.md b/docs/setup/emr.zh.md index ff0f8148566..e7b0a6aa93e 100644 --- a/docs/setup/emr.zh.md +++ b/docs/setup/emr.zh.md @@ -17,7 +17,7 @@ under the License. --> -在 EMR 上推荐使用 Sedona-1.3.1-incubating 及以上版本。本教程使用 AWS Elastic MapReduce (EMR) 6.9.0，已预装以下应用：Hadoop 3.3.3、JupyterEnterpriseGateway 2.6.0、Livy 0.7.1、Spark 3.3.0。 +本教程使用 AWS Elastic MapReduce (EMR) 7.9.0，已预装以下应用：Hadoop 3.4.1、JupyterEnterpriseGateway 2.6.0、Livy 0.8.0-incubating、Spark 3.5.5。任何 EMR 7.x 版本均可，因为它们都搭载 Spark 3.5。 本教程在带 EMR Studio（notebook）的 EMR on EC2 上完成测试。EMR on EC2 使用 YARN 进行资源管理。 @@ -35,7 +35,7 @@ sudo mkdir /jars # 下载 Sedona jar -sudo curl -o /jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.3_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar" +sudo curl -o /jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.5_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar" # 下载 GeoTools jar sudo curl -o /jars/geotools-wrapper-{{ sedona.current_geotools }}.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar" @@ -60,7 +60,7 @@ sudo python3 -m pip install attrs matplotlib descartes apache-sedona=={{ sedona. { "Classification":"spark-defaults", "Properties":{ - "spark.yarn.dist.jars": "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar,/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", + "spark.yarn.dist.jars": "/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar,/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", "spark.serializer": "org.apache.spark.serializer.KryoSerializer", "spark.kryo.registrator": "org.apache.sedona.core.serde.SedonaKryoRegistrator", "spark.sql.extensions": "org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions" diff --git a/docs/setup/install-python.md b/docs/setup/install-python.md index c754780ac25..6b780488017 100644 --- a/docs/setup/install-python.md +++ b/docs/setup/install-python.md @@ -67,7 +67,7 @@ config = ( SedonaContext.builder() .config( "spark.jars.packages", - "org.apache.sedona:sedona-spark-3.3_2.12:{{ sedona.current_version }}," + "org.apache.sedona:sedona-spark-3.5_2.12:{{ sedona.current_version }}," "org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}", ) .config( @@ -94,7 +94,7 @@ spark = ( .config("spark.kryo.registrator", SedonaKryoRegistrator.getName) .config( "spark.jars.packages", - "org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }}," + "org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }}," "org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}", ) .getOrCreate() diff --git a/docs/setup/install-python.zh.md b/docs/setup/install-python.zh.md index af03ba19dfe..c102d6f5a36 100644 --- a/docs/setup/install-python.zh.md +++ b/docs/setup/install-python.zh.md @@ -68,7 +68,7 @@ config = ( SedonaContext.builder() .config( "spark.jars.packages", - "org.apache.sedona:sedona-spark-3.3_2.12:{{ sedona.current_version }}," + "org.apache.sedona:sedona-spark-3.5_2.12:{{ sedona.current_version }}," "org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}", ) .config( @@ -95,7 +95,7 @@ spark = ( .config("spark.kryo.registrator", SedonaKryoRegistrator.getName) .config( "spark.jars.packages", - "org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }}," + "org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }}," "org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}", ) .getOrCreate() diff --git a/docs/setup/install-scala.md b/docs/setup/install-scala.md index 4cb7d53e3b3..928363dce6b 100644 --- a/docs/setup/install-scala.md +++ b/docs/setup/install-scala.md @@ -40,12 +40,12 @@ Please refer to [Sedona Maven Central coordinates](maven-coordinates.md) to sele * Local mode: test Sedona without setting up a cluster ``` - ./bin/spark-shell --packages org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} + ./bin/spark-shell --packages org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} ``` * Cluster mode: you need to specify Spark Master IP ``` - ./bin/spark-shell --master spark://localhost:7077 --packages org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} + ./bin/spark-shell --master spark://localhost:7077 --packages org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} ``` ### Download Sedona jar manually @@ -61,16 +61,16 @@ Please refer to [Sedona Maven Central coordinates](maven-coordinates.md) to sele ./bin/spark-shell --jars /Path/To/SedonaJars.jar ``` -Please use jars with Spark major.minor versions in the filename, such as `sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}`. +Please use jars with Spark major.minor versions in the filename, such as `sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}`. * Local mode: test Sedona without setting up a cluster ``` - ./bin/spark-shell --jars /path/to/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar + ./bin/spark-shell --jars /path/to/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar ``` * Cluster mode: you need to specify Spark Master IP ``` - ./bin/spark-shell --master spark://localhost:7077 --jars /path/to/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar + ./bin/spark-shell --master spark://localhost:7077 --jars /path/to/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar ``` ## Spark SQL shell diff --git a/docs/setup/install-scala.zh.md b/docs/setup/install-scala.zh.md index acec355b750..b29d3b0bc16 100644 --- a/docs/setup/install-scala.zh.md +++ b/docs/setup/install-scala.zh.md @@ -40,12 +40,12 @@ * 本地模式：无需搭建集群即可测试 Sedona ``` - ./bin/spark-shell --packages org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} + ./bin/spark-shell --packages org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} ``` * 集群模式：需要指定 Spark Master IP ``` - ./bin/spark-shell --master spark://localhost:7077 --packages org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} + ./bin/spark-shell --master spark://localhost:7077 --packages org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }},org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} ``` ### 手动下载 Sedona jar @@ -61,16 +61,16 @@ ./bin/spark-shell --jars /Path/To/SedonaJars.jar ``` -请使用文件名中包含 Spark major.minor 版本号的 jar，例如 `sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}`。 +请使用文件名中包含 Spark major.minor 版本号的 jar，例如 `sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}`。 * 本地模式：无需搭建集群即可测试 Sedona ``` - ./bin/spark-shell --jars /path/to/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar + ./bin/spark-shell --jars /path/to/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar ``` * 集群模式：需要指定 Spark Master IP ``` - ./bin/spark-shell --master spark://localhost:7077 --jars /path/to/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar + ./bin/spark-shell --master spark://localhost:7077 --jars /path/to/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar,/path/to/geotools-wrapper-{{ sedona.current_geotools }}.jar ``` ## Spark SQL shell diff --git a/docs/tutorial/raster.md b/docs/tutorial/raster.md index e930922b493..2a3e4e40840 100644 --- a/docs/tutorial/raster.md +++ b/docs/tutorial/raster.md @@ -78,14 +78,14 @@ If you already have a SparkSession (Wherobots, AWS EMR, Databricks), skip ahead SedonaContext.builder() .config( "spark.jars.packages", - "org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }}," + "org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }}," "org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}", ) .getOrCreate() ) sedona = SedonaContext.create(config) ``` - Replace `3.3` with the major.minor version of your Spark install (for example `sedona-spark-shaded-4.0_2.13`). + Replace `3.5` with the major.minor version of your Spark install (for example `sedona-spark-shaded-4.0_2.13`). You can also register Sedona by passing `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` to `spark-submit` or `spark-shell`. diff --git a/docs/tutorial/raster.zh.md b/docs/tutorial/raster.zh.md index fa0ffa44971..f95cda35fe5 100644 --- a/docs/tutorial/raster.zh.md +++ b/docs/tutorial/raster.zh.md @@ -78,14 +78,14 @@ Scala、Java、Python、R 等所有 Sedona 语言绑定都已支持栅格能力 SedonaContext.builder() .config( "spark.jars.packages", - "org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }}," + "org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }}," "org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}", ) .getOrCreate() ) sedona = SedonaContext.create(config) ``` - 请将 `sedona-spark-shaded-3.3` 中的 `3.3` 替换为对应的 Spark 主.次版本号，例如 `sedona-spark-shaded-4.0_2.13`。 + 请将 `sedona-spark-shaded-3.5` 中的 `3.5` 替换为对应的 Spark 主.次版本号，例如 `sedona-spark-shaded-4.0_2.13`。 你也可以通过给 `spark-submit` 或 `spark-shell` 传入 `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` 来注册 Sedona。 diff --git a/docs/tutorial/sql-pure-sql.md b/docs/tutorial/sql-pure-sql.md index 9edd3a157a1..0ddc3e45220 100644 --- a/docs/tutorial/sql-pure-sql.md +++ b/docs/tutorial/sql-pure-sql.md @@ -27,16 +27,16 @@ Start `spark-sql` as following (replace `` with actual version like `{{ !!! abstract "Run spark-sql with Apache Sedona" - === "Spark 3.3+ and Scala 2.12" + === "Spark 3.5 and Scala 2.12" ```sh - spark-sql --packages org.apache.sedona:sedona-spark-shaded-3.3_2.12:,org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} \ + spark-sql --packages org.apache.sedona:sedona-spark-shaded-3.5_2.12:,org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf spark.kryo.registrator=org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator \ --conf spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions ``` - Please replace the `3.3` in artifact names with the corresponding major.minor version of Spark. + Spark 4.0 and above are built with Scala 2.13 only; for those, replace the artifact name with `sedona-spark-shaded-4.0_2.13`. This will register all Sedona types, functions and optimizations in SedonaSQL and SedonaViz. diff --git a/docs/tutorial/sql-pure-sql.zh.md b/docs/tutorial/sql-pure-sql.zh.md index de4b9b6dc3d..523b3b38be9 100644 --- a/docs/tutorial/sql-pure-sql.zh.md +++ b/docs/tutorial/sql-pure-sql.zh.md @@ -27,16 +27,16 @@ SedonaSQL 支持 SQL/MM Part3 空间 SQL 标准。SedonaSQL 详细的 API 说明 !!! abstract "使用 Apache Sedona 启动 spark-sql" - === "Spark 3.3+ 与 Scala 2.12" + === "Spark 3.5 与 Scala 2.12" ```sh - spark-sql --packages org.apache.sedona:sedona-spark-shaded-3.3_2.12:,org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} \ + spark-sql --packages org.apache.sedona:sedona-spark-shaded-3.5_2.12:,org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }} \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf spark.kryo.registrator=org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator \ --conf spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions ``` - 请将 artifact 名称中的 `3.3` 替换为对应的 Spark major.minor 版本。 + Spark 4.0 及以上仅提供 Scala 2.13 构建，请将 artifact 名称替换为 `sedona-spark-shaded-4.0_2.13`。 这会注册 SedonaSQL 与 SedonaViz 的全部类型、函数与优化规则。 diff --git a/docs/tutorial/sql.md b/docs/tutorial/sql.md index 26dfdb44903..b7213741b99 100644 --- a/docs/tutorial/sql.md +++ b/docs/tutorial/sql.md @@ -103,11 +103,11 @@ You can add additional Spark runtime config to the config builder. For example, config = SedonaContext.builder() .\ config('spark.jars.packages', - 'org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},' + 'org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }},' 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \ getOrCreate() ``` - If you are using a different Spark version, please replace the `3.3` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-4.0_2.13:{{ sedona.current_version }}`. + If you are using a different Spark version, please replace the `3.5` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-4.0_2.13:{{ sedona.current_version }}`. ## Initiate SedonaContext diff --git a/docs/tutorial/sql.zh.md b/docs/tutorial/sql.zh.md index 339eb7328cc..1c2b3122d2e 100644 --- a/docs/tutorial/sql.zh.md +++ b/docs/tutorial/sql.zh.md @@ -103,11 +103,11 @@ SedonaSQL 详细 API 说明请参阅 [SedonaSQL API](../api/sql/Overview.md)。 config = SedonaContext.builder() .\ config('spark.jars.packages', - 'org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},' + 'org.apache.sedona:sedona-spark-shaded-3.5_2.12:{{ sedona.current_version }},' 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \ getOrCreate() ``` - 如使用其他 Spark 版本，请将 sedona-spark-shaded 包名中的 `3.3` 替换为对应的 Spark major.minor 版本，例如 `sedona-spark-shaded-4.0_2.13:{{ sedona.current_version }}`。 + 如使用其他 Spark 版本，请将 sedona-spark-shaded 包名中的 `3.5` 替换为对应的 Spark major.minor 版本，例如 `sedona-spark-shaded-4.0_2.13:{{ sedona.current_version }}`。 ## 初始化 SedonaContext From ee0b0a58eb00080cbe7b51e7c5707c32d8c1c98b Mon Sep 17 00:00:00 2001 From: Jia Yu Date: Sat, 29 Aug 2026 21:13:30 -0700 Subject: [PATCH 5/6] [GH-3297] Apply the docker Spark guard only when building from source The guard added earlier ran before the published-version branch, so it also rejected rebuilds of historical images: docker/build.sh 3.4.1 1.9.0 local 33.5 => Unsupported Spark version: 3.4.1... That build is legitimate. It never compiles: install-sedona.sh downloads sedona-spark-shaded-3.4_2.13 from Maven Central, which is published for 1.9.0. Only the SEDONA_VERSION=latest branch runs mvn and so can silently fall back to the default Spark profile. Move the check, and the SEDONA_SPARK_VERSION it guards, into that branch. SEDONA_SPARK_VERSION had no other reader. Verified against a stubbed mvn and docker: - 3.4.1 1.9.0, 3.3.0 1.7.1, 3.5.8 1.9.1 -> exit 0, reaches image build - 3.4.1 latest, 3.3.0 latest -> exit 1, rejected - 3.5.8/4.0.1/4.1.1 latest -> exit 0, compiles with -Dspark --- docker/build.sh | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/docker/build.sh b/docker/build.sh index 69f5b9c53b9..4aa1cc72355 100755 --- a/docker/build.sh +++ b/docker/build.sh @@ -22,19 +22,6 @@ SEDONA_VERSION=$2 BUILD_MODE=$3 GEOTOOLS_VERSION=${4:-auto} -SEDONA_SPARK_VERSION=${SPARK_VERSION:0:3} -# Reject Spark versions Sedona no longer builds for. Without this check an -# unsupported -Dspark value simply matches no Maven profile, so the build -# silently falls back to the default Spark version and produces jars that do -# not match the Spark image they are installed into. -case "$SEDONA_SPARK_VERSION" in - 3.5 | 4.0 | 4.1) ;; - *) - echo "Unsupported Spark version: ${SPARK_VERSION}. Sedona supports Spark 3.5, 4.0 and 4.1." >&2 - exit 1 - ;; -esac - # Function to compare two version numbers version_gt() { # Compare two version numbers @@ -86,6 +73,20 @@ if [ "$SEDONA_VERSION" = "latest" ]; then fi echo "Using latest geotools-wrapper version: $GEOTOOLS_WRAPPER_VERSION" + # Building master resolves Sedona's Spark profile from -Dspark. An unsupported + # value matches no profile, so Maven would fall back to the default Spark + # version and produce jars that do not match the Spark image they are + # installed into. Only this branch compiles; images for a published Sedona + # version take the jar from Maven Central and can still target older Spark. + SEDONA_SPARK_VERSION=${SPARK_VERSION:0:3} + case "$SEDONA_SPARK_VERSION" in + 3.5 | 4.0 | 4.1) ;; + *) + echo "Cannot build Sedona master against Spark ${SPARK_VERSION}. Supported: 3.5, 4.0, 4.1." >&2 + exit 1 + ;; + esac + # The compilation must take place outside Docker to avoid unnecessary maven packages mvn clean install -DskipTests -Dspark="${SEDONA_SPARK_VERSION}" -Dscala=2.13 else From ed3e76a0ee6488a78e44b6ab2c78367718f8ebf8 Mon Sep 17 00:00:00 2001 From: Jia Yu Date: Sat, 29 Aug 2026 21:40:47 -0700 Subject: [PATCH 6/6] [GH-3297] Match the artifact's Spark suffix instead of naming 4.0 The Spark 4 guidance sent every "4.0 and above" reader to sedona-spark-shaded-4.0_2.13, but 4.1 is supported and publishes its own sedona-spark-shaded-4.1_2.13. Keep the Scala 2.13 point and tell readers to match the artifact's Spark major.minor suffix, naming both 4.0 and 4.1. emr.md carried the same sentence from the same edit, so it is fixed too. raster.md and sql.md already said to substitute the major.minor version and only used 4.0_2.13 as an illustration, so they stand. Checked every Sedona spark coordinate named under docs/ against Maven Central. All resolve at 1.9.1 except the deliberately historical ones, which resolve at the Sedona version their page pins: sedona-spark-shaded-3.3_2.12 at 1.7.1 (glue.md's Glue 4.0 fallback), sedona-spark-shaded-3.4_2.12 at 1.5.1 and 1.6.x (the Fabric and Synapse walkthroughs), plus release-notes.md and a legacy notebook transcript. --- docs/setup/emr.md | 2 +- docs/setup/emr.zh.md | 2 +- docs/tutorial/sql-pure-sql.md | 2 +- docs/tutorial/sql-pure-sql.zh.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/setup/emr.md b/docs/setup/emr.md index 854abf6a090..e39ac0ab8a5 100644 --- a/docs/setup/emr.md +++ b/docs/setup/emr.md @@ -22,7 +22,7 @@ In the tutorial, we use AWS Elastic MapReduce (EMR) 7.9.0. It has the following This tutorial is tested on EMR on EC2 with EMR Studio (notebooks). EMR on EC2 uses YARN to manage resources. !!!note - If you are using Spark 3.5 and Scala 2.12, please use `sedona-spark-shaded-3.5_2.12`. Spark 4.0 and above are built with Scala 2.13 only, so use `sedona-spark-shaded-4.0_2.13` there. Please pay attention to the Spark version postfix and Scala version postfix. + Use the artifact whose Spark major.minor version matches your cluster: `sedona-spark-shaded-3.5_2.12` for Spark 3.5, `sedona-spark-shaded-4.0_2.13` for Spark 4.0, `sedona-spark-shaded-4.1_2.13` for Spark 4.1. Spark 4.0 and above are built with Scala 2.13 only. ## Prepare initialization script diff --git a/docs/setup/emr.zh.md b/docs/setup/emr.zh.md index e7b0a6aa93e..a9adea4d41c 100644 --- a/docs/setup/emr.zh.md +++ b/docs/setup/emr.zh.md @@ -22,7 +22,7 @@ 本教程在带 EMR Studio（notebook）的 EMR on EC2 上完成测试。EMR on EC2 使用 YARN 进行资源管理。 !!!note - 如果您使用 Spark 3.5 与 Scala 2.12，请使用 `sedona-spark-shaded-3.5_2.12`。Spark 4.0 及以上仅提供 Scala 2.13 构建，请使用 `sedona-spark-shaded-4.0_2.13`。请注意 Spark 与 Scala 的版本后缀。 + 请使用 Spark 主.次版本与集群一致的 artifact：Spark 3.5 用 `sedona-spark-shaded-3.5_2.12`，Spark 4.0 用 `sedona-spark-shaded-4.0_2.13`，Spark 4.1 用 `sedona-spark-shaded-4.1_2.13`。Spark 4.0 及以上仅提供 Scala 2.13 构建。 ## 准备初始化脚本 diff --git a/docs/tutorial/sql-pure-sql.md b/docs/tutorial/sql-pure-sql.md index 0ddc3e45220..026c9e0b1c3 100644 --- a/docs/tutorial/sql-pure-sql.md +++ b/docs/tutorial/sql-pure-sql.md @@ -36,7 +36,7 @@ Start `spark-sql` as following (replace `` with actual version like `{{ --conf spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions ``` - Spark 4.0 and above are built with Scala 2.13 only; for those, replace the artifact name with `sedona-spark-shaded-4.0_2.13`. + Spark 4.0 and above are built with Scala 2.13 only. Replace the artifact with the one carrying your Spark major.minor version, such as `sedona-spark-shaded-4.0_2.13` for Spark 4.0 or `sedona-spark-shaded-4.1_2.13` for Spark 4.1. This will register all Sedona types, functions and optimizations in SedonaSQL and SedonaViz. diff --git a/docs/tutorial/sql-pure-sql.zh.md b/docs/tutorial/sql-pure-sql.zh.md index 523b3b38be9..118b1a22ff6 100644 --- a/docs/tutorial/sql-pure-sql.zh.md +++ b/docs/tutorial/sql-pure-sql.zh.md @@ -36,7 +36,7 @@ SedonaSQL 支持 SQL/MM Part3 空间 SQL 标准。SedonaSQL 详细的 API 说明 --conf spark.sql.extensions=org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions ``` - Spark 4.0 及以上仅提供 Scala 2.13 构建，请将 artifact 名称替换为 `sedona-spark-shaded-4.0_2.13`。 + Spark 4.0 及以上仅提供 Scala 2.13 构建。请替换为与您的 Spark 主.次版本一致的 artifact，例如 Spark 4.0 用 `sedona-spark-shaded-4.0_2.13`，Spark 4.1 用 `sedona-spark-shaded-4.1_2.13`。 这会注册 SedonaSQL 与 SedonaViz 的全部类型、函数与优化规则。