Skip to content

[GH-3362] Move Shapefile and GeoParquet metadata readers into Spark common - #3363

Open
jiayuasu wants to merge 1 commit into
apache:masterfrom
jiayuasu:refactor/datasource-common
Open

[GH-3362] Move Shapefile and GeoParquet metadata readers into Spark common#3363
jiayuasu wants to merge 1 commit into
apache:masterfrom
jiayuasu:refactor/datasource-common

Conversation

@jiayuasu

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Yes, I have read the Contributor Rules and Contributor Development Guide.

Is this PR related to a ticket?

Yes, and the PR name follows the format [GH-XXX] my subject. Closes #3362.

What changes were proposed in this PR?

Move the identical Shapefile reader and GeoParquet metadata reader implementations from the Spark 3.5, 4.0, and 4.1 modules into spark/common:

  • Move 14 production Scala files and two existing test suites without changing their contents.
  • Move both data-source registrations into the common service descriptor.
  • Remove the duplicate versioned copies, eliminating 5,410 lines of duplicate source and test code.

Packages, public APIs, dependencies, and behavior remain unchanged. This is separate from the GeoPackage refactor in #3361; GeoPackage files are unchanged here.

How was this patch tested?

  • Clean spark-shaded reactor builds for Spark 3.5 / Scala 2.12, Spark 4.0 / Scala 2.13, and Spark 4.1 / Scala 2.13:
    mvn -q clean package -pl spark-shaded -am -Dspark=<version> -Dscala=<version> -DskipTests -Dmaven.javadoc.skip=true
  • Ran ShapefileTests (34 tests), GeoParquetMetadataTests (7 tests), and geoparquetIOTests (54 tests) sequentially on each Spark version. All 95 tests passed both with common-module production classes only and with the freshly built shaded JAR, for six successful runs.
  • Checked each profile's artifacts: the moved classes are in the common JAR, absent from the versioned JAR, and each provider appears exactly once in the shaded service descriptor.
  • Verified all moved Scala files are byte-for-byte identical to the original copies in all three Spark modules.
  • Commit hooks and git diff --check passed.

Did this PR include necessary documentation updates?

No, this PR does not affect any public API so no need to change the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move Shapefile and GeoParquet metadata readers into Spark common

1 participant