[902] Move the build to Spark 3.5 and Delta 3.3

[slachiewicz]

Part of #902, which folds the skipped Spark 3.5 step into the Spark 4.0 work. This is that step on its own: Spark 3.5 and Delta 3.3 with Scala still on 2.12 and the compiler target still Java 8, so the Delta 2.4 to 3.x jump can be reviewed apart from the Scala 2.13 and Java 17 flip that follows.

The branch is rebased onto #912, which carries Delta's nested field ids into the converted schema, and holds five commits of its own. It no longer carries the commits it once borrowed: #910 has merged, and #912's commit is now its base.

What is the purpose of the pull request

Moves the build to spark.version 3.5.9 and delta.version 3.3.3, and ports the Delta call sites that 3.x moved. Hudi 1.2.0, Iceberg 1.9.2 and Paimon 1.3.1 all publish their 3.5 variants, so no engine version changes. delta-spark_2.12:3.3.3 is still class-file 52, so Java 8 remains a valid target and CI stays on JDK 11.

Brief change log

• Renamed io.delta:delta-core_ to io.delta:delta-spark_, which Delta did in 3.0, across the four consuming modules, the xtable-utilities shade include, the demo dependency scripts and two docs snippets that still named Delta 2.x.
• AddFile gained baseRowId, defaultRowCommitVersion and clusteringProvider, so its constructor takes 11 arguments; the sync leaves all three unset.
• DeltaLog.getSnapshotAt now takes a catalog table, and ensureLogDirectoryExist is createLogDirectoriesIfNotExists.

Verify this pull request

Three test level consequences of the Delta jump, each in its own commit:

• reuseMetadataDisabledReconstructsSnapshotPerCommit no longer asserts that getChangeLogFiles goes uncalled. As of Delta 3.x DeltaLog.getChanges delegates to it, so the spy cannot tell that call apart from one DeltaIncrementalChangesState makes. The snapshot per commit assertion it shares the test with is unchanged.
• ITDeltaDeleteVectorConvert reloads its table handle after enabling deletion vectors, since Delta 3.x plans a merge against the snapshot the handle was resolved on and exposes the row_index metadata column only where deletion vectors are readable (DeltaParquetFileFormat.scala:255-262), and it keeps its upserts on the rewrite path, since Delta 3.x persists deletion vectors for MERGE as well as DELETE.
• The column mapped table in TestDeltaHelper enables IcebergCompatV2. Delta 3.x writes parquet field ids for top level columns but assigns none to a map key, a map value or a list element, and Iceberg reads a file carrying any id by id alone, so without it those children cannot be resolved. Delta to Iceberg conversion loses the field ids of map and list children under column mapping #911 and [911] Carry Delta's nested field ids into the converted schema #912 cover that, and this branch is where the fix is exercised end to end, by ITConversionController#testColumnMappingEnabledDeltaToIceberg.

ITConversionController#testColumnMappingWithoutIcebergCompatIsNotReadableAsIceberg pins the case that carrying ids cannot fix. It builds a column mapped table without IcebergCompatV2, syncs it to Iceberg, asserts the sync reports SUCCESS, and asserts that reading the result throws. That is the current behaviour rather than the desired one, so if the project would rather refuse the sync or warn at sync time, that test is the place that changes. #911 has the protocol detail behind it.

Verification, on JDK 11 with the Maven wrapper:

• After the rebase, ./mvnw -pl xtable-core -am -DskipITs test runs 495 unit tests, all green.
• The integration suites this upgrade touches were run before the rebase and were green: ITConversionController, ITDeltaConversionSource, ITDeltaDeleteVectorConvert and ITHudiConversionSource. They have not been re-run since the rebase.

Nine further errors seen locally before #910 merged, all NoSuchElementException: No value present in Option from scheduleCompaction on MERGE_ON_READ tables, were an artefact of a non-UTC developer machine rather than of this change. #909 has the root cause and #910 the fix, now in main.

This change was created with AI assistance.