[911] Carry Delta's nested field ids into the converted schema - #912
Draft
slachiewicz wants to merge 1 commit into
Draft
[911] Carry Delta's nested field ids into the converted schema#912slachiewicz wants to merge 1 commit into
slachiewicz wants to merge 1 commit into
Conversation
This was referenced Aug 23, 2026
Delta assigns column mapping ids to struct fields only, so the key, value and element of a collection have no id of their own. When IcebergCompatV2 is enabled Delta does assign them and records them per field under delta.columnMapping.nested.ids, keyed by the path the child takes in the parquet file. Read those and put them on the converted fields, which the Iceberg target already prefers over the ids it generates itself.
slachiewicz
force-pushed
the
delta-nested-field-ids
branch
from
August 25, 2026 22:56
976bba2 to
b05ac71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #911.
What is the purpose of the pull request
Delta assigns column mapping IDs to struct fields only, so a map key, a map value, and a list element have no ID of their own. When
delta.enableIcebergCompatV2is set, Delta assigns them, writes them into the Parquet file, and records them on the enclosing field underdelta.columnMapping.nested.ids, keyed by the path the child takes in the file, such ascol-1234.key. This change reads that metadata and puts the IDs on the converted fields.It matters because Delta 3.x writes Parquet field IDs where Delta 2.4 writes none, and Iceberg applies a name mapping only to files that carry no IDs at all. Once any ID is present, the nested children of a collection must resolve by ID too, and they have none. For the footers from both Delta versions and the failure that follows, see #911.
Brief change log
DeltaSchemaExtractorthreads a field'sdelta.columnMapping.nested.idsmetadata through the conversion and assigns the IDs it holds to the map key, map value, and list element fields.IcebergSchemaExtractorneeds no change: it already prefers a field's own ID over one it generates..gitignoreignores.claude/, which otherwise failsrat:checkon a local build.Verify this pull request
This change added tests and can be verified as follows:
TestDeltaSchemaExtractor#testNestedFieldIdsInDeltaSchemacovers a map and a list that carry the metadata, and a field without it, which leaves the children unassigned as before../mvnw -pl xtable-core test -Dtest=TestDeltaSchemaExtractor,TestIcebergSchemaExtractorruns 21 tests, all green.The end-to-end effect is observable only on Delta 3.x, because Delta 2.4 has no IcebergCompatV2 and writes no Parquet field IDs, so the change is inert on this branch. Stacked under the Spark 3.5 and Delta 3.3 upgrade,
ITConversionController#testColumnMappingEnabledDeltaToIcebergpasses with IcebergCompatV2 enabled on the source table, where it fails on that branch without this change.This change was created with AI assistance.