[905] Merge ServiceLoader registrations when shading xtable-utilities - #906
Merged
the-other-tim-brown merged 1 commit intoAug 25, 2026
Merged
Conversation
The shade plugin had no ServicesResourceTransformer, so every META-INF/services/* file was last-one-wins: whichever artifact shade processed last silently overwrote the others. Two visible consequences. The bundled jar could not read Delta sources at all, because Delta's DataSourceRegister entry lost to Spark's and sourceFormat: DELTA died with ClassNotFoundException: delta.DefaultSource despite the class being present. And only one Hadoop FileSystem registration survived, which is why xtable-hadoop-defaults.xml has to name fs.file.impl, fs.s3a.impl and the GCS classes by hand. Closes apache#905.
the-other-tim-brown
approved these changes
Aug 25, 2026
the-other-tim-brown
left a comment
Contributor
There was a problem hiding this comment.
Thanks @slachiewicz!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #905.
What is the purpose of the pull request
The shade plugin in
xtable-utilitieshad noServicesResourceTransformer, so everyMETA-INF/services/*file in the bundled jar was last-one-wins: whichever artifact shadeprocessed last silently overwrote all the others' registrations.
Brief change log
ServicesResourceTransformerto the shade transformer list. One line, no sourcechanges.
Why it matters
Two user-visible bugs share this single cause.
The bundled jar could not read Delta sources at all. Delta's
org.apache.spark.sql.delta.sources.DeltaDataSourcelost itsDataSourceRegisterentry toSpark's copy, so
sourceFormat: DELTAfailed withClassNotFoundException: delta.DefaultSource— despite the class and 1,616io/delta/*entries being present in the jar. Delta as a target was unaffected, which is why this went
unnoticed.
Only one Hadoop
FileSystemregistration survived. That is very likely whyxtable-utilities/src/main/resources/xtable-hadoop-defaults.xmlhas to namefs.file.impl,fs.s3a.impl,fs.gs.impland the GCSAbstractFileSystemclasses by hand — a bareFsShelloff the jar reportsNo FileSystem for scheme "file"without them. Those defaultsare left in place here; this PR does not attempt to remove them.
Verify this pull request
Registrations recovered in the bundled jar:
DataSourceRegisterdelta entriesorg.apache.hadoop.fs.FileSystementriesMETA-INF/services/filesEnd to end against a real S3 bucket, same fresh Delta table, same command, no workaround on the
classpath:
Verified:
java -jar xtable-utilities-*-bundled.jar --datasetConfig <delta source> --hadoopConfig <s3a>→Sync is successful, against live S3 on Corretto 11.Worth a reviewer's eye: merging service files is strictly more registration than before, so an
engine or filesystem that was previously masked is now visible to
ServiceLoader. I did notfind a case where that is unwanted, but I have only exercised the Delta, Iceberg, S3A and ABFS
paths.
Note for whoever merges: #897 also edits
xtable-utilities/pom.xml, in theartifactSetandfilterssections rather thantransformers, so the two should not conflict textually.This change was created with AI assistance.