Skip to content

[905] Merge ServiceLoader registrations when shading xtable-utilities - #906

Merged
the-other-tim-brown merged 1 commit into
apache:mainfrom
slachiewicz:fix-services-transformer
Aug 25, 2026
Merged

[905] Merge ServiceLoader registrations when shading xtable-utilities#906
the-other-tim-brown merged 1 commit into
apache:mainfrom
slachiewicz:fix-services-transformer

Conversation

@slachiewicz

Copy link
Copy Markdown
Member

Closes #905.

What is the purpose of the pull request

The shade plugin in xtable-utilities had no ServicesResourceTransformer, so every
META-INF/services/* file in the bundled jar was last-one-wins: whichever artifact shade
processed last silently overwrote all the others' registrations.

Brief change log

  • Added ServicesResourceTransformer to the shade transformer list. One line, no source
    changes.

Why it matters

Two user-visible bugs share this single cause.

The bundled jar could not read Delta sources at all. Delta's
org.apache.spark.sql.delta.sources.DeltaDataSource lost its DataSourceRegister entry to
Spark's copy, so sourceFormat: DELTA failed with
ClassNotFoundException: delta.DefaultSource — despite the class and 1,616 io/delta/*
entries being present in the jar. Delta as a target was unaffected, which is why this went
unnoticed.

Only one Hadoop FileSystem registration survived. That is very likely why
xtable-utilities/src/main/resources/xtable-hadoop-defaults.xml has to name fs.file.impl,
fs.s3a.impl, fs.gs.impl and the GCS AbstractFileSystem classes by hand — a bare
FsShell off the jar reports No FileSystem for scheme "file" without them. Those defaults
are left in place here; this PR does not attempt to remove them.

Verify this pull request

Registrations recovered in the bundled jar:

before after
DataSourceRegister delta entries 0 1
org.apache.hadoop.fs.FileSystem entries 5 14
total META-INF/services/ files 135 141

End to end against a real S3 bucket, same fresh Delta table, same command, no workaround on the
classpath:

# before: exit 0, but
org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND]
  Failed to find the data source: delta
# 0 metadata files written

# after:
INFO ConversionController:215 - Sync is successful for the following formats ICEBERG
# 6 metadata files written

Verified: java -jar xtable-utilities-*-bundled.jar --datasetConfig <delta source> --hadoopConfig <s3a>Sync is successful, against live S3 on Corretto 11.

Worth a reviewer's eye: merging service files is strictly more registration than before, so an
engine or filesystem that was previously masked is now visible to ServiceLoader. I did not
find a case where that is unwanted, but I have only exercised the Delta, Iceberg, S3A and ABFS
paths.

Note for whoever merges: #897 also edits xtable-utilities/pom.xml, in the artifactSet and
filters sections rather than transformers, so the two should not conflict textually.

This change was created with AI assistance.

The shade plugin had no ServicesResourceTransformer, so every
META-INF/services/* file was last-one-wins: whichever artifact shade
processed last silently overwrote the others.

Two visible consequences. The bundled jar could not read Delta sources
at all, because Delta's DataSourceRegister entry lost to Spark's and
sourceFormat: DELTA died with ClassNotFoundException: delta.DefaultSource
despite the class being present. And only one Hadoop FileSystem
registration survived, which is why xtable-hadoop-defaults.xml has to
name fs.file.impl, fs.s3a.impl and the GCS classes by hand.

Closes apache#905.

@the-other-tim-brown the-other-tim-brown left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @slachiewicz!

@the-other-tim-brown
the-other-tim-brown merged commit fdf9ac2 into apache:main Aug 25, 2026
5 checks passed
@slachiewicz
slachiewicz deleted the fix-services-transformer branch August 26, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bundled jar cannot read Delta sources: shade drops Delta's DataSourceRegister entry

2 participants