Skip to content

[xtable-utilities] Remove duplicate AWS SDK v2 modules from shaded jar #923

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abdul-traya wants to merge 5 commits into
base: main
Choose a base branch
from
+73 −34
Open

[xtable-utilities] Remove duplicate AWS SDK v2 modules from shaded jar #923

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d76590e
[xtable-utilities] Remove duplicate AWS SDK v2 modules from shaded jar
abdul-traya Aug 26, 2026
6cd78d5
[xtable-utilities] Drop all standalone AWS SDK v2 modules, keep only …
abdul-traya Aug 26, 2026
dd0a6a7
fix
abdul-traya Aug 29, 2026
561a337
fix package
abdul-traya Aug 29, 2026
1a86525
fix catalog
abdul-traya Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 31 additions & 8 deletions xtable-aws/src/main/java/org/apache/xtable/glue/table/DeltaGlueCatalogTableBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import static org.apache.xtable.glue.GlueCatalogSyncClient.GLUE_EXTERNAL_TABLE_TYPE;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

Expand All @@ -48,6 +49,20 @@
/** Delta specific table operations for Glue catalog sync */
public class DeltaGlueCatalogTableBuilder implements CatalogTableBuilder<TableInput, Table> {

// Standard Hive-compatible Parquet classes (Delta's on-disk file format).
// A generic Hive-Metastore-compatible reader (e.g. a UC Glue Federation
// foreign catalog) validates every table's StorageDescriptor when listing a
// database and throws InvalidObjectException for the whole database on the
// first table with a null SerDe/format -- these must be set even though
// Athena's own Delta reader (which keys off the table_type/
// spark.sql.sources.provider parameters instead) tolerates them being null.
private static final String PARQUET_INPUT_FORMAT =
"org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat";
private static final String PARQUET_OUTPUT_FORMAT =
"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat";
private static final String PARQUET_SERDE_CLASS =
"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe";

private final GlueSchemaExtractor schemaExtractor;
private static final String tableFormat = TableFormat.DELTA;

Expand All @@ -68,11 +83,7 @@ public TableInput getCreateTableRequest(
.tableType(GLUE_EXTERNAL_TABLE_TYPE)
.parameters(getTableParameters())
.storageDescriptor(
StorageDescriptor.builder()
.columns(schemaExtractor.getNonPartitionColumns(table, columnsMap))
.location(table.getBasePath())
.serdeInfo(SerDeInfo.builder().parameters(getSerDeParameters(table)).build())
.build())
getStorageDescriptor(table, schemaExtractor.getNonPartitionColumns(table, columnsMap)))
.partitionKeys(schemaExtractor.getPartitionColumns(table, columnsMap))
.build();
}
Expand All @@ -90,13 +101,25 @@ public TableInput getUpdateTableRequest(
.tableType(GLUE_EXTERNAL_TABLE_TYPE)
.parameters(parameters)
.storageDescriptor(
catalogTable.storageDescriptor().toBuilder()
.columns(schemaExtractor.getNonPartitionColumns(table, columnsMap))
.build())
getStorageDescriptor(table, schemaExtractor.getNonPartitionColumns(table, columnsMap)))
.partitionKeys(schemaExtractor.getPartitionColumns(table, columnsMap))
.build();
}

private StorageDescriptor getStorageDescriptor(InternalTable table, List<Column> columns) {
return StorageDescriptor.builder()
.columns(columns)
.location(table.getBasePath())
.inputFormat(PARQUET_INPUT_FORMAT)
.outputFormat(PARQUET_OUTPUT_FORMAT)
.serdeInfo(
SerDeInfo.builder()
.serializationLibrary(PARQUET_SERDE_CLASS)
.parameters(getSerDeParameters(table))
.build())
.build();
}

@VisibleForTesting
Map<String, String> getTableParameters() {
Map<String, String> parameters = new HashMap<>();
Expand Down
3 changes: 3 additions & 0 deletions xtable-aws/src/test/java/org/apache/xtable/glue/table/TestDeltaGlueCatalogTableBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,11 @@ private StorageDescriptor getTestStorageDescriptor(List<Column> columns) {
return StorageDescriptor.builder()
.columns(columns)
.location(TEST_BASE_PATH)
.inputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat")
.outputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat")
.serdeInfo(
SerDeInfo.builder()
.serializationLibrary("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")
.parameters(
deltaGlueCatalogTableBuilder.getSerDeParameters(TEST_DELTA_INTERNAL_TABLE))
.build())
Expand Down
65 changes: 39 additions & 26 deletions xtable-utilities/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,21 @@
<artifactId>hudi-hadoop-common</artifactId>
<version>${hudi.version}</version>
</dependency>
<!--
Same story as hudi-hadoop-common above: xtable-aws declares this
provided (a real cluster supplies it), which doesn't propagate to
this standalone jar. Needed for org.apache.hudi.hive.MultiPartKeysValueExtractor,
referenced by Jackson when deserializing GlueCatalogConfig's
partition value extractor field. Without it, RunCatalogSync's Glue
catalog sync fails with NoClassDefFoundError at runtime despite
compiling fine (the reference is a string class name resolved via
reflection, so javac never catches the missing dependency).
-->
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-hive-sync</artifactId>
<version>${hudi.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hudi</groupId>
<artifactId>hudi-java-client</artifactId>
Expand Down Expand Up @@ -270,6 +285,17 @@
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<transformers>
<!-- Several bundled Spark data sources (delta-spark, hudi-spark,
parquet, avro, etc) each ship their own
META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
file at the same path. Without merging them, shade keeps only one
and silently drops the rest, including delta-spark's "delta"
shortname registration, so Spark's ServiceLoader lookup for
format("delta")/DeltaTable.forPath() finds nothing and falls back
to guessing the legacy class name "delta.DefaultSource", which
doesn't exist: ClassNotFoundException. This transformer merges
same-path service files instead of overwriting. -->
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"> </transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"> </transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resources>
Expand Down Expand Up @@ -594,6 +620,7 @@
<include>org.apache.hudi:hudi-client-common</include>
<include>org.apache.hudi:hudi-common</include>
<include>org.apache.hudi:hudi-hadoop-common</include>
<include>org.apache.hudi:hudi-hive-sync</include>
<include>org.apache.hudi:hudi-io</include>
<include>org.apache.hudi:hudi-java-client</include>
<include>org.apache.hudi:hudi-timeline-service</include>
Expand Down Expand Up @@ -738,33 +765,19 @@
<include>org.xerial.snappy:snappy-java</include>
<include>org.yaml:snakeyaml</include>
<include>oro:oro</include>
<include>software.amazon.awssdk:annotations</include>
<include>software.amazon.awssdk:apache-client</include>
<include>software.amazon.awssdk:auth</include>
<include>software.amazon.awssdk:aws-core</include>
<include>software.amazon.awssdk:aws-json-protocol</include>
<!-- Only bundle; it already contains internally-consistent
copies of every individual software.amazon.awssdk:* module
(annotations/auth/aws-core/glue/profiles/etc). Including any
of those separately alongside bundle makes the shade plugin
merge two different builds of the same classes, pairing
(e.g.) bundle's auth/sts classes with an IoUtils/AttributeMap
from the standalone jar that's missing a method they call,
causing a NoSuchMethodError at JVM shutdown when closing S3A's
credentials provider. Compile-time resolution of the individual
modules is untouched (controlled by <dependencies>, not this
list); this only changes which copy's bytes end up in the
final jar. -->
<include>software.amazon.awssdk:bundle</include>
<include>software.amazon.awssdk:checksums</include>
<include>software.amazon.awssdk:checksums-spi</include>
<include>software.amazon.awssdk:endpoints-spi</include>
<include>software.amazon.awssdk:glue</include>
<include>software.amazon.awssdk:http-auth</include>
<include>software.amazon.awssdk:http-auth-aws</include>
<include>software.amazon.awssdk:http-auth-aws-eventstream</include>
<include>software.amazon.awssdk:http-auth-spi</include>
<include>software.amazon.awssdk:http-client-spi</include>
<include>software.amazon.awssdk:identity-spi</include>
<include>software.amazon.awssdk:json-utils</include>
<include>software.amazon.awssdk:metrics-spi</include>
<include>software.amazon.awssdk:netty-nio-client</include>
<include>software.amazon.awssdk:profiles</include>
<include>software.amazon.awssdk:protocol-core</include>
<include>software.amazon.awssdk:regions</include>
<include>software.amazon.awssdk:retries</include>
<include>software.amazon.awssdk:retries-spi</include>
<include>software.amazon.awssdk:sdk-core</include>
<include>software.amazon.awssdk:third-party-jackson-core</include>
<include>software.amazon.awssdk:utils</include>
<include>software.amazon.eventstream:eventstream</include>
<include>stax:stax-api</include>
<include>tomcat:jasper-compiler</include>
Expand Down