feat: add xtable-converter Claude Code skill

.claude/settings.json

@@ -0,0 +1,11 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(mktemp -d /tmp/xtable_run_*)",
+      "Bash(python3 .claude/skills/xtable-converter/scripts/*)",
+      "Bash(java -cp *)",
+      "Bash(find ~/.m2/repository/org/openjdk/jol*)",
+      "Bash(bash .claude/skills/xtable-converter/scripts/*)"
+    ]
+  }
+}

.claude/skills/xtable-converter/SKILL.md

@@ -0,0 +1,147 @@
+---
+name: xtable-converter
+description: Convert lakehouse tables between Apache Hudi, Apache Iceberg, and Delta Lake formats using Apache XTable (incubator-xtable) driven by natural language instead of hand-written YAML. Use this skill whenever the user asks to convert, sync, translate, or expose a table in another format — e.g. "convert my Hudi table to Iceberg", "make this Delta table queryable from Snowflake", "sync s3://bucket/table to Delta and Iceberg", "run xtable on these tables" — or mentions XTable, table format interoperability, or datasetConfig YAML. Every conversion MUST end with an explicit per-table SUCCESS / FAILED / UNVERIFIED verdict; never report an implied or assumed success.
+---
+
+# XTable Converter
+
+Turn a natural-language request like *"convert my Hudi table at s3://lake/orders to Iceberg"* into a validated XTable `datasetConfig` YAML, run the conversion, and report a verified per-table verdict. XTable converts **metadata only** — it never copies or rewrites data files, so conversions are cheap and non-destructive.
+
+## Non-negotiable rules
+
+1. **No silent failures.** Every run ends with the Conversion Report (format below) showing SUCCESS, FAILED, or UNVERIFIED per (table, target format) pair. The XTable jar catches per-table exceptions and can exit 0 even when a table failed — exit code alone is NEVER proof of success.
+2. **No guessed paths or credentials.** If the user hasn't given a table path, source format, or required partition spec, ask. Never invent S3/ADLS/GCS paths or credentials.
+3. **Run immediately when slots are complete.** If the request contains all required slots, write the YAML and run without asking for confirmation — show the config inline as part of reporting, not as a blocker. Only pause when a required slot is missing or ambiguous.
+4. **SUCCESS requires ground-truth verification.** "The log said it worked" is only UNVERIFIED until target metadata is confirmed on storage.
+
+## Workflow
+
+### Step 1 — Extract conversion slots from the user's request
+
+| Slot | Required? | Notes |
+|---|---|---|
+| `sourceFormat` | Yes | One of HUDI, ICEBERG, DELTA. Infer from phrases like "my Hudi table". Ask if ambiguous. |
+| `targetFormats` | Yes | One or more of the other two formats. |
+| `tableBasePath` | Yes (per table) | s3://, gs://, abfs://, hdfs://, or local file path. Never guess. |
+| `tableName` | Yes (per table) | Default to the last path segment if user doesn't name it; mention the assumption. |
+| `tableDataPath` | Required for ICEBERG sources | Usually `<basePath>/data`. Optional otherwise. |
+| `partitionSpec` | Required for partitioned HUDI sources | Format `path:type[:format]`, comma-separated for multi-partition. See `references/config-schema.md`. |
+| `namespace` | Optional | Only when syncing to a catalog. |
+| Catalog / hadoop / converter configs | Optional | `--icebergCatalogConfig`, `--hadoopConfig`, `--convertersConfig`. See `references/config-schema.md`. |
+
+If the source is HUDI and you don't know whether the table is partitioned, ask: "Is the table partitioned? If so, which column(s) and are they string/date-string values or numeric timestamps?" — a wrong partitionSpec is the #1 cause of failed Hudi conversions. Use `VALUE` for string/date-string partition columns (identity transform); use `DAY`/`MONTH`/`YEAR`/`HOUR` only for numeric timestamp or long epoch columns.
+
+### Step 2 — Build and validate the YAML, then write it to disk
+
+Construct the config directly. Validate before writing:
+- `sourceFormat` and each entry in `targetFormats` must be HUDI, ICEBERG, or DELTA (uppercase).
+- `targetFormats` must not include `sourceFormat`, and must have no duplicates.
+- Each dataset must have `tableBasePath` (valid scheme) and `tableName`.
+- ICEBERG sources must have `tableDataPath`.
+- `partitionSpec` is only valid for HUDI sources; each entry must be `path:type` or `path:type:format`, where type is VALUE, YEAR, MONTH, DAY, or HOUR, and date types require a format string.
+
+If any slot is invalid, report the reason and ask the user to clarify — do not write a bad config.
+
+Example output for *"convert my partitioned Hudi table at /tmp/orders to Iceberg and Delta"* where `order_date` is a string column:
+
+```yaml
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: /tmp/orders
+    tableName: orders
+    partitionSpec: order_date:VALUE
+```
+
+Write it to disk immediately — do not wait for confirmation:
+
+```
+RUN_DIR=$(mktemp -d /tmp/xtable_run_$(date -u +%Y%m%d_%H%M%SZ).XXXXXX)
+# write config.yaml to $RUN_DIR/config.yaml
+```
+
+Show the YAML in the same message that reports the conversion result.
+
+### Step 3 — Run the conversion
+
+```bash
+bash .claude/skills/xtable-converter/scripts/run_xtable.sh "$RUN_DIR/config.yaml" \
+  [--hadoopConfig path] [--icebergCatalogConfig path] [--convertersConfig path]
+```
+
+The wrapper locates the bundled jar (env var `XTABLE_JAR`, or searches `xtable-utilities*/target` and the current dir), records the run start timestamp, streams the jar output to `$RUN_DIR/run_<ts>.log`, and writes `$RUN_DIR/run_meta.json` (exit code, log path, start epoch). It requires Java 11+. A non-zero exit code means the whole run FAILED — still continue to Step 4 to extract the reason.
+
+**Known issue — `jol-core` missing from bundled jar:** If the run fails with `NoClassDefFoundError: org/openjdk/jol/info/GraphLayout`, bypass `run_xtable.sh` and invoke the jar directly with `jol-core` on the classpath:
+
+```bash
+JOL_JAR=$(find ~/.m2/repository/org/openjdk/jol -name "jol-core-*.jar" | grep -v sources | head -1)
+XTABLE_JAR=<path-to-bundled.jar>
+JAVA11_HOME=<path-to-java11>   # e.g. /opt/homebrew/Cellar/openjdk@11/11.0.25/libexec/openjdk.jdk/Contents/Home
+TS=$(date +%s)
+LOG=$RUN_DIR/run_${TS}.log
+PATH="$JAVA11_HOME/bin:$PATH" java -cp "$XTABLE_JAR:$JOL_JAR" org.apache.xtable.utilities.RunSync \
+  --datasetConfig "$RUN_DIR/config.yaml" > "$LOG" 2>&1
+EXIT=$?
+printf '{"exit_code": %s, "log": "%s", "start_epoch": %s, "config": "%s/config.yaml"}\n' \
+  "$EXIT" "$LOG" "$TS" "$RUN_DIR" > "$RUN_DIR/run_meta.json"
+```
+
+### Step 4 — Parse the log for per-table outcomes
+
+```bash
+python3 .claude/skills/xtable-converter/scripts/parse_result.py "$RUN_DIR/run_meta.json" --config "$RUN_DIR/config.yaml"
+```
+
+Outputs `log_results.json`: one record per (table, target) with `log_status` of `ok`, `error`, or `unknown`, plus the captured error message. Log markers are version-dependent — if you see many `unknown` statuses, read `references/troubleshooting.md` (section "Pinning log markers") and rely on Step 5 as ground truth.
+
+### Step 5 — Verify against storage and produce the final verdict
+
+```bash
+python3 .claude/skills/xtable-converter/scripts/verify.py "$RUN_DIR/run_meta.json" \
+  --log-results "$RUN_DIR/log_results.json" \
+  --config "$RUN_DIR/config.yaml"
+```
+
+For each (table, target) the script checks that the expected metadata exists at the target path **and is newer than the run start**: `_delta_log/*.json` for DELTA, `metadata/*.metadata.json` for ICEBERG, `.hoodie` timeline for HUDI. Local paths are checked directly; `s3://` via aws CLI; other schemes via `hadoop fs` if available. It merges with the log results into `final_report.json` using this verdict matrix:
+
+| Log says | Storage check | Final verdict |
+|---|---|---|
+| ok | fresh metadata found | **SUCCESS** |
+| ok | metadata missing or stale | **FAILED** (log/storage mismatch — investigate) |
+| ok | check not possible (no access/tooling) | **UNVERIFIED** |
+| error | anything | **FAILED** (with error message) |
+| unknown | fresh metadata found | **SUCCESS** (storage is ground truth) |
+| unknown | missing/stale or not possible | **FAILED** / **UNVERIFIED** |
+
+### Step 6 — Report. ALWAYS use this exact template
+
+```
+## Conversion Report — <date time> (run log: <path>)
+
+| Table | Target | Verdict | Evidence |
+|---|---|---|---|
+| orders | ICEBERG | ✅ SUCCESS | metadata/v3.metadata.json written <time> |
+| orders | DELTA   | ❌ FAILED  | <first line of error> |
+| sales  | DELTA   | ⚠️ UNVERIFIED | log reports success; no read access to verify s3://... |
+
+<one line per FAILED row: probable cause + suggested fix (see references/troubleshooting.md)>
+<for UNVERIFIED rows: exactly what the user can check manually>
+```
+
+Never summarize a run as "done" or "completed" without this table. If the process itself failed before any table was attempted (bad jar path, Java missing, malformed YAML), report a single FAILED row with the cause.
+
+## Bundled resources
+
+- `references/config-schema.md` — full datasetConfig schema, partitionSpec grammar, catalog/converter/hadoop config formats. Read when building a non-trivial config (multi-partition, catalogs, custom converters).
+- `references/troubleshooting.md` — common failures mapped to fixes, and how to pin log markers to your XTable version. Read whenever any row is FAILED or many rows are `unknown`.
+- `scripts/run_xtable.sh` — locates the jar, runs it, captures exit code and log path into `run_meta.json`.
+- `scripts/parse_result.py` — parses the run log into per-(table, target) `log_status`.
+- `scripts/verify.py` — checks target metadata on storage and produces the final SUCCESS/FAILED/UNVERIFIED verdict.
+
+## Environment assumptions (state these to the user on first run)
+
+- Java 11+ on PATH (`java -version`).
+- The XTable bundled jar available; set `XTABLE_JAR=/path/to/xtable-utilities_*-bundled.jar` or place it where the wrapper can find it. Built via `./mvnw install -DskipTests` or downloaded from a release.
+- Read access to target storage paths from this machine for verification (aws CLI for s3://, or hadoop fs for other schemes). Without it, the best possible verdict is UNVERIFIED — say so up front, not after the run.

.claude/skills/xtable-converter/evals/evals.json

@@ -0,0 +1,35 @@
+{
+  "skill_name": "xtable-converter",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Convert my Hudi table at s3://lake/orders to Iceberg. It's partitioned by order_date (daily, paths look like 2026-06-11).",
+      "expected_output": "Valid YAML with partitionSpec order_date:DAY:yyyy-MM-dd shown for confirmation, then run + Conversion Report with per-table verdict",
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Sync s3://lake/orders and s3://lake/sales from Hudi to both Delta and Iceberg, register under namespace analytics.core in our Glue catalog.",
+      "expected_output": "Multi-dataset config with namespace, asks for/uses icebergCatalogConfig, asks about partitioning for each Hudi table, ends with Conversion Report",
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "Make this Delta table queryable from Snowflake: abfs://gold@eglake.dfs.core.windows.net/bookings",
+      "expected_output": "Infers DELTA source -> ICEBERG target, asks nothing unnecessary, flags that abfs verification needs hadoop CLI (UNVERIFIED otherwise) before running",
+      "files": []
+    },
+    {
+      "id": 4,
+      "prompt": "Convert my table to Iceberg",
+      "expected_output": "Asks for source format and table path; never guesses a path",
+      "files": []
+    },
+    {
+      "id": 5,
+      "prompt": "Convert the Iceberg table at s3://lake/inventory to Hudi",
+      "expected_output": "Requires tableDataPath (suggests s3://lake/inventory/data), no partitionSpec, Conversion Report at the end",
+      "files": []
+    }
+  ]
+}

.claude/skills/xtable-converter/references/config-schema.md

@@ -0,0 +1,85 @@
+# XTable datasetConfig schema reference
+
+Verified against apache/incubator-xtable README (release 0.3.0-incubating). Re-check against the user's version if conversions behave unexpectedly.
+
+## Top-level datasetConfig YAML
+
+```yaml
+sourceFormat: HUDI            # HUDI | ICEBERG | DELTA
+targetFormats:
+  - DELTA
+  - ICEBERG
+datasets:
+  - tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center
+    tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data   # optional; REQUIRED for Iceberg sources
+    tableName: call_center
+    namespace: my.db          # optional; used when syncing to a catalog
+  - tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales
+    tableName: catalog_sales
+    partitionSpec: cs_sold_date_sk:VALUE          # Hudi sources only
+  - tableBasePath: s3://hudi/multi-partition-dataset
+    tableName: multi_partition_dataset
+    partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE
+```
+
+## Field rules
+
+- **sourceFormat / targetFormats**: uppercase HUDI, ICEBERG, DELTA. targetFormats must not include sourceFormat.
+- **tableBasePath**: base path of the table. Supported schemes: s3://, s3a://, gs://, abfs://, abfss://, hdfs://, file://, or a plain local path.
+- **tableDataPath**: where the data files live. For ICEBERG sources you must point at the `/data` directory. If omitted, basePath is used.
+- **namespace**: dot-separated (e.g. `my.db`). Only meaningful with a catalog sync.
+- **partitionSpec** (HUDI sources only):
+  - Comma-separated list of `path:type[:format]`.
+  - `path` = dot-separated path to the partition field (nested fields allowed: `meta.event_date`).
+  - `type` = one of:
+    - `VALUE` — identity transform (no format).
+    - `YEAR` | `MONTH` | `DAY` | `HOUR` — date-derived partitioning; **format is required**, e.g. `yyyy-MM-dd` or `yyyy/MM/dd`, matching how the date string appears in file paths.
+  - Unpartitioned Hudi table → omit partitionSpec entirely.
+  - DELTA and ICEBERG sources never need partitionSpec (partitioning is read from their own metadata).
+
+## Optional companion configs (CLI flags)
+
+### --icebergCatalogConfig catalog.yaml
+```yaml
+catalogImpl: io.my.CatalogImpl     # e.g. org.apache.iceberg.aws.glue.GlueCatalog
+catalogName: name
+catalogOptions:                    # passed through as a map
+  key1: value1
+  warehouse: s3://bucket/warehouse
+```
+
+### --convertersConfig converters.yaml
+Replace default converter implementations:
+```yaml
+tableFormatConverters:
+  HUDI:
+    conversionSourceProviderClass: org.apache.xtable.hudi.HudiConversionSourceProvider
+  DELTA:
+    conversionTargetProviderClass: org.apache.xtable.delta.DeltaConversionTarget
+    configuration:
+      spark.master: local[2]
+      spark.app.name: xtable
+```
+
+### --hadoopConfig hadoop.xml
+Standard Hadoop XML for storage credentials/endpoints (S3 keys, ADLS auth, GCS service accounts). The bundled jar already includes AWS, Azure, and GCP hadoop dependencies and ships defaults in `xtable-hadoop-defaults.xml`; a custom file overrides them. Never fabricate credential values — ask the user for their existing core-site/hadoop config.
+
+## Run command
+
+```bash
+java -jar xtable-utilities_2.12-<version>-bundled.jar \
+  --datasetConfig config.yaml \
+  [--hadoopConfig hdfs-site.xml] \
+  [--convertersConfig converters.yaml] \
+  [--icebergCatalogConfig catalog.yaml]
+```
+
+Java 11 is the supported build/runtime version.
+
+## What metadata appears on success (used by verify.py)
+
+| Target | Expected at tableBasePath | Notes |
+|---|---|---|
+| DELTA | `_delta_log/` containing `NNNNNNNNNNNNNNNNNNNN.json` commits | new commit file per sync |
+| ICEBERG | `metadata/` containing `vN.metadata.json` (+ snapshots/manifests) | new metadata version per sync |
+| HUDI | `.hoodie/` with `hoodie.properties` and timeline files (`*.commit`, `*.deltacommit`, or under `.hoodie/timeline/`) | new timeline instant per sync |

.claude/skills/xtable-converter/references/troubleshooting.md

@@ -0,0 +1,44 @@
+# Troubleshooting XTable conversions
+
+Read this when any report row is FAILED, or when parse_result.py returns many `unknown` statuses.
+
+## Common failures → fixes
+
+| Symptom in log | Probable cause | Fix |
+|---|---|---|
+| `UnsupportedClassVersionError` / `class file version 55.0` | Running with Java 8 | Use Java 11+ (`java -version`; switch with jenv/sdkman or JAVA_HOME) |
+| `NoSuchMethodError` / `ClassNotFoundException` on hadoop/parquet classes | Classpath clash — extra jars on CLASSPATH alongside the bundled jar | Run the bundled jar alone with `java -jar`; unset CLASSPATH |
+| `AccessDenied` / `403` on s3 paths | Missing/wrong AWS credentials or region | Provide credentials via env/instance profile, or pass `--hadoopConfig` with fs.s3a settings; verify with `aws s3 ls <path>` |
+| `AuthorizationPermissionMismatch` / abfs auth errors | ADLS auth not configured | Pass `--hadoopConfig` with the user's existing core-site for abfs |
+| `Unable to infer partition` / partition values null or wrong in target | Missing or incorrect `partitionSpec` for a Hudi source | Confirm partition column, transform type, and date format exactly as it appears in file paths (e.g. `yyyy/MM/dd` vs `yyyy-MM-dd`) |
+| `Cannot find table` / empty metadata on ICEBERG source | `tableDataPath` not set to the `/data` directory | Add `tableDataPath: <basePath>/data` |
+| Iceberg catalog errors (`NoSuchNamespaceException`, Glue/HMS auth) | Catalog config missing or namespace doesn't exist | Pass `--icebergCatalogConfig`; create the namespace; check `namespace` field |
+| Spark session errors on Delta target | Converter Spark conf | Pass `--convertersConfig` setting `spark.master: local[2]` for the DELTA target |
+| Schema evolution / incompatible type errors | Source has types unsupported by target snapshot | Check XTable GitHub issues for the specific type; sometimes fixed in newer release |
+| Process succeeded for table A, failed for table B, exit code 0 | Expected XTable behavior: per-table exceptions are caught so the batch continues | This is exactly why verify.py exists — trust the per-table verdict, not the exit code |
+
+## Log says ok but storage check FAILED (mismatch)
+
+Possible causes, in order of likelihood:
+1. verify.py checked the wrong path — confirm tableBasePath has no trailing-slash/scheme differences (s3 vs s3a) between config and check.
+2. Eventual consistency or a listing cache on the object store — re-run verify.py after ~30s before concluding.
+3. The sync genuinely wrote nothing (e.g. no new commits since last sync — XTable is incremental). If source had no new commits, target metadata mtimes may legitimately predate run start. Ask the user whether the source changed since the last sync; if not, an unchanged target is correct behavior, report SUCCESS with note "no new commits to sync".
+
+## UNVERIFIED rows
+
+Means the tooling couldn't read target storage. Tell the user exactly what to check manually, e.g.:
+- DELTA: `aws s3 ls s3://lake/orders/_delta_log/ | tail` — look for a new `.json` commit at the run time.
+- ICEBERG: `aws s3 ls s3://lake/orders/metadata/ | tail` — look for a new `vN.metadata.json`.
+- HUDI: list `.hoodie/` (or `.hoodie/timeline/`) for a new instant file.
+
+To make future runs verifiable: install/configure aws CLI for s3 paths, or ensure `hadoop` is on PATH with the same core-site used for the conversion.
+
+## Pinning log markers to your XTable version
+
+parse_result.py matches generic markers (lines containing `ERROR`, `Exception`, and success phrases like `Completed sync`/`completed successfully` near a table name). Log wording can change between releases. To pin:
+
+1. Run one known-good conversion and one deliberately broken one (bad path).
+2. `grep -i -E "sync|complete|error|exception" run_<ts>.log` and note the exact per-table success and failure lines.
+3. Update `SUCCESS_PATTERNS` / `ERROR_PATTERNS` at the top of `scripts/parse_result.py`.
+
+Even unpinned, the final verdict stays trustworthy because verify.py's storage check is ground truth.