docs: Spark runtime guide, 0.4.0 release page, and xtable-service security note - #847
docs: Spark runtime guide, 0.4.0 release page, and xtable-service security note#847vinishjail97 wants to merge 5 commits into
Conversation
|
|
||
| ## Supported formats | ||
|
|
||
| Paimon and Parquet are read-only sources (there is no corresponding write target). |
There was a problem hiding this comment.
The bundle IT covers Hudi↔Iceberg and Hudi↔Delta, but the matrix also marks Iceberg↔Delta and Paimon/Parquet sources as supported. Are those validated somewhere, or is this the core capability matrix rather than what this bundle is tested for? A line distinguishing the two would help.
Separately, the README describes the IT as "one case per direction across Hudi, Iceberg, and Delta" — that reads a bit broader than the actual directions(), which has no Iceberg↔Delta case.
|
|
||
| | Spark version | Hudi / Iceberg | Delta implementation | | ||
| | --- | :---: | --- | | ||
| | 3.4.x | ✅ | Delta Standalone (`delta-core`) | |
There was a problem hiding this comment.
Minor naming thing — the delta-core in brackets is unambiguous, but "Delta Standalone" usually names io.delta:delta-standalone, a separate JVM library now deprecated in favor of Delta Kernel. The 3.4 path here is delta-core, so is the "Delta Standalone" label intentional? Delta's own naming is confusing (delta-core was renamed delta-spark in 3.0), so feel free to ignore if this is the convention we want.
|
|
||
| | Spark version | Hudi / Iceberg | Delta | | ||
| | --- | :---: | --- | | ||
| | 3.4.x | ✅ | Delta Standalone (`delta-core`) | |
There was a problem hiding this comment.
Same minor naming question as the docs page — "Delta Standalone" is a separate (deprecated) artifact from the delta-core in brackets.
vinishjail97
changed the title
Document running an XTable sync with spark-submit using the xtable-spark-runtime_2.12 0.4.0-incubating jar published to Maven Central: a quickstart, adding the runtime to an existing Spark job, multi-table --datasetconfig, the full CLI option table, the source/target matrix, and Spark version compatibility. Linked from Installation and the docs sidebar.
The REST service exposes POST /v1/conversion/table with no authentication or authorization. State that it must run locally or inside a trusted private network, and must never be exposed to the public internet.
7957369 to
2525435
Compare
Add worked examples for the case the xtable-spark-runtime proposal was written for: a job already writing Hudi that should also be readable as Iceberg and Delta, and a job writing Iceberg that should also be readable as Hudi and Delta. Both call XTableSyncService after the write.
demo/spark-runtime holds a spark-submit job that calls XTableSyncService from inside the job that writes the table, covering both directions and asserting row counts, with a README for building the engine classpath and running it. Document that a Hudi target must be read with hoodie.metadata.enable=true. XTable records the target's file listing in the Hudi metadata table, so without it the sync reports success and the read returns zero rows.
|
Closing this PR as the docs need to be re-written..as there have been new changes. I don't want to add more changes to an approved PR..better to do a fresh review from the community. |
2 participants
What
Website documentation only. No module sources are touched.
Run XTable on Apache Spark(website/docs/spark-runtime.md) — aspark-submitquickstart with worked examples, the full CLI option table, a source/target format matrix, and Spark version compatibility. Linked from the Installation page and the docs sidebar (Quick Start).website/releases/release-0.4.0-incubating.mdx) and a matching entry on the Downloads page.xtable-service/README.md— call out that the REST service ships with no authentication or authorization, so it must run locally or inside a trusted private network and must not be exposed to the public internet.Spark quickstart (excerpt)
xtable-spark-runtime_2.120.4.0-incubatingis published to Maven Central, so the docs point at the released jar rather than a source build:The jar must be the positional application jar.
--packagesalone does not work, becausespark-submitthen reportsMissing application resource, and the--basepathflag is rejected as an unrecognized option before it ever reachesXTableSparkSync.--packagesis documented only for adding the runtime to a job that already submits its own jar.Every dependency in the published pom is
providedortestscoped, so the jar resolves with no transitive downloads.The docs also cover Spark version compatibility: Hudi/Iceberg sync is Spark-free, and a Delta source/target auto-switches from
delta-core(Spark 3.4) to the Spark-free Delta Kernel on Spark 3.5+.Merge order
No longer blocked. Earlier revisions of this PR added
xtable-spark-runtime/README.mdand an examples file, which would have been orphaned because the module is not onmain. Those files are dropped, and the docs now reference the released Maven artifact instead of a source build, so this merges intomainon its own.