Skip to content

feat(ci): Publish Maven artifacts and release tarballs on release - #43

Merged
giggsoff merged 1 commit into
develop/4.4.0/4.2.0.2from
feature/NGSOK-1746
Sep 2, 2026
Merged

giggsoff merged 1 commit into
develop/4.4.0/4.2.0.2from
feature/NGSOK-1746

Conversation

@giggsoff

@giggsoff giggsoff commented Jun 23, 2026 •

Copy link
Copy Markdown
Collaborator

The 4.2.x half of the pair; #61 is the 3.5.x counterpart. Publish Spark to the Arenadata GitHub Packages Maven registry and attach the binary distributions to the GitHub release when a release is created.

Guards — a verify job runs first and is the only thing that can reject the release, so a bad tag fails in a minute rather than two hours into a reactor build. Ported from kyuubi's release workflow: the tag must sit on a release/** or develop/** branch, and CI must have passed for that commit. Added to those, the tag must match the pom version. Unlike the kyuubi original, an unfinished CI run is reported as still running rather than as missing — conclusion is empty in both cases.

Maven — mvn deploy with deployAtEnd, so the whole reactor builds before any upload and a mid-reactor failure cannot leave a partial release that GitHub Packages then refuses to re-publish. Manual dispatch can resume the reactor via -rf. Javadoc/scaladoc jars are skipped (attach-scaladocs is bound to verify, so deploy would build docs for the whole reactor); source jars are still published.

Credentials — the reactor reads the Arenadata Hive artifacts (hive.version) from the arenadata GitHub Packages repository, and GitHub Packages requires authentication even for public repos. Since actions/setup-java configures only one server, the deploy target id is renamed github → arenadata to share the id of the <repositories> entry; one <server> then covers both the reads and the deploy. Nothing else referenced the old id.

Assets — builds spark-<version>-bin-hadoop3.tgz, spark-<version>-bin-hadoop3-cloud.tgz and the PySpark sdist and attaches them to the release, matching the asset set of v4.2.0.1-4.3.0-2. The profile list is the one recorded in that release's RELEASE file; -Phadoop-cloud is added only for the -cloud distribution and the Maven deploy, since it pulls the AWS SDK bundle into the assembly and doubles the tarball. Asset paths are globbed and asserted to match exactly one file rather than derived, because the PySpark version is normalised by PEP 440 and is not a substring of the pom version. Reference sizes to check the first run against: 525 / 1134 / 1040 MB.

Note that a release event resolves the workflow from the tag's tree, so this takes effect for tags cut after it merges; it cannot retroactively attach assets to an existing release. The release_tag dispatch input is the re-run path for a failed upload.

@giggsoff giggsoff self-assigned this Jun 23, 2026
@giggsoff
giggsoff changed the base branch from develop/4.3.0/4.2.0.1 to develop/4.4.0/4.2.0.2 September 2, 2026 12:58
@giggsoff giggsoff changed the title Publish artifacts to GitHub Packages on release Publish Maven artifacts and release tarballs on release Sep 2, 2026
@giggsoff giggsoff changed the title Publish Maven artifacts and release tarballs on release feat(ci): Publish Maven artifacts and release tarballs on release Sep 2, 2026
Deploy the reactor to the Arenadata GitHub Packages Maven registry and attach
spark-<version>-bin-hadoop3.tgz, spark-<version>-bin-hadoop3-cloud.tgz and the
PySpark sdist to the GitHub release. A verify job gates both on the tag sitting
on a release/** or develop/** branch, CI having passed for that commit, and the
tag matching the pom version. The deploy target now shares the id of the
`arenadata` <repositories> entry, so one settings.xml server authenticates both
the reads and the deploy.
@giggsoff
giggsoff merged commit 3c6aec2 into develop/4.4.0/4.2.0.2 Sep 2, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants