Dockerfile - Add cuda13.3.dockerfile on pytorch:26.07-py3 - #846
Draft
gusui-msft wants to merge 4 commits into
Draft
Dockerfile - Add cuda13.3.dockerfile on pytorch:26.07-py3#846gusui-msft wants to merge 4 commits into
gusui-msft wants to merge 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new CUDA 13.3 container image definition based on nvcr.io/nvidia/pytorch:26.07-py3, updates the GitHub Actions image build/merge matrices to build/publish cuda13.3 instead of cuda13.0, and adjusts third-party build logic to select a newer CUTLASS for CUDA 13.3+.
Changes:
- Add
dockerfile/cuda13.3.dockerfile(new base image + updated HPC-X/UCX/Docker client handling). - Update
.github/workflows/build-image.ymlmatrix entries to build/merge/publishcuda13.3tags. - Update
third_party/MakefileCUTLASS selection to usev4.7.0whenCUDA_VER >= 13.3.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
third_party/Makefile |
Adds a CUDA 13.3+ conditional branch to select CUTLASS v4.7.0. |
dockerfile/cuda13.3.dockerfile |
Introduces the CUDA 13.3 image based on nvcr.io/nvidia/pytorch:26.07-py3 with updated dependency versions. |
.github/workflows/build-image.yml |
Switches CI build/merge matrices from cuda13.0 to cuda13.3 tags/dockerfile. |
Suppressed comments (1)
third_party/Makefile:47
git clone ... && cd cutlassis misleading/no-op here: each recipe line runs in its own shell, and the later build uses-S ./cutlassanyway, so thecd cutlasshas no effect on subsequent steps. Consider removing the trailing&& cd cutlassto avoid implying later commands run from inside the repo.
$(eval ARCHS := "100;103")
if [ -d cutlass ]; then rm -rf cutlass; fi
git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (2)
third_party/Makefile:48
git clone ... && cd cutlassis misleading here: Make runs each recipe line in its own shell, so thecd cutlassdoes not affect subsequent commands (which already reference./cutlass). Consider dropping the&& cd cutlasssuffix to avoid implying that later steps run inside the repo directory.
ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
$(eval ARCHS := "100;103")
if [ -d cutlass ]; then rm -rf cutlass; fi
git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)
dockerfile/cuda13.3.dockerfile:162
- This Dockerfile changes the established pattern of pinning
setuptoolsto an exact version (e.g.,cuda13.0.dockerfileandcuda12.9.dockerfileusesetuptools==...). Using a lower-bound specifier (setuptools>=...) makes image builds non-reproducible because the resolved version can change over time. Consider pinning to a specific known-good version (e.g., the base image’s current setuptools) and bump it deliberately when needed.
RUN python3 -m pip install --upgrade "setuptools>=78.1.1" && \
python3 -m pip install --no-cache-dir .[nvworker] && \
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #846 +/- ##
=======================================
Coverage 86.02% 86.02%
=======================================
Files 103 103
Lines 7950 7950
=======================================
Hits 6839 6839
Misses 1111 1111
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
third_party/Makefile:48
- The new CUDA_VER>=13.3 branch duplicates the same ARCHS/rm/clone recipe used for the >=12.9 branch, which increases maintenance cost and makes future tag/arch updates easy to miss. Consider setting CUTLASS_TAG (and ARCHS) inside the conditional branches, then running the shared rm/clone logic once after the conditional (or factoring a small helper target).
# The CUTLASS tag is picked per CUDA_VER because each release only supports the SM archs of its own CUDA generation.
cuda_cutlass:
ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
$(eval ARCHS := "100;103")
if [ -d cutlass ]; then rm -rf cutlass; fi
git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)
Comment on lines
+69
to
+74
| # Install Docker | ||
| ENV DOCKER_VERSION=29.7.2 | ||
| RUN TARGETARCH_HW=$(uname -m) && \ | ||
| wget -q https://download.docker.com/linux/static/stable/${TARGETARCH_HW}/docker-${DOCKER_VERSION}.tgz -O docker.tgz && \ | ||
| tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/ && \ | ||
| rm docker.tgz |
3 tasks
Author
|
@microsoft-github-policy-service agree company="microsoft" |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
third_party/Makefile:47
- The
&& cd cutlassat the end of this recipe line is misleading/no-op: each Make recipe line runs in its own shell (no.ONESHELLhere), and the later build already uses-S ./cutlass. Dropping thecdavoids implying subsequent lines run inside the repo directory.
git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
dockerfile/cuda13.3.dockerfile:138
- This UCX build leaves the downloaded tarball and extracted source tree under
/tmp, which increases final image size and can make layer caching less effective. Consider removing the UCX tarball/source directory in the sameRUNlayer aftermake install.
tar xzf ucx-${UCX_VERSION}.tar.gz && \
cd ucx-${UCX_VERSION} && \
./contrib/configure-release-mt --prefix=/usr/local && \
make -j ${NUM_MAKE_JOBS} && \
make install
Comment on lines
+80
to
+82
| sed -i "s/[# ]*PermitRootLogin prohibit-password/PermitRootLogin yes/" /etc/ssh/sshd_config && \ | ||
| sed -i "s/[# ]*PermitUserEnvironment no/PermitUserEnvironment yes/" /etc/ssh/sshd_config && \ | ||
| sed -i "s/[# ]*Port.*/Port 22/" /etc/ssh/sshd_config && \ |
…ndle Carries over the review outcome from #845: the RUN unpacks all eight binaries from the tarball into /usr/local/bin/, not only the client, so the header must not say "Docker Client".
gusui-msft
force-pushed
the
dev/gusui/cuda13.3-image
branch
from
August 24, 2026 19:54
9f89402 to
24cb5ff
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
.github/workflows/build-image.yml:34
- These entries remove the last CUDA 13.0 build and make the unvalidated CUDA 13.3 image the only CUDA 13 artifact, while the PR's own checklist still leaves both architecture builds, representative runs, and rescans unchecked. If the simultaneous CUDA/PyTorch/OpenMPI/UCX changes fail, this replaces the known-good CI artifact rather than introducing a canary. Please keep cuda13.0 in the matrix or gate this replacement on successful validation of both architectures and representative scans/runs.
- name: cuda13.3-arm64
dockerfile: cuda13.3
tags: superbench/main:cuda13.3-arm64
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
cuda13.3image built onnvcr.io/nvidia/pytorch:26.07-py3, replacingcuda13.0(25.08-py3) in the CI matrix.Motivation is security. A Trivy 0.72.0 scan of
main-cuda13.0(2026-07-15) reports 58 Critical occurrences, from exactly two sources. This PR closes both, so the new image starts at 0 Critical:25.08-py3)FROM nvcr.io/nvidia/pytorch:26.07-py3DOCKER_VERSION=29.7.2Both changes land together on purpose. Following the one-Dockerfile-per-CUDA-version convention, this is a new file rather than an edit to
cuda13.0.dockerfile, and a new file should not be born carrying a known-vulnerable Docker bundle. Keeping the two bumps in one PR means one CUDA version maps to exactly one image that gets built, benchmarked, and scanned once. This supersedes #845, which is now closed.Major Revision
dockerfile/cuda13.3.dockerfilebased onnvcr.io/nvidia/pytorch:26.07-py3.DOCKER_VERSION=29.7.2in the new file, rather than inheriting20.10.8fromcuda13.0.dockerfile.docker-buildanddocker-mergematrices atcuda13.3instead ofcuda13.0.CUDA_VER >= 13.3branch tothird_party/Makefileselecting CUTLASSv4.7.0, whose release notes state "Optimal code generation with CUDA toolkit versions 13.3". The existing>= 12.9branch would otherwise still pickv4.1.0, which predates CUDA 13.3.Minor Revision
HPCX_VERSIONv2.24.1->v2.50. Not cosmetic: the install block doesrm -rf /opt/hpcxand/etc/bash.bashrcsourceshpcx-init.sh, so keepingv2.24.1would replace the base image's HPC-X 2.50 with an older stack.UCX_VERSION1.18.0->1.21.0, matching the base image. Also not cosmetic:LD_LIBRARY_PATHputs/usr/local/libahead of the base libraries, so the source-built UCX shadows the base's. The URL is also corrected from the-rc1tag to the final release tag. Theconfigure-release-mtmulti-threaded build is kept.setuptools==78.1.0->setuptools==81.0.0. The old pin is a downgrade now (base ships81.0.0) and78.1.0is itself CVE-2025-47273 (HIGH), which the current scans report. Pinning to the version the base already carries keeps the line reproducible and makes it a no-op.26.07-py3component versions. The Docker line readsfull static bundle installed in this dockerfile, because theRUNunpacks all eight binaries, not only the client.cuda_cutlassinthird_party/Makefile; it claimed 12.9+ buildsv3.9while the recipe already clonedv4.1.0, and adding a third branch made it more misleading.1. Base image findings cleared, 18 occurrences
These are inherited and cannot be fixed in our own layers.
25.08-py326.07-py3shipslinux-libc-dev6.8.0-71.716.8.0-134.1346.8.0-136.136jupyter_server2.16.02.20.02.20.0efa_metrics/nic_samplerv1.23.41.26.0-rc.3go1.26.1Each of those three versions was read directly out of the
26.07-py3registry layers, not taken from release notes. The upgrade additionally bringswheel 0.45.1 -> 0.46.3(CVE-2026-24049, HIGH) andpip 25.2 -> 26.1.2(4 Medium, 1 Low).2. Docker bundle findings cleared, 40 occurrences
cuda13.0.dockerfilepins the Docker static distribution at20.10.8, built with Go 1.16.6. Trivy reports 40 Critical occurrences against the eight binaries theRUNunpacks into/usr/local/bin/. These are compiled-in Go dependencies, not installed packages, soaptandpipcannot touch them; the bundle itself has to be replaced.Docker
29.7.2(released 2026-08-05) is built with Go 1.26.5 and carries gRPCv1.82.1:stdlib1.24.13, 1.25.7, 1.26.0-rc.3go1.26.5stdlib1.16.14, 1.17.7go1.26.5stdlib1.19.8, 1.20.3go1.26.5stdlib1.19.9, 1.20.4go1.26.5stdlib1.21.11, 1.22.4go1.26.5Verified before opening, carried over from #845:
x86_64/docker-29.7.2.tgzandaarch64/docker-29.7.2.tgzboth return 200.${TARGETARCH_HW}already resolves correctly for both.go1.26.5, on both architectures.dockerdembedsgoogle.golang.org/grpc v1.82.1.Docker version 29.7.2, build a7dcaa6.Host compatibility. SuperBench only uses the Docker client against the host daemon socket (
monitor.py,docker_base.py,runner.py,system_info.py, anddeploy.yaml, which mounts/var/run/docker.sock). There is no reference todockerd,containerd,runc,ctr, ordocker-proxyanywhere in the tree. Docker29.3.0lowered the minimum daemon API version from v1.44 back to v1.40 (Docker 19.03), so the newer client still talks to older host daemons.docker login --username/--passwordand every subcommand we invoke are still present in 29.7.2. One note for reviewers: the v1containerd-shimbinary no longer ships in the 29.x bundle, and nothing here references it.3. What changed in the base image
Unchanged, so no migration work needed: Python 3.12, Ubuntu 24.04,
CUDA_HOME,_CUDA_COMPAT_PATH.25.08-py326.07-py313.0.0.04413.3.1.0082.8.0a02.13.0a0+9186a089.12.0.469.24.0.4313.0.0.1913.6.0.22.27.72.30.72.52.174.1.75.0.102.242.501.19.01.21.056.063.02025.4.1.1362026.3.1.117Artifact availability, checked 2026-08-18
hpcx-v2.50-gcc-doca_ofed-ubuntu24.04-cuda13-{x86_64,aarch64}.tbz: both 200ucx-1.21.0.tar.gzat tagv1.21.0: 200MLNX_OFED_LINUX-24.10-1.1.4.0-ubuntu24.04-{x86_64,aarch64}.tgz: both 200NVIDIA/cuda-samplestagv13.3exists, so the unconditional clone incuda_bandwidthTestwill not breakNVIDIA/cutlasstagv4.7.0existsDraft, because this is unvalidated
This replaces CUDA, cuDNN, NCCL, PyTorch, HPC-X, UCX, and Docker at once, so it cannot be accepted on a scan result alone.
docker --versioninside the built image reports 29.7.2Risks, in the order I expect them to bite:
2.8.0a0->2.13.0a0, five minor releases. Affectsmegatron_lm,megatron_deepspeed, Apex, and TransformerEngine.setup.pyonly declarestorch>=1.7.0a0and there is notorch.__version__guard anywhere insuperbench/, so pip will not surface a break; it has to be found by building and running.python3-mpi4pycomes from apt and links the distro MPI, whilenccl-testsandperftestbuild againstMPI_HOME=/usr/local/mpi.v4.7.0has not been built here against CUDA 13.3 yet.MLNX_OFED 24.10-1.1.4.0user-space now sits on top of rdma-core 63.0 rather than 56.0. Pre-existing pattern, wider gap.Docker is deliberately low on that list: the client-side surface is small and the compatibility floor was checked above. It is bundled here because splitting it out produces a
cuda13.0image variant that CI would no longer build once this PR lands.Questions for reviewers
dockerfile/cuda13.0.dockerfilebe deleted in this PR? CI/CD - Clean up image builds, remove cuda 12.8/12.4/12.2 and add cuda13.0 merge #819 kept superseded Dockerfiles and only changed the matrix, so this PR follows that precedent and leaves the file in place. Note that if it is kept, it keeps Docker20.10.8, since Dockerfile - Upgrade bundled Docker to 29.7.2 in cuda13.0 #845 is closed; that is intentional, as CI will no longer build it.git clone ... && cd cutlassinthird_party/Makefileis a no-op, since Make runs each recipe line in its own shell. Raised by review, but it is pre-existing onmainin all three branches, so cleaning it up belongs in its own change rather than here.Not in this PR
cuda12.9andcuda11.1.1are also built by CI and still carry Docker20.10.8. Same bundle, same 40 occurrences each. They need either the same one-line bump or their own base upgrade, tracked separately.dockerclient instead of the full bundle. That would drop 7 unused binaries and ~181 MB, but it changes image contents rather than a version string, so it is left as a follow-up.dockerfile/*.dockerfilehere, so it belongs in one change across all of them rather than only this one.rocm6.4.x.dockerfilein Dockerfile - Add ROCm6.4 dockerfile #837.