Skip to content

Build and run the suite on Blackwell (sm_103) under CUDA 13.2 - #91

Open
tgrogers wants to merge 6 commits into
devfrom
b300-build-on-dev
Open

tgrogers wants to merge 6 commits into
devfrom
b300-build-on-dev

Conversation

@tgrogers

@tgrogers tgrogers commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Builds and runs the whole suite on a GB300 (Blackwell, sm_103, aarch64) under
CUDA 13.2. make goes from failing immediately on dev to completing with
472 binaries and zero errors.

Why this is a re-port

This work was originally done on a branch cut from release. That made a direct
merge into dev produce 36 conflicts, so the diff was re-applied onto dev
instead. Everything below is the dev version of that work, plus the extra
issues that only dev's own code exposed.

Architecture selection

setup_environment previously carried a hand-maintained per-CUDA-version arch
table that stopped at compute_100. It now intersects a desired-arch wish-list
with whatever nvcc --list-gpu-arch reports, with a static fallback for
pre-CUDA-11 toolkits. On CUDA 13.2 this resolves to 75/80/86/90/100/103.

Gencode uses the bracket form code=[sm_X,compute_X] rather than
code="sm_X,compute_X": identical SASS and PTX, but it contains no spaces, so
passing it through MAKE_ARGS cannot split into a stray make goal. PTX stays
embedded so GPGPU-Sim can still run these binaries.

Multi-arch is deliberate. The suite is built once and simulated as many
different machines, so each target wants the PTX a real build for that arch
would emit. Narrowing to the local GPU is available via GPUAPPS_ARCHS="103"
to cut build time.

CUDA 13 API removals

  • Legacy texture-reference API (texture<>, cudaBindTexture,
    cudaBindTextureToArray, cudaUnbindTexture) is gone. Migrated ~68 files to
    cudaTextureObject_t, declaring the handles __constant__ under their
    original names so every fetch site and dispatch macro is unchanged; the host
    fills them with cudaMemcpyToSymbol. Only the explicit template arguments
    (tex1Dfetch<float>, tex2D<uint4>) differ.
  • cudaDeviceProp::clockRate / ::memoryClockRate / ::computeMode are gone;
    replaced with cudaDeviceGetAttribute. Five sites.
  • cudaThreadSynchronize() is gone (lonestargpu pta).
  • thrust::unary_function is gone, and using namespace thrust now collides
    with ::cuda/::cub in nvcc-generated stubs.

Blackwell / newer-toolkit build breaks

  • arch_filter.mk (new) intersects any hardcoded ARCH list with
    nvcc --list-gpu-arch, so the five ubenches pinning sm_101 and friends stop
    being fatal. A MIN_ARCH mechanism in common.mk drops gencodes below a
    benchmark's floor (cp.async needs 80).
  • Submodule bumps: cutlass-bench v4.1.0 to v4.8.1 (4.1.0 warns
    unsupported or deprecated compute capabilities 103a), cuda-samples v12.8
    to v13.2 (12.8 pins compute_50, fatal under CUDA 13).
  • heterosync hardcoded compute_60; cleared so CUDA_CPPFLAGS decides.
  • ispass-2009/WP linked -lGL although no source includes a GL header.
  • MapSMtoCores tables stopped at SM 7.0 and SM 3.0; a GB300 fell off the end
    and reported 152x64 cores, or -1. Both now cover SM 3.2 to 12.1.

aarch64

  • -m64 is x86-only; made conditional in pathfinder and pathfinder_k1,
    matching the pattern already used in huffman and mummergpu.
  • char is unsigned by default, so getopt_long returning -1 compared as 255.
    Fixed in dwt2d along with a missing {0,0,0,0} longopts sentinel.

Correctness bugs found by running, not building

These are pre-existing and would silently corrupt results:

  • rodinia-3.1 kmeans used the pre-CUDA-5 cudaMemcpyToSymbol("c_clusters",...)
    string-literal form, which passes the address of the string. Unquoted.
  • pathfinder was missing two __syncthreads(), a real data race.
  • parboil sgemm declared readColMajorMatrixFile as bool and fell off the
    end, giving SIGTRAP under a modern compiler.
  • mummergpu has a if (this) null check inside a member function, which is
    UB that modern GCC deletes; built with -fno-delete-null-pointer-checks.

CUTLASS build

Examples 10 and 11 link cutlass_library, which drags in all 56,823 generated
kernels spanning sm_50 to sm_103; a serial sub-make then spent over six minutes
in a single ptxas on an FP4 kernel for an arch with no FP4 hardware. Filtering
to CUTLASS_LIBRARY_KERNELS=*planar_complex* keeps all 97 examples building
while cutting generated sources to 1,363. The profiler is no longer built, since
the install step only ever copied from build/examples. Arches are a
CUTLASS_ARCHS variable instead of a hardcoded 80;90;90a.

Makefile reconciliation

dev had drifted from the sub-Makefiles: 24 stale mv lines in the
rodinia-3.1 target relocated binaries that now link straight into BINDIR.
Removed.

Re-enabled

dev disabled several apps that release builds. They build and run clean
under CUDA 13, so they are back: ispass-2009 AES and MUM, rodinia-2.0-ft
kmeans, lonestar bfs-wlc and sssp-wlc (which needed uncommenting in two
places -- the VARIANTS list and the install mv), and parboil in all:.

Verification

Full make from clean: exit 0, zero errors, 472 binaries.

All nine suites run under the accel-sim run_hw.py harness at exit 0:
rodinia_2.0-ft, rodinia-3.1, ispass-2009, parboil, polybench, pannotia,
proxy-apps-doe, lonestargpu-2.0, GPU_Microbenchmark -- 94 apps, 130 run
configurations, with no app missing a binary.

Known pre-existing failures, unchanged by this PR: lonestar bh deadlocks and
pta goes out of bounds under Volta+ independent thread scheduling (both are
pre-Volta warp-synchronous code), mnistCUDNN needs APIs removed in cuDNN 9,
and shoc-Stability tries a 256GB allocation.

Hardware counter collection could not be exercised here: this machine has
RmProfilingAdminOnly: 1, so every ncu invocation fails ERR_NVGPUCTRPERM.
The applications themselves still ran to completion under the harness.

Deliberately out of scope

src/cuda/gpuwattch-ubench still calls cudaThreadSynchronize(), which CUDA 13
removed, so it will not compile on a CUDA 13 toolkit. This PR does not fix it.

An earlier revision did -- a mechanical cudaThreadSynchronize() ->
cudaDeviceSynchronize() rename across 181 files -- but no Makefile target
references that directory (the only wattch target points at
accelwattch-ubench), so nothing compiled the result and the change could not
be verified. It was 40% of this PR's file count for zero tested benefit, so it
has been dropped. The rename is almost certainly still correct and belongs in
its own commit, alongside a target that actually builds the suite.

Also reverted to dev's content: src/cuda/shoc-master/build-aux/config.guess
and config.sub. Those were regenerated as a byproduct of running autotools
during a shoc build attempt, not a deliberate change, and shoc is not re-enabled
here. They were ~16% of the line count.

Together these took the diff from 482 files / +15,277 -3,825 to 298 files /
+12,471 -1,457.

Companion changes not in this PR

🤖 Generated with Claude Code

tgrogers and others added 6 commits October 26, 2026 12:04
b300-build was cut from release, which split from dev in April 2021, so the
work could not be merged directly. This replays it onto dev: 439 files that
dev had not touched transfer unchanged, 25 are new, and the rest were
reconciled by hand against what dev had done in the meantime.

Where dev had solved the same problem differently its version was kept:
andersen.cu keeps dev's fully-qualified thrust:: names (only the
unary_function base, removed in CUDA 13, is dropped), dwt2d keeps dev's
NUL off-by-one fixes, kmeans keeps dev's removal of the hardcoded
cudaSetDevice(1), and the cutlass and heterosync recipes stay as dev has
them. The parboil driver is taken from this work because dev's python 3
port left ppkg.py and sanity_check.py unparseable and three python2
shebangs in place.

setup_environment keeps the architecture autodetection rather than dev's
per-version table: the table is why dev stops at compute_100 and emits no
native code for a GB300. dev's GPU_Microbenchmark restructure already feeds
CUDA_CPPFLAGS into every ubench, so it inherits sm_103 from that change
alone, and the 19 per-benchmark makefiles dev deleted needed no porting.
cutlass-bench is a submodule in dev, so its clockRate fix belongs upstream.

rodinia-3.1 kmeans is re-enabled in the build: dev had commented it out
over the legacy texture API, which is what this work replaced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both arch tables predate Volta, so on a GB300 (SM 10.3) they fell off the
end: lonestargpu printed "MapSMtoCores for SM 10.3 is undefined. Default
to use 64 Cores/SM" and reported 152*64 instead of 152*128, and the
sdk-compat copy -- which stopped at SM 3.0 -- returned -1 outright.

Fill both in from SM 3.2 up to SM 12.1 so core counts are right on
current hardware and the tables do not need touching again for a while.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test-12-8 job broke in three places, all from this branch assuming the
toolkit is CUDA 13.

setup_environment widened the gencode wish-list down to compute_10. On
CUDA 13 that changes nothing -- sm_75 is the floor -- but on CUDA 12.8 it
starts emitting compute_50/60/62, and <mma.h> only declares nvcuda::wmma
for __CUDA_ARCH__ >= 700, so config_tensor, tensor_bw_half and
tensor_lat_half failed with "wmma is not a class or namespace name". The
suite has effectively required Volta since the CUDA 11 era, so the floor
is now sm_70. Those three ubenches also declare MIN_ARCH = 70, which is
what makes an explicit GPUAPPS_ARCHS override safe -- their GENCODE_SMxx
assignments predate CUDA_CPPFLAGS and never applied to it.

CUTLASS_ARCHS was hardcoded to 100a;103a, which no CUDA 12 nvcc accepts,
and cutlass_mini had been switched onto it even though it builds the
Hopper examples by name and needs 90a. Both lists are now filtered
against nvcc --list-gpu-arch, and cutlass_mini keeps the 80;90;90a that
ci has always used.

cuda-samples is pinned at v13.2, whose CMakeLists hardcodes
set(CMAKE_CUDA_ARCHITECTURES ... 110 ...) -- it shadows any -D we pass,
and no CUDA 12 nvcc knows sm_110. Every cuda-samples tag targets exactly
one toolkit, so the target skips with a message below CUDA 13 instead of
failing the whole make.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in CI exercised the toolkit this branch is actually about, so the
three CUDA 13 regressions fixed in the previous commit only surfaced in
reverse -- as 12.8 breakage -- and a genuine CUDA 13 break would not have
surfaced at all.

The accel-sim org publishes no CUDA 13 container (the newest is
Ubuntu-24.04-cuda-12.9), so this starts from the stock NVIDIA devel image
and apt-installs what the minimal image would have provided. git goes in
before the checkout so actions/checkout clones with git rather than
falling back to the REST tarball, which would leave the submodules
unavailable.

13.2 is the toolkit the Blackwell port was developed and verified
against. Every target in `ci` builds on it locally: rodinia_2.0-ft,
rodinia-3.1, GPU_Microbenchmark, cutlass_mini and cuda_samples.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CUTLASS's CMakeLists does find_package(Python3 ... REQUIRED), and the
stock nvidia/cuda devel image ships no interpreter -- the accel-sim
minimal image did, which is why this only shows up on the new job.

rodinia_2.0-ft, rodinia-3.1 and GPU_Microbenchmark all built on 13.2
before cutlass_mini hit this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tgrogers pushed a commit to accel-sim/accel-sim-framework that referenced this pull request Oct 26, 2026
Re-enabling it here was premature: it depends on the CUDA 13 texture-object
port in accel-sim/gpu-app-collection#91, which has not merged. CI clones
gpu-app-collection's default branch, so the app is not built there --
run_hw_trace.py correctly reported "SKIP - no binary for
kmeans-rodinia-2.0-ft" and then run_simulations.py failed looking for traces
that were never generated.

SASS-Simulation has a second dependency besides #91: it replays a shared
pre-collected trace corpus that contains no kmeans traces, so that corpus
needs regenerating before this app can be turned back on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tgrogers
tgrogers requested a review from JRPan October 27, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant