[tgrogers]

Fixes found while bringing the hardware-run and tracing flows up on a GB300 (Grace-Blackwell, sm_103, aarch64, CUDA 13.2). Companion to accel-sim/gpu-app-collection#91, which does the application-side CUDA 13 port, and accel-sim/gpgpu-sim_distribution#150, which fixes the simulator build on the same machine.

Nothing here is Blackwell-specific except the NVBit tarball selection — the rest are bugs that simply happen to bite hardest when the app suite and the framework disagree about what is built.

A missing binary no longer aborts a whole suite

run_hw.py resolved each benchmark's executable with common.file_option_test(...), which raises on a missing path. Nothing caught it, so the first unbuilt app in a suite ended the run and every app listed after it was silently never launched. This reads as "the suite failed" when in fact most of it was never attempted — four suites looked broken here for exactly this reason.

The binary is now resolved up front; a missing one prints SKIP - no binary for <exe> and moves to the next app. Suite membership legitimately differs between branches and toolkits, so skipping is the right behaviour rather than a workaround.

The same fix is applied to run_hw_trace.py and pm_cupti_tools/run_cupti.py, which had copies of the same code with the same flaw.

Deduplicate the run-directory setup

The three runner scripts each carried their own copy of: make the run directory, link the benchmark's data dir, link the suite-wide data_dirs, resolve the binary, write run.sh, chmod u+x. The copies had already drifted (only run_hw.py checked the chmod exit status), which is how one fix in one script would have left the other two broken.

Two helpers in util/job_launching/common.py:

• prepare_run_dir(...) — directory, both data links, binary resolution; returns the executable path, or None when the app is not built
• write_run_script(...) — write and chmod, with a consistent error

Net ~130 lines of duplication removed. Ordering changed in one deliberate way: the binary is resolved before the directory is created, so a skipped app no longer leaves an empty run directory behind.

NVBit tarball matches the host architecture

install_nvbit.sh hardcoded nvbit-Linux-x86_64-1.8.tar.bz2, so on aarch64 it downloaded an x86 build and the tracer failed to link. It now picks the tarball from uname -m, accepts x86_64 and aarch64, errors clearly on anything else, and takes NVBIT_VERSION from the environment so bumping the version no longer means editing three string literals.

run.sh uses the interpreter that launched it

The nsys post-processing step shelled out to bare python, which does not exist on a stock Ubuntu 24.04 (only python3). Now sys.executable, so the generated script uses whatever interpreter ran run_hw.py.

App list corrections

define-all-apps.yml, all verified against a full build of gpu-app-collection#91 on CUDA 13.2:

• proxy-apps-doe — exec_dir pointed at $GPUAPPS_ROOT/src/cuda/proxy-apps-doe/ (a source tree) instead of the release bin dir, and cns/cns_all carried a path prefix the installed binary does not have. Neither app in this suite could ever have launched.
• kmeans-rodinia-2.0-ft — I re-enabled this and then backed it out; it stays disabled. See below.
• Deepbench_nvidia_normal — conv_bench and rnn_bench commented out. They use cuDNN APIs removed in cuDNN 9 and are not built, so listing them only produced skips. gemm_bench is unaffected and stays.

kmeans stays disabled

My first push re-enabled kmeans-rodinia-2.0-ft, on the grounds that gpu-app-collection#91 replaces its legacy texture-reference code with texture objects. That was premature and it broke CI, so it is reverted here.

Two things gate it, and #91 is only the first:

1. CI clones gpu-app-collection's default branch, which does not have Some issues with Pascal and probably a global bug #91, so the app is not built. The new skip logic behaved exactly as intended — SKIP - no binary for kmeans-rodinia-2.0-ft — but run_simulations.py then failed looking for traces that were never generated. The runners tolerate a missing binary; the simulation launcher does not tolerate a missing trace, which is reasonable since that usually is a real error.
2. SASS-Simulation replays a shared pre-collected trace corpus (.../for-sharing/accel-sim/QV100/hw_run/traces/) that contains no kmeans traces. Even once Some issues with Pascal and probably a global bug #91 merges, that corpus needs regenerating before this app can come back.

Worth a follow-up issue rather than a change here.

Testing

On a GB300, CUDA 13.2, driver 610.43.02, against the gpu-app-collection#91 build (472 binaries):

• All nine suites run to completion, exit 0 — 94 apps / 130 run configurations, zero SKIP lines, which is the real check on the proxy-apps-doe and kmeans entries above.
• run_cupti.py and run_hw_trace.py generate their run directories and scripts correctly.
• Python files compile, install_nvbit.sh passes bash -n, the YAML parses.

Not tested here: actually executing a trace. NVBit 1.8 refuses to attach to this machine's 610.43.02 driver (it caps at 595.xx), so the tracer path is verified only as far as script and run-directory generation. The architecture fix is what lets install_nvbit.sh fetch the right tarball on aarch64 at all; whether a newer NVBit then attaches is a separate question. Profiled (--profile) runs are likewise untested, as this host has NVreg_RestrictProfilingToAdminUsers set.

Merged with dev

dev moved while this was in flight. #570 rewrote run_hw_trace.py around multi-GPU worker pools and, independently, added its own local write_script() helper with the same body as the common.write_run_script here. Resolved by keeping dev's structure wholesale and pointing both of its script-writing call sites at the shared helper, so the deduplication still applies in its new location and there is only one copy again. prepare_run_dir and the app-list changes merged cleanly.

x86 behaviour is unchanged throughout: the architecture branch picks the same tarball it always did, and the refactor is behaviour-preserving apart from the skip and the empty-directory ordering.

🤖 Generated with Claude Code

[github-actions]

🤖 CI Runs failed on 2f1e87d — run. AI analysis (Qwopus3.8-27B-Flash-MTP-Q8_0):

CI Failure Analysis

Commit: 2f1e87d Merge branch 'dev' into b300-hw-run-fixes

Summary

The SASS-Simulation-Docker job failed immediately after queuing Slurm jobs because the kmeans-rodinia-2.0-ft benchmark was missing its pre-collected traces, causing GPGPU-Sim to exit with code 1; this failure cascaded into other simulation and tracer jobs that also lacked required trace files or encountered similar infrastructure issues.

Failed job(s)

• SASS-Simulation-Docker / Build and Run in the Image: Infrastructure (missing pre-collected traces for kmeans benchmark)
• SASS-Simulation / Run SASS Simulations: Infrastructure (Slurm array jobs failed to launch, likely due to missing trace files or resource constraints)
• Tracer-Tool / Test New Traces: Wrong result/Infrastructure (simulation exited with code 1 after queuing Slurm arrays; the preceding "Cannot find traces" warning indicates missing pre-collected SASS traces for kmeans)

Root cause

The literal error in SASS-Simulation-Docker_.excerpt.md is:
srun: error: tgrogers-bigram-08: task 0: Exited with exit code 1
##[error]Process completed with exit code 1.

This occurred immediately after the log line:
Cannot find traces in any of the paths: ['./hw_run/rodinia_2.0-ft/9.1/kmeans-rodinia-2.0-ft/_i___data_400_txt__g_data_400_result_txt__o/traces']

The kmeans-rodinia-2.0-ft benchmark requires pre-collected SASS traces to run in GPGPU-Sim, but the Docker image or CI environment did not have these trace files available at the expected path (./hw_run/rodinia_2.0-ft/9.1/kmeans-...). The simulator exited with code 1 when it could not find the required input data.

The same "Cannot find traces" error appears in Tracer-Tool_.excerpt.md during its "Test New Traces" step, confirming that this is a systemic issue with missing pre-collected trace files for the kmeans benchmark across multiple CI jobs. The SASS-Simulation job also failed immediately after queuing Slurm array tasks (exit code 1), suggesting either similar missing traces or a resource allocation failure in the Slurm cluster.

The recent commits indicate that kmeans-rodinia-2.0-ft was re-enabled in commit 330bc2ab ("re-enable kmeans-rodinia-2.0-ft now that it uses texture objects and passes its gold check again"), but this change may have assumed the pre-collected traces were available when they are not present in all CI environments (particularly Docker).

Suggested fix

1. Immediate: Ensure pre-collected SASS traces for kmeans-rodinia-2.0-ft are included in the Docker image or downloaded during CI setup before running simulations. Check that the path structure matches what GPGPU-Sim expects (./hw_run/rodinia_2.0-ft/9.1/kmeans...).

2. Alternative: If kmeans traces cannot be easily added to all environments, temporarily disable kmeans-rodinia-2.0-ft in CI configurations until trace availability is resolved across all runners (Docker and bare-metal).

3. Investigation: Verify that the Slurm cluster has sufficient resources for the array jobs being submitted by SASS-Simulation; if this was a resource issue rather than missing traces, adjust memory/CPU requests or queue settings in main-sass.sh.

_{Automated analysis; check it against the log before acting on it.}

[github-actions]

🤖 CI Runs failed on 2f1e87d — run. AI analysis (Qwopus3.8-27B-Flash-MTP-Q8_0):

CI Failure Analysis

Commit: 2f1e87d Merge branch 'dev' into b300-hw-run-fixes

Summary

The SASS-Simulation-Docker job failed because a Slurm task exited with code 1 immediately after the simulation runner reported that it could not find traces for kmeans-rodinia-2.0-ft, causing the entire step to abort before any simulations completed; this is an infrastructure/data issue, and the other failures are side effects of this first failure or unrelated cancellations.

Failed job(s)

• SASS-Simulation-Docker / Build and Run in the Image: Infrastructure (missing trace data + Slurm task exit 1)
• Tracer-Tool / Test New Traces: Side effect (same missing-trace condition, but this step is a downstream test; primary failure already occurred earlier in SASS-Simulation-Docker)
• SASS-Simulation / Run SASS Simulations: Side effect (cancelled/failed after the Docker job failed first)

Root cause

The literal error that made the failing step exit non-zero is:

srun: error: tgrogers-bigram-08: task 0: Exited with exit code 1
##[error]Process completed with exit code 1.

This appears in SASS-Simulation-Docker_.excerpt.md immediately after the runner printed:

Cannot find traces in any of the paths: ['./hw_run/rodinia_2.0-ft/9.1/kmeans-rodinia-2.0-ft/_i___data_400_txt__g_data_400_result_txt__o/traces']

The simulation runner (main-docker.sh run) launched Slurm jobs for the QV100-SASS configuration, but one of those tasks (task 0 on node tgrogers-bigram-08) exited with code 1 because it could not locate the expected trace directory for kmeans-rodinia-2.0-ft. The runner then propagated that exit code to GitHub Actions (Process completed with exit code 1).

The same “Cannot find traces” message also appears in:

• Tracer-Tool / Test New Traces (the step failed after queuing Slurm arrays, again due to missing trace paths)
• SASS-Simulation-Docker is the first job that actually hit this error and caused the run to fail.

The recent commits (3a602a2f, ca90f215, 330bc2ab) refactored how run directories, data symlinks, and missing binaries are handled in run_hw.py, run_hw_trace.py, and common.prepare_run_dir(). The merge commit (2f1e87da) combined those changes with the multi-GPU worker-pool rewrite from dev (#570). It is plausible that a change to how trace paths are resolved or symlinked (especially in the Docker environment) left the expected hw_run/rodinia_2.0-ft/.../traces directory absent for at least one benchmark (kmeans-rodinia-2.0-ft).

Suggested fix

1. Re-run the workflow first to confirm whether this is a transient infrastructure issue (e.g., missing shared data on the Docker runner). If it passes, no code change is needed.
2. If it reproduces: inspect util/job_launching/common.py (prepare_run_dir) and .github/scripts/main-docker.sh to verify that trace directories are created or symlinked before Slurm jobs start in the Docker image. Specifically check whether kmeans-rodinia-2.0-ft is being skipped (missing binary) but its run directory/trace path is still expected by the simulation runner, causing a hard failure instead of a skip.
3. Ensure that when a benchmark has no trace or binary, the runner either skips it gracefully (as intended by commits ca90f215 and 3a602a2f) or creates an empty placeholder so Slurm tasks do not exit 1 due to missing paths.

_{Automated analysis; check it against the log before acting on it.}

[github-actions]

🤖 CI Runs failed on b86f3d9 — run. AI analysis (Qwopus3.8-27B-Flash-MTP-Q8_0):

CI Failure Analysis

The analyst produced no usable report. See the run log.

_{Automated analysis; check it against the log before acting on it.}