B200 / B300 testbed checks

Before signing contracts, it's generally best to test 2 representative nodes from a given provider. I find the following to catch most common issues:

1. Compute test – Repeated BF16 matmuls on every GPU for 1 hour. This checks GPU compute, cooling, and power stability.
2. NCCL all-reduce – intra-node NVLink and inter-node InfiniBand bandwidth.

Use the NGC PyTorch container, for example:

nvcr.io/nvidia/pytorch:<tag>

---

1. Compute test

Run this on each node:

./run_compute_test.sh
python3 analyze_compute_test.py ./compute_test_<host>_<timestamp>

Some example args being set:

DURATION=3600 SIZE=16384 DTYPE=bf16 ./run_compute_test.sh

The script starts one worker per GPU and records nvidia-smi on each node. Each GPU gets a CSV with TFLOPS, temperature, power, and SM clock.

Expected result

Roughly 1.4–1.9 PFLOPS per GPU. We also expect consistency across all GPUs, nodes, and over time.

Pass criteria

A clean run should have:

• Roughly 1.4–1.9 PFLOPS per GPU.
• Similar results across all GPUs and both nodes.
• Less than about 3% droop from the first 10% to the last 10% of the run.
• No thermal or power throttle detected via nvidia-smi.
• Power near TDP and temperature below the throttle point for the full hour.

---

2. NCCL all-reduce

Build nccl-tests once on a shared path:

./build_nccl_tests.sh

Single-node:

./nccl_intra.sh

Two-node run with Slurm:

sbatch nccl_2node.sbatch

Or with MPI:

cp hostfile.example hostfile
# edit hostfile
./nccl_mpi_2node.sh

Before the two-node run, set any necessary inter-node variables:

NCCL_IB_HCA          IB devices, for example mlx5_0,...,mlx5_7
NCCL_SOCKET_IFNAME   management NIC used for bootstrap

Use ibstat, ls /sys/class/infiniband, and ip -br addr to confirm the right names.

Read the busbw column at large message sizes.

Expected NCCL ranges

Test	Fabric	Healthy busbw
Intra-node, 8 GPUs	NVLink 5 / NVSwitch	~700 GB/s rising toward ~880–900 GB/s
Inter-node, CX-7 400G per GPU	InfiniBand NDR	~40–47 GB/s per GPU
Inter-node, CX-8 800G per GPU	InfiniBand XDR	~80–95 GB/s per GPU

Pass criteria

• Intra-node NVLink: busbw should be above 90-95% of expected large-message bandwidth.
• Inter-node InfiniBand: busbw should be above 80-90% of expected line-rate bandwidth.
• Below about 70% usually means wrong HCA, missing GPUDirect RDMA, socket fallback, oversubscription, or bad routing.
• In NCCL_DEBUG=INFO, confirm the inter-node path uses NET/IB with GDRDMA, not NET/Socket.

---

Files

File	Purpose
compute_test.py	Per-GPU compute test with TFLOPS and telemetry logging
run_compute_test.sh	Starts one worker per GPU and records node-level telemetry
analyze_compute_test.py	Summarizes throughput, droop in performance over time, throttling, and pass/review status
build_nccl_tests.sh	Builds nccl-tests with MPI support
nccl_intra.sh	Single-node NVLink all-reduce test
nccl_2node.sbatch	Two-node InfiniBand all-reduce through Slurm
nccl_mpi_2node.sh	Two-node InfiniBand all-reduce through MPI
hostfile.example	MPI hostfile template