yury-lysogorskiy · Oct 29, 2026 · Oct 29, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,43 @@
 Notable changes of grace-runtime, grace-runtime-cuda12 and grace-runtime-rocm6 (they share one version). Each
 release has a section "Changed defaults" when a default behaves differently.
 
+## [0.1.1](https://github.com/ICAMS/grace-runtime/releases/tag/0.1.1) (2026-10-29)
+
+Bug fixes and documentation.
+
+### Fixed
+
+- NVIDIA and AMD GPUs: GRACE-1L with 33.5 million atoms (2^25) or more stopped the whole process instead of running.
+  It now runs up to the GPU memory (A100 80 GB: 41 million atoms) and beyond that raises the usual "does not fit the
+  GPU memory" error.
+- The PyPI page of `grace-runtime` shows the README.
+- macOS: `pip install "grace-runtime[torchsim]"` installed PyTorch from PyPI, whose OpenMP runtime clashes with the
+  one of grace-runtime (`OMP: Error #15`). On macOS the extra now installs nothing; install PyTorch from conda-forge
+  (see the installation guide).
+- macOS: `grace-runtime info` says that the GPU modules are Linux-only, instead of listing where it looked for the
+  NVIDIA driver.
+
+### LAMMPS (`grace-runtime` branch of the LAMMPS fork)
+
+- CPU styles: `GRACE_BLAS=AUTO` uses a BLAS only if it is MKL, OpenBLAS, BLIS or Accelerate, otherwise the built-in
+  kernel. Other BLAS libraries (for example ATLAS) made GRACE-3L about 4x slower, and on HPE Cray systems the BLAS
+  linked by the compiler made the link fail.
+- CMake warns when the CPU styles are built with GCC older than 12: with GCC 11 GRACE-3L runs about 37 % slower,
+  with GCC 9 GRACE-1L about 4.6x and GRACE-3L about 6.6x slower.
+
+### Documentation
+
+- LAMMPS guide: build the fork for CPU, NVIDIA and AMD GPUs (with an example for LUMI), get a model, and run a first
+  input that also prints the extrapolation grade.
+- Windows through WSL2; GCC 12 or newer for source builds.
+- Benchmarks: time per atom and the largest system per GPU (L40S, A100, MI250X), and time per atom on a CPU.
+
+### Changed defaults
+
+- LAMMPS `GRACE_BLAS=AUTO` (see above): with a BLAS other than MKL, OpenBLAS, BLIS or Accelerate the CPU styles use
+  the built-in kernel. `-D GRACE_BLAS=ON` uses any BLAS that provides `cblas_sgemm`.
+- macOS: the `torchsim` extra installs no packages.
+
 ## [0.1.0](https://github.com/ICAMS/grace-runtime/releases/tag/0.1.0) (2026-10-29)
 
 First release on PyPI: `grace-runtime` (CPU), `grace-runtime-cuda12` (NVIDIA) and `grace-runtime-rocm6` (AMD). It

diff --git a/README.md b/README.md
@@ -3,13 +3,13 @@
 Fast, efficient execution of GRACE-1L, GRACE-2L and GRACE-3L interatomic potentials without TensorFlow, PyTorch or JAX.
 CPU execution is standalone; GPU execution uses Kokkos for NVIDIA CUDA and AMD HIP.
 Use ASE-compatible calculators, batch processing, an optional TorchSim adapter or the shared kernels in LAMMPS with MPI.
-PACE (ACE) potentials fitted with pacemaker run from their `.yaml` ([PACE models](docs/pace.md)).
+PACE (ACE) potentials fitted with pacemaker run from their `.yaml` ([PACE models](https://github.com/ICAMS/grace-runtime/blob/master/docs/pace.md)).
 
-**[Read the manual](docs/index.md)** · [Installation](docs/installation.md) · [API reference](docs/api.md)
+**[Read the manual](https://github.com/ICAMS/grace-runtime/blob/master/docs/index.md)** · [Installation](https://github.com/ICAMS/grace-runtime/blob/master/docs/installation.md) · [API reference](https://github.com/ICAMS/grace-runtime/blob/master/docs/api.md)
 
 ## Install
 
-Install one package from PyPI ([source builds](docs/installation.md#cpu-from-source) for other targets):
+Install one package from PyPI ([source builds](https://github.com/ICAMS/grace-runtime/blob/master/docs/installation.md#cpu-from-source) for other targets):
 
 | Backend | Install |
 | --- | --- |
@@ -18,7 +18,7 @@ Install one package from PyPI ([source builds](docs/installation.md#cpu-from-sou
 | AMD | `pip install grace-runtime-rocm6` |
 
 GPU packages install the matching CPU package automatically.
-See the manual for [GPU requirements](docs/backends.md) and [environment setup](docs/installation.md#python-environment).
+See the manual for [GPU requirements](https://github.com/ICAMS/grace-runtime/blob/master/docs/backends.md) and [environment setup](https://github.com/ICAMS/grace-runtime/blob/master/docs/installation.md#python-environment).
 
 ## First calculation
 
@@ -38,22 +38,22 @@ Named models download on first use; local `kokkos.npz` files work too.
 
 ## Guides
 
-The [manual](docs/index.md#guides) covers [batching and MD](docs/usage.md), [relaxation](docs/relax.md),
-[properties](docs/properties.md), [dispersion](docs/dispersion.md), [PACE models](docs/pace.md), [TorchSim](docs/torchsim.md) and
-[development](docs/development.md). An interactive tour is in the [ASE notebook](python/examples/ase/grace_runtime_ase.ipynb).
+The [manual](https://github.com/ICAMS/grace-runtime/blob/master/docs/index.md#guides) covers [batching and MD](https://github.com/ICAMS/grace-runtime/blob/master/docs/usage.md), [relaxation](https://github.com/ICAMS/grace-runtime/blob/master/docs/relax.md),
+[properties](https://github.com/ICAMS/grace-runtime/blob/master/docs/properties.md), [dispersion](https://github.com/ICAMS/grace-runtime/blob/master/docs/dispersion.md), [PACE models](https://github.com/ICAMS/grace-runtime/blob/master/docs/pace.md), [TorchSim](https://github.com/ICAMS/grace-runtime/blob/master/docs/torchsim.md) and
+[development](https://github.com/ICAMS/grace-runtime/blob/master/docs/development.md). An interactive tour is in the [ASE notebook](https://github.com/ICAMS/grace-runtime/blob/master/python/examples/ase/grace_runtime_ase.ipynb).
 
 ## LAMMPS
 
 The [LAMMPS fork](https://github.com/yury-lysogorskiy/lammps/tree/grace-runtime) supplies pair styles
 `grace/runtime` (CPU) and `grace/runtime/kk` (GPU), which take the GRACE-1L / 2L / 3L model from the `kokkos.npz` given to
 `pair_coeff`, and `pace/runtime(/kk)` for PACE models. Installing the Python package does not install LAMMPS:
-[build and run it](docs/lammps.md).
+[build and run it](https://github.com/ICAMS/grace-runtime/blob/master/docs/lammps.md).
 
 ## Licence
 
-Academic Software Licence (ASL, [LICENSE.md](LICENSE.md)): academic non-commercial use. For commercial use contact
+Academic Software Licence (ASL, [LICENSE.md](https://github.com/ICAMS/grace-runtime/blob/master/LICENSE.md)): academic non-commercial use. For commercial use contact
 ACEworks GmbH, contact@aceworks.works. Bundled third-party code keeps its own licence
-([THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)). Changes: [CHANGELOG.md](CHANGELOG.md).
+([THIRD_PARTY_NOTICES.md](https://github.com/ICAMS/grace-runtime/blob/master/THIRD_PARTY_NOTICES.md)). Changes: [CHANGELOG.md](https://github.com/ICAMS/grace-runtime/blob/master/CHANGELOG.md).
 
 ## Reference
 

diff --git a/docs/benchmarks.md b/docs/benchmarks.md
@@ -0,0 +1,22 @@
+# Benchmarks
+
+How fast grace-runtime evaluates the GRACE foundation models, and how many atoms fit on one GPU.
+
+![Time per atom and largest system on one GPU, for GRACE-1L, 2L and 3L on NVIDIA L40S, NVIDIA A100 and AMD MI250X](images/benchmarks-gpu-light.png#only-light)
+![Time per atom and largest system on one GPU, for GRACE-1L, 2L and 3L on NVIDIA L40S, NVIDIA A100 and AMD MI250X](images/benchmarks-gpu-dark.png#only-dark)
+
+![Time per atom on one CPU with 1 and 16 cores, for GRACE-1L, 2L and 3L](images/benchmarks-cpu-light.png#only-light)
+![Time per atom on one CPU with 1 and 16 cores, for GRACE-1L, 2L and 3L](images/benchmarks-cpu-dark.png#only-dark)
+
+## How we measured
+
+| | |
+|---|---|
+| Models | GRACE-1L-OMAT-large-base, GRACE-2L-OMAT-large-base, GRACE-3L-OMAT-large (cutoff 6 Å) |
+| System | bcc crystal (a = 3.165 Å) with 25 elements (Al to Os) in random order, atoms displaced by about 0.02 Å; about 58 neighbours per atom |
+| Calculation | `GRACERuntimeCalculator` with default settings; each call computes energy, forces and stress |
+| Time per atom | median of 3 calls after the first; the lowest value over system sizes from about 1000 atoms up. It changes little above about 16000 atoms (GPU) or 2000 atoms (CPU) |
+| Largest system | double the number of atoms until a structure does not fit the GPU memory, then bisect; the largest structure that runs, each size in a new process |
+| GPUs | NVIDIA L40S 48 GB, NVIDIA A100 80 GB PCIe, AMD Instinct MI250X: one GCD (half of the card) with 64 GB |
+| CPU | Intel Xeon Gold 6526Y (16 cores per socket); 1 thread, or 16 threads on one socket (`OMP_NUM_THREADS`) |
+| Software | grace-runtime 0.1 wheels from PyPI (`grace-runtime`, `grace-runtime-cuda12`, `grace-runtime-rocm6`) |
diff --git a/docs/images/benchmarks-cpu-dark.png b/docs/images/benchmarks-cpu-dark.png
diff --git a/docs/images/benchmarks-cpu-light.png b/docs/images/benchmarks-cpu-light.png
diff --git a/docs/images/benchmarks-gpu-dark.png b/docs/images/benchmarks-gpu-dark.png
diff --git a/docs/images/benchmarks-gpu-light.png b/docs/images/benchmarks-gpu-light.png
diff --git a/docs/index.md b/docs/index.md
@@ -50,6 +50,7 @@ print([r["energy"] for r in results])
 | Use TorchSim integrators and optimizers | [TorchSim](torchsim.md) |
 | Build LAMMPS and run GRACE or PACE in it | [LAMMPS](lammps.md) |
 | Manage memory, threads and fallbacks | [Performance and diagnostics](advanced.md) |
+| See speed and the largest systems per GPU | [Benchmarks](benchmarks.md) |
 | Look up parameters, defaults, CLI and results | [API reference](api.md) |
 | Build and test | [Development](development.md) |
 

diff --git a/docs/installation.md b/docs/installation.md
@@ -79,8 +79,13 @@ micromamba install -c conda-forge llvm-openmp
 ```
 
 The macOS wheel ships its OpenMP runtime and shares it with conda-forge packages in the same process (NumPy, PyTorch).
-PyTorch from PyPI ships its own, and the two abort with `OMP: Error #15`: for TorchSim on macOS, use conda-forge
-PyTorch, which shares the environment's OpenMP runtime.
+PyTorch from PyPI ships its own, and the two abort with `OMP: Error #15`. So on macOS the `torchsim` extra installs
+nothing; install PyTorch from conda-forge, which shares the environment's OpenMP runtime, and then TorchSim:
+
+```bash
+micromamba install -c conda-forge pytorch
+pip install grace-runtime torch-sim-atomistic
+```
 
 ### GPU source builds
 
@@ -93,7 +98,7 @@ For editable installs and library tests, see [development](development.md).
 | Feature | Install |
 | --- | --- |
 | Phonons | `pip install "grace-runtime[phonons]"` |
-| TorchSim adapter (Python >= 3.12; on a CUDA 12 driver torch from the CUDA 12 index, see [TorchSim](torchsim.md)) | `pip install "grace-runtime[torchsim]"` |
+| TorchSim adapter (Python >= 3.12; on a CUDA 12 driver torch from the CUDA 12 index, see [TorchSim](torchsim.md); on macOS see [macOS](#macos)) | `pip install "grace-runtime[torchsim]"` |
 | Tests | `pip install "grace-runtime[test]"` |
 | cuBLAS for the NVIDIA package (GRACE-3L up to ~15 % faster, UQ slightly faster) | `pip install "grace-runtime-cuda12[cublas]"` |
 

diff --git a/docs/lammps.md b/docs/lammps.md
@@ -30,7 +30,7 @@ cd lammps
 ## 2. Build
 
 The pair styles are part of the `ML-PACE` package. When CMake configures the build, it downloads grace-runtime
-0.1.0 from GitHub, so the configure step needs network access. It also downloads the ML-PACE library. Pick one of the
+0.1.1 from GitHub, so the configure step needs network access. It also downloads the ML-PACE library. Pick one of the
 three recipes; run each from the `lammps` folder.
 
 ### CPU
@@ -45,6 +45,12 @@ cmake --build build-cpu -j 8
 binary on the same kind of CPU. `BUILD_OMP=yes` enables the OpenMP threads that the styles use inside each MPI rank;
 the `OPENMP` package is not needed.
 
+GRACE-3L uses a BLAS for its matrix products only if the BLAS that CMake finds is MKL, OpenBLAS, BLIS or Accelerate;
+otherwise it uses a built-in kernel. Other BLAS libraries, such as ATLAS, are slower than the built-in kernel. The
+configure step prints the choice in a line `CPU GRACE styles: ...`. To pick a BLAS, add for example
+`-D BLA_VENDOR=Intel10_64lp_seq` (sequential MKL) or `-D BLA_VENDOR=OpenBLAS`. `-D GRACE_BLAS=ON` uses any BLAS
+that provides `cblas_sgemm` and stops if there is none; `-D GRACE_BLAS=OFF` always uses the built-in kernel.
+
 ### NVIDIA GPU (Kokkos CUDA)
 
 You need the CUDA toolkit 12.2 or newer, 12.8 for Blackwell GPUs (`nvcc` on `PATH`).
@@ -89,6 +95,33 @@ On HPE Cray systems (`PrgEnv-amd` with `cray-mpich`), also add `-D GRACE_BLAS=OF
 a BLAS at configure time that `hipcc` does not link, and the link then fails with `undefined symbol: cblas_sgemm`. The
 setting affects only the CPU styles, which then use their built-in matrix product.
 
+Example for [LUMI](https://docs.lumi-supercomputer.eu/) (MI250X, `PrgEnv-amd`). Configure on a login node, because
+the configure step needs network access; build in a batch job and run on a GPU node. On the login node, ROCm's CMake
+files warn that `AMDGPU_TARGETS` is not set and no GPU was detected; the warning is harmless, the binary is built
+for gfx90a:
+
+```bash
+module --force purge
+module load LUMI/25.03 partition/G
+module load rocm/6.3.4 buildtools/25.03 PrgEnv-amd cray-mpich
+
+cmake -S cmake -B build-hip -D CMAKE_BUILD_TYPE=Release -D CMAKE_CXX_COMPILER=hipcc \
+  -D PKG_KOKKOS=yes -D Kokkos_ENABLE_HIP=yes -D Kokkos_ARCH_AMD_GFX90A=yes \
+  -D GPU_TARGETS=gfx90a -D CMAKE_CXX_FLAGS="-munsafe-fp-atomics" -D BUILD_MPI=yes -D PKG_ML-PACE=yes \
+  -D GRACE_BLAS=OFF
+sbatch -A project_<number> -p small-g -N 1 -n 1 -c 8 --gpus=1 -t 00:30:00 --wrap "cmake --build build-hip -j 8"
+```
+
+The build takes about 12 minutes; its output goes to `slurm-<jobid>.out`.
+
+One GCD (half of an MI250X) per run, with the input of [step 4](#4-run):
+
+```bash
+srun -A project_<number> -p small-g -N 1 -n 1 -c 7 --gpus=1 -t 00:30:00 \
+  build-hip/lmp -k on g 1 -sf kk -pk kokkos newton on neigh half -in in.lmp \
+  -var model ~/.cache/grace/GRACE-2L-OMAT-medium-ft-E/kokkos.npz
+```
+
 Other AMD GPUs ([GPU backends](backends.md#amd-rocm)):
 
 | GPU | Flags |
@@ -98,10 +131,10 @@ Other AMD GPUs ([GPU backends](backends.md#amd-rocm)):
 
 ### Without network access
 
-Download grace-runtime 0.1.0 on a machine with network access, copy it over and pass its path:
+Download grace-runtime 0.1.1 on a machine with network access, copy it over and pass its path:
 
 ```bash
-git clone --depth 1 -b 0.1.0 https://github.com/ICAMS/grace-runtime.git
+git clone --depth 1 -b 0.1.1 https://github.com/ICAMS/grace-runtime.git
 cmake -S cmake -B build-cpu ... -D LOCAL_GRACE_RUNTIME=$PWD/grace-runtime
 ```
 
@@ -130,7 +163,8 @@ set).
 ## 4. Run
 
 Save this input as `in.lmp`. It is a bcc W/Mo alloy of 1024 atoms; `pair_coeff` maps LAMMPS atom types 1 and 2 to W
-and Mo, and the model file is given on the command line as the variable `model`:
+and Mo, and the model file is given on the command line as the variable `model`. The input also prints the largest
+extrapolation grade γ of all atoms (the model's uncertainty) and writes γ per atom to `uq.dump`:
 
 ```text
 units        metal
@@ -146,12 +180,23 @@ mass         2 95.95
 pair_style   grace/runtime
 pair_coeff   * * ${model} W Mo
 
+fix          gamma all pair 20 grace/runtime gamma 1
+compute      max_gamma all reduce max f_gamma
+
 velocity     all create 300 4928459
 fix          1 all nvt temp 300 300 0.1
+thermo_style custom step temp pe press c_max_gamma
 thermo       20
+dump         uq all custom 100 uq.dump id type x y z f_gamma
 run          100
 ```
 
+`fix pair` asks the pair style for γ every 20 steps, the same interval as `thermo`. γ needs a model with an
+uncertainty head (GRACE-2L-OMAT-medium-ft-E has one; see [uncertainty quantification](usage.md#uncertainty-quantification)).
+For a model without one, leave out the `fix pair`, `compute` and `dump` lines and `c_max_gamma`. The fields
+`atomic_sigma` and `gmm_cluster` are available in the same way; see the
+[`pair_style grace/runtime` documentation](https://github.com/yury-lysogorskiy/lammps/blob/grace-runtime/doc/src/pair_grace_runtime.rst).
+
 CPU, 4 MPI ranks with 4 OpenMP threads each:
 
 ```bash
@@ -178,23 +223,26 @@ The style prints which model kind it loaded. On the CPU (`grace/runtime`):
 ```text
 [grace/runtime] loading the GRACE-2L model .../GRACE-2L-OMAT-medium-ft-E/kokkos.npz
 [GRACE-2L] lmax 4 | L1 42 ch, A1/AA/AAA/AAAA 25/141/107/378 | I 4x32 | L2 32 / 64 ch, ... | spline N 1024
-   Step          Temp          E_pair         E_mol          TotEng         Press
-         0   300           -12248.756      0             -12209.086      40140.971
+   Step          Temp          PotEng         Press       c_max_gamma
+         0   300           -12248.756      40140.972      0.42767153
+        20   184.01952     -12233.328      43814.062      0.58249744
 ```
 
+γ is the distance of an atom's environment from the training data, divided by a threshold: below 1 the atom is
+inside the region the model was trained on, above 1 it extrapolates. Here all atoms stay below 1.
+
 On the GPU (`grace/runtime/kk`), also the element mapping and the chunk size chosen from the free GPU memory:
 
 ```text
 [grace/runtime/kk] loading the GRACE-2L model .../GRACE-2L-OMAT-medium-ft-E/kokkos.npz
 [grace/runtime/kk] LAMMPS type 1 -> W (model index 83)
 [grace/runtime/kk] LAMMPS type 2 -> Mo (model index 43)
 [GRACE-2L/KK] chunksize auto: 1126 atoms per chunk (...)
-   Step          Temp          E_pair         E_mol          TotEng         Press
-         0   300           -12248.756      0             -12209.086      40140.93
 ```
 
-The energy at step 0 is the same on the CPU and the GPU. Both evaluate the network in fp32, in a different order, so
-the last digits and later steps of the trajectory can differ slightly.
+With `fix pair` in the input, LAMMPS also warns that it switches to legacy (host) communication and sorting for
+Kokkos; this is expected and harmless. The thermo output follows, as on the CPU. The energy at step 0 is the same on the CPU and the GPU. Both evaluate the
+network in fp32, in a different order, so the last digits and later steps of the trajectory can differ slightly.
 
 Wrong input stops LAMMPS with an error that names the problem: a PACE `.yaml` in `grace/runtime`, a GRACE model in
 `pace/runtime`, a keyword that does not fit the model, or units other than `metal`.

diff --git a/docs/torchsim.md b/docs/torchsim.md
@@ -2,7 +2,8 @@
 
 [Installation](installation.md#optional-dependencies) · [API reference](api.md#torchsim-adapter)
 
-Install the `torchsim` extra with Python >= 3.12. This complete example uses the default device selection:
+Install the `torchsim` extra with Python >= 3.12 (on macOS, PyTorch from conda-forge instead: see
+[macOS](installation.md#macos)). This complete example uses the default device selection:
 
 ```python
 import numpy  # load before torch