Skip to content

nvidia_configs: add nequip and allegro envs hosting the nequip.net foundation models - #232

Open
OwenPriceSkelly wants to merge 4 commits into
mainfrom
nequip-allegro-envs
Open

OwenPriceSkelly wants to merge 4 commits into
mainfrom
nequip-allegro-envs

Conversation

@OwenPriceSkelly

Copy link
Copy Markdown
Member

Summary

Re-adds nequip and allegro to the NVIDIA catalog (both were pruned on 2026-07-30 because they had no checkpoints to ship). They now host the pretrained foundation models published on nequip.net:

Env Canonical ids
nequip nequip-oam-s, nequip-oam-m, nequip-oam-l, nequip-oam-xl, nequip-mp-l, nequip:custom
allegro allegro-oam-l, allegro-mp-l, allegro:custom
  • Weights: fixed Zenodo uploads (the files nequip.net resolves mir-group/<model>:0.1 to), md5-checked and downloaded into the shared cache at rootstock add time. They're fetched directly because nequip's own nequip.net: loader queries the nequip.net API even when the file is already cached, which fails on offline workers.
  • GPU: setup() compiles each model once with nequip's AOTInductor path (--target ase) for each combination of model, GPU architecture (sm), torch version and nequip version. The result is cached in {cache}/nequip/compiled/, or in a per-user dir under TORCHINDUCTOR_CACHE_DIR when the shared cache is read-only. The compile runs in a child process that moves its own output into place, so a worker killed by the startup timeout still leaves a usable cache entry. setup_kwargs={"compile": False} runs eager instead.
  • CUDA toolkit: when none is loaded, CUDA headers come from the env's cuda-toolkit[cccl,nvcc] wheels, and -lcuda resolves through a temporary symlink to the driver library the process has loaded.
  • CPU: runs eager, including rootstock add's device="cpu" download step. This uses nequip's private NequIPCalculator._from_saved_model, which is why nequip is pinned to <0.20.
  • Custom weights: setup_from_path accepts .nequip.zip, .ckpt, or an already-compiled .nequip.pt2.
  • modal_app.py: adds probe_nequip / probe_allegro, plus extra_index_url, cpu and timeout options on probe_image, so the probes build against the configs' cu128 torch.

Testing

  • Local (macOS CPU): all 7 package md5s match. Every model loaded gave sensible energies, forces and stress on a rattled Cu supercell, and setup_from_path on the cached package matched setup(). The AOTInductor CPU compile also matched eager to within 2e-5 eV.
  • Modal (A10G, 8 CPU, torch 2.10.0+cu128): probe_nequip passes for nequip-oam-s, -l and -xl, and probe_allegro passes for allegro-oam-l, including fresh compiles. Compile times: Allegro-OAM-L about 2 min, NequIP-OAM-L about 4.6 min, NequIP-OAM-XL about 6.4 min.
  • pytest tests/sample_configs passes, including the setup_from_path signature contract. ruff check and ruff format --check are clean on the touched files.
  • Not yet run on an HPC cluster. Staging for the ALCF install next.

Notes for review

  • XL's first compile comes close to the 600 s default startup/verify timeout. The docstring tells users to raise timeout= / --verify-timeout.
  • AOTInductor needs a C++17 host compiler on the compute node. Older system gccs may need a module load or CXX.

🤖 Generated with Claude Code

OwenPriceSkelly and others added 4 commits October 5, 2026 12:59
…undation models

nequip.py serves NequIP-OAM-{S,M,L,XL} and NequIP-MP-L; allegro.py serves
Allegro-OAM-L and Allegro-MP-L; both declare a <family>:custom entry with a
setup_from_path hook (.nequip.zip, .ckpt, or a ready .nequip.pt2).

Package files are pinned Zenodo artifacts, md5-checked and fetched directly
into the shared cache: nequip's own nequip.net loader queries the nequip.net
API even on a cache hit, which fails on offline workers.

On GPU, setup() AOTInductor-compiles each model once per (model, sm arch,
torch, nequip) into {cache}/nequip/compiled, or a per-user dir under
TORCHINDUCTOR_CACHE_DIR when the shared cache is read-only. The compile runs
in a child process that moves its own output into place, so a worker killed
by a startup timeout still leaves a usable cache entry. When no CUDA toolkit
is loaded, CUDA headers come from the env's cuda-toolkit[cccl,nvcc] wheels
and -lcuda resolves through a temporary symlink to the loaded driver. CPU
(including rootstock add's download step) runs eager.

modal_app.py gains probe_nequip/probe_allegro plus extra_index_url, cpu and
timeout options on probe_image, so the probes build against the configs'
cu128 torch. Validated on Modal (A10G, 8 CPU): fresh compiles of
Allegro-OAM-L ~2 min, NequIP-OAM-L ~4.6 min, NequIP-OAM-XL ~6.4 min.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
torch.utils.cpp_extension imports setuptools at module level and uv venvs
do not seed it, so the first GPU compile on the ALCF install died with
ModuleNotFoundError before compiling anything. Modal's base image had it
preinstalled, which is why the probes passed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
AOTInductor builds its C++ wrapper with $CXX (default g++) using gcc-style
flags. Polaris's default PrgEnv exports CXX=nvc++ and CC=nvc, so every
compile on the ALCF install died with "nvc++-Error-Unknown switch". The
compile child now keeps the caller's CXX only if it is gcc >= 9 or clang,
otherwise takes the first qualifying g++-14..g++-9, g++ or clang++ on PATH,
and sets CC to the matching C compiler for Triton's helper build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With CUDA_HOME unset, torch derives it from nvcc on PATH. On Polaris that
nvcc is the NVIDIA HPC SDK wrapper, so the AOTInductor build got
-I .../hpc_sdk/.../compilers/include: the SDK's own immintrin.h (which g++
rejects) and no cuda.h. The compile child now uses CUDA_HOME, CUDA_PATH or
nvcc's toolkit only when it has include/cuda.h, and otherwise sets CUDA_HOME
to the CUDA wheel headers explicitly so torch does not fall back to nvcc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant