[Austin243]

Before: a model that returns NaN or inf during MD can leave damage that outlives the job. In checks on stub calculators:

• With a model that accepts NaN positions, a NaN frame is written before its checkpoint refuses it. The next job replays correctly on a good GPU but keeps that frame, because a frame with that number exists, and later events read it. One barrier moved from 0.06100 to 0.06066 eV with nothing flagged, and other events became neb_failed, unresolved, or relaxation_failed.
• A first evaluation that gives NaN saves a step-0 checkpoint no later job can use.
• NaN stress on the last step of an interval saves a checkpoint with finite atoms and a NaN barostat, which stops every later job. This and the first-evaluation case also happen with EMT's real neighbor list.
• At constant pressure, on a stub that accepts a NaN cell, the cell raises in the bond detector and marks the trajectory failed for good. EMT's neighbor list raises on a NaN cell inside MD instead, so the trajectory stays running.
• inf passes the checkpoint's comparison, so a run at constant volume with inf forces can finish with exit 0 and most of its frames bad.

After: the job stops before any of that is saved.

• Condition: after each MD interval, or before a new run's first checkpoint, a position, momentum, cell vector, or floating-point integrator array (accel, and at constant pressure p_eps, force_eps, and gamma_mod) is NaN or inf.
• Action: the job exits non-zero before that state is observed, written, or checkpointed. The trajectory stays at its last saved phase: running, new if it has no checkpoint yet, or refining with an empty queue if the fault comes in the first interval after a refinement. The next job repeats from the last checkpoint or starts over. Nothing else is cancelled.

There is no option, because a state like that cannot continue, and a healthy run is unchanged.

How: _require_finite in run.py, called at those two places in _run_segment, reads runtime.snapshot(). It evaluates no model and takes 18 microseconds on 288 atoms. It raises outside _fatal, so it does not mark the run failed. run.py is not in the checkpoint's environment record, so running trajectories resume with no warning. That is 19 added lines in run.py. A smaller check of only positions, momenta, and cell, plus forces at the start, misses the NaN-stress cases, where the atoms stay finite.

Test: one, written before the code, with nine cases: first evaluation, last step of an interval, and inside an interval, each for forces at constant volume, forces at constant pressure, and stress at constant pressure. All nine fail on main. 349 passed, 25 skipped; ruff is clean.

Checked outside the test suite, with real reactionflow run processes and BAOAB MD, on stub calculators unless noted:

• 61 faults: NaN or inf in forces or stress, from the first evaluation, on the last step of an interval, or inside one, for one evaluation or for good, including the interval that would confirm an event, at constant volume (isolated and periodic) and hydrostatic constant pressure. All 61 stop with exit 1, with no bad frame, no bad checkpoint, and the run not marked failed. Another job on the same bad GPU loses nothing. A job on a good GPU ends with frames bitwise equal to, and events the same as, a run that never met the fault. On main, 6 of the 61 do this: 29 can never finish even on a good GPU, 2 more after another job on the bad GPU, 34 keep a bad frame, and 6 with inf forces finish with exit 0 and 28 to 60 of their 61 frames bad.
• A model that raises on NaN input was already safe on main for faults inside an interval, in the six cases run. A NaN first evaluation damages it there too.
• The same 61 on this, Refine a trajectory's barriers after its MD when the campaign asks for it #82, and Stop refining one barrier at a time limit, four hours by default #83 merged, with refine_after_md on and off: the same results, against those two merged without this.
• 10 more under hydrostatic: false: the same result, where main cannot finish 9 of them and keeps a bad frame in the tenth. All-inf stress inside an interval takes about 4.5 minutes to stop, on main too, because SciPy's expm spends about 73 s on each such matrix before this check can run (SciPy 1.18.1 on an arm64 Mac).
• EMT's real neighbor list, Cu with H at 1 GPa, 6 faults: all finish identical to the control on a good GPU. On main, the 2 that leave a NaN barostat cannot finish, and none is marked failed. The 4 inside an interval stop here with EMT's own error from inside MD, as on main.
• A fault in the first interval after a refinement, 5 cases: the phase stays refining with an empty queue, and a good GPU gives the control.
• NaN in the energy for one evaluation does not stop MD: 6 of 6 finish with exit 0 and match the control. An energy that stays NaN does not stop MD either and the frames match, but every saddle after it ends neb_failed with exit 0, as on main.
• A custom runtime with string, bytes, or structured arrays in its integrator state runs as on main. A NaN in a floating-point array there stops the job.
• Healthy runs, against main and against Refine a trajectory's barriers after its MD when the campaign asks for it #82 and Stop refining one barrier at a time limit, four hours by default #83 merged with the option on and off: periodic wells at constant volume and constant pressure, and a chain model with stepwise events. Frames bitwise, events, statuses, and barriers are identical. So are EMT on acetonitrile and water with no metal, and EMT on Cu with H at 1 GPa, against main.
• Model evaluations per run are unchanged, at constant volume and constant pressure, with calculators that compute everything at once and ones that compute only what is asked.
• A trajectory started on main and killed resumes here, and one started here resumes on main, with no environment warning and frames bitwise equal to main's.
• A NaN or a raise tied to a geometry, on every GPU, under refine_after_md: the queue waits through resubmissions, and option false gives the same 23 results before MD stops, as on Refine a trajectory's barriers after its MD when the campaign asks for it #82. The NaN frame those two keep is gone.
• status reports the error while the trajectory is stopped, and scan still matches every recorded event. Neither changes the run.

Limits:

• This repairs nothing an earlier version wrote. In a run resumed here, a NaN frame stays, a poisoned checkpoint stops each job until the trajectory's directory is deleted, and a trajectory already marked failed stays failed. The message on a poisoned checkpoint varies: this one if the model accepts NaN input, otherwise whatever it or ASE raises first, such as You have 0 lattice vectors: volume not defined.
• Refinement is not checked. NaN from the model during refinement is saved as that event's failed result, as on main, and the job then stops at the next MD interval.
• Finite but wrong model output is not caught.
• Calling run() again on the same Python object after it raises can continue from the wrong atoms, as on main with a model that raises, so the docs now say to reopen the run. The CLI is not affected.
• docs/how-it-works.md still says a trajectory stops only if it cannot save its state, which is already loose on main. Refine a trajectory's barriers after its MD when the campaign asks for it #82 corrects it.

Not run: a real MLIP on a GPU. Whether UMA, MACE, AIMNet2, or ANI ever return NaN without raising, or accept NaN positions or a NaN cell, is not known. AIMNet2's preflight refuses non-finite values on a new trajectory (read, not run). ASE 3.28, which requirements/perlmutter-ani1xnr.txt pins, was not run; its integrator source initializes these arrays the same way as 3.29 (read).

Merges with #82 and #83 in every order with no conflicts and the same tree; 356 passed, 25 skipped on the merge.