Skip to content

Stop refining one barrier at a time limit, four hours by default - #83

Merged
Austin243 merged 4 commits into
mainfrom
max-hours-per-barrier
Nov 5, 2026
Merged

Austin243 merged 4 commits into
mainfrom
max-hours-per-barrier

Conversation

@Austin243

@Austin243 Austin243 commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Before: only step counts limit a refinement, and on 288 atoms those ran up to 27 hours. Resubmitting does not help: an interrupted refinement starts over, so a job shorter than the refinement repeats it forever.

After: once one event's refinement has run for max_hours_per_barrier hours, it ends as time_limit, with no barrier and its band as far as it got, and the trajectory carries on. The key goes at the top of the campaign. The default is 4, null turns the limit off, and 0 is refused. Steps of an event already refined keep their results, and its remaining steps also end as time_limit, with no images. MD is never stopped, and nothing is cancelled except that one refinement. status counts the refinements that hit the limit.

The clock counts the time a refinement has run, not queue time or MD between events. It carries across resubmissions in state.json, saved once a minute, so work lost to an interruption counts against the limit, less up to a minute per job. A job that refines for under a minute adds nothing to the count. A save that fails is skipped until the next minute, so it cannot become the refinement's result. The key is not part of the trajectory contract: running trajectories keep their hash and take the 4-hour default when they resume on this code, and raising the limit on resubmission applies to the event in progress and later ones. Results are never overwritten, so an event that already ended as time_limit is not rerun.

Why 4 hours, from the refinements I could time in the saved runs (288-atom acetonitrile at 5 to 35 GPa, UMA, SSNEB, one GPU, older commits):

Outcome Hours
Connected saddle, so a barrier 1.07, 1.76
One mode, not connected 2.02, 2.17
No mode 1.53, 3.33
The nine longest that finished, none reaching a saddle (4 NEB or CI-NEB not converged, 5 converged with no imaginary mode) 8.5 to 26.7

The first three rows are file times, which can include time between jobs. The last row is run time from the job logs: seven of the nine ran inside one uninterrupted job, and the other two each crossed two jobs, with last attempts of 15.3 and 10.9 hours, still past 4. Four hours keeps both barriers and would have saved about 116 hours of GPU time on those nine.

The six refinements in the first three rows are 700 K runs at 20 GPa, and the nine longest that finished are at 2000 K or 2600 K. None of the 80 results from runs at 2000 K or hotter reached a connected saddle, at any length, so these records cannot say how long one would need there. Elsewhere, the 15-atom xTB example takes seconds per event and EC on Li(001) took 9 minutes on 4 CPU cores. More atoms, more images, a slower model, or a CPU can need more than 4 hours.

Standalone SSNEB bands on the same 288-atom system with UMA give a wider sample, including events from hot MD, quenched first. They are not ReactionFlow refinements: they have 9 interior images where a refinement has 5, and a tighter force test. Of 180 with time stamps, the median is 1.57 hours and 7 pass 4 hours. Of the 52 whose saddle check found a connected saddle, 2 pass 4 hours: N01 at 30 and 35 GPa, 5.45 and 5.33 hours. Stamps on a resumed band cover only its last attempt, so counted by steps about 8 of the 52 would pass 4 hours at 9 interior images, and none at 5. That is scaling, not a run. A campaign that raises the image count should raise the limit too.

How: _time_limited in run.py wraps the calculator provider. It checks the clock before each model load and, through a small wrapper around each calculator's calculate, before each evaluation. BarrierTimeLimit derives from BaseException because the Hessian and the connectivity check record any Exception as their own failure and carry on. refine_pathway catches it in the two places that already catch failures, which keeps the partial band and earlier steps.

Limits: a single evaluation that never returns is not stopped. The provider must yield an ASE Calculator, as its type says and the built-in adapters enforce. A subclass that evaluates without going through calculate is checked only when its lease starts.

Tests: three, written before the code: the limit landing in a relaxation, the Hessian, and the connectivity check; jobs too short to finish a barrier; and the campaign key. 345 passed, 25 skipped; ruff is clean.

Checked outside the test suite, with real reactionflow run processes and BAOAB MD unless noted, where "main" is the commit before #84 and #82:

  • Default limit, stateless double-well potential: frames, events, statuses, and barriers are identical to main.
  • A limit short enough to stop all 33 events: MD frames are still bitwise identical to main and the same events are found, pausing and with Refine a trajectory's barriers after its MD when the campaign asks for it #82. status reports all 33.
  • Jobs killed with SIGKILL at 80 s under a 130 s limit, on bands that would take hours: the count carried across each kill, and all five slow events ended as time_limit over 10 jobs.
  • A three-step event, in a harness, with the limit at each stage: validated steps keep their barriers and every remaining step reads time_limit, with no images.
  • EMT on Cu with H at 1 GPa: the limit stops the refinement in relaxation, recorded with method ssneb, with MD frames unchanged. That event never reaches a band, so a stateless SSNEB band was checked in a harness: no evaluation escaped the clock, through the saddle check.
  • A state.json that cannot be saved during a refinement: one failed save leaves the event's result unchanged. A lasting fault stops the job at the next save outside the check, as on main, and the resubmission gives main's barrier.
  • A barrier already out of time loads no model, and the next event gets the whole limit.
  • A trajectory started on main resumes here, and state written here opens on main.

Not run: a real MLIP on a GPU, or this on a cluster.

#84 and #82 are merged, and this merges with them with no conflicts. All three together: 356 passed, 25 skipped; ruff is clean.

Those images came from the step before and contradicted the step's bonds and frames. Correct the
timing in the docs from the job logs, which cover all nine of the longest refinements.
@Austin243
Austin243 marked this pull request as ready for review November 5, 2026 18:35
@Austin243
Austin243 merged commit 0ce9706 into main Nov 5, 2026
4 checks passed
@Austin243
Austin243 deleted the max-hours-per-barrier branch November 5, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant