Repository navigation
Stop refining one barrier at a time limit, four hours by default - #83
Merged
Merged
Conversation
…t, refuse a limit of 0
Those images came from the step before and contradicted the step's bonds and frames. Correct the timing in the docs from the job logs, which cover all nine of the longest refinements.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: only step counts limit a refinement, and on 288 atoms those ran up to 27 hours. Resubmitting does not help: an interrupted refinement starts over, so a job shorter than the refinement repeats it forever.
After: once one event's refinement has run for
max_hours_per_barrierhours, it ends astime_limit, with no barrier and its band as far as it got, and the trajectory carries on. The key goes at the top of the campaign. The default is 4,nullturns the limit off, and 0 is refused. Steps of an event already refined keep their results, and its remaining steps also end astime_limit, with no images. MD is never stopped, and nothing is cancelled except that one refinement.statuscounts the refinements that hit the limit.The clock counts the time a refinement has run, not queue time or MD between events. It carries across resubmissions in
state.json, saved once a minute, so work lost to an interruption counts against the limit, less up to a minute per job. A job that refines for under a minute adds nothing to the count. A save that fails is skipped until the next minute, so it cannot become the refinement's result. The key is not part of the trajectory contract: running trajectories keep their hash and take the 4-hour default when they resume on this code, and raising the limit on resubmission applies to the event in progress and later ones. Results are never overwritten, so an event that already ended astime_limitis not rerun.Why 4 hours, from the refinements I could time in the saved runs (288-atom acetonitrile at 5 to 35 GPa, UMA, SSNEB, one GPU, older commits):
The first three rows are file times, which can include time between jobs. The last row is run time from the job logs: seven of the nine ran inside one uninterrupted job, and the other two each crossed two jobs, with last attempts of 15.3 and 10.9 hours, still past 4. Four hours keeps both barriers and would have saved about 116 hours of GPU time on those nine.
The six refinements in the first three rows are 700 K runs at 20 GPa, and the nine longest that finished are at 2000 K or 2600 K. None of the 80 results from runs at 2000 K or hotter reached a connected saddle, at any length, so these records cannot say how long one would need there. Elsewhere, the 15-atom xTB example takes seconds per event and EC on Li(001) took 9 minutes on 4 CPU cores. More atoms, more images, a slower model, or a CPU can need more than 4 hours.
Standalone SSNEB bands on the same 288-atom system with UMA give a wider sample, including events from hot MD, quenched first. They are not ReactionFlow refinements: they have 9 interior images where a refinement has 5, and a tighter force test. Of 180 with time stamps, the median is 1.57 hours and 7 pass 4 hours. Of the 52 whose saddle check found a connected saddle, 2 pass 4 hours: N01 at 30 and 35 GPa, 5.45 and 5.33 hours. Stamps on a resumed band cover only its last attempt, so counted by steps about 8 of the 52 would pass 4 hours at 9 interior images, and none at 5. That is scaling, not a run. A campaign that raises the image count should raise the limit too.
How:
_time_limitedinrun.pywraps the calculator provider. It checks the clock before each model load and, through a small wrapper around each calculator'scalculate, before each evaluation.BarrierTimeLimitderives fromBaseExceptionbecause the Hessian and the connectivity check record anyExceptionas their own failure and carry on.refine_pathwaycatches it in the two places that already catch failures, which keeps the partial band and earlier steps.Limits: a single evaluation that never returns is not stopped. The provider must yield an ASE
Calculator, as its type says and the built-in adapters enforce. A subclass that evaluates without going throughcalculateis checked only when its lease starts.Tests: three, written before the code: the limit landing in a relaxation, the Hessian, and the connectivity check; jobs too short to finish a barrier; and the campaign key. 345 passed, 25 skipped; ruff is clean.
Checked outside the test suite, with real
reactionflow runprocesses and BAOAB MD unless noted, where "main" is the commit before #84 and #82:statusreports all 33.time_limitover 10 jobs.time_limit, with no images.ssneb, with MD frames unchanged. That event never reaches a band, so a stateless SSNEB band was checked in a harness: no evaluation escaped the clock, through the saddle check.state.jsonthat cannot be saved during a refinement: one failed save leaves the event's result unchanged. A lasting fault stops the job at the next save outside the check, as on main, and the resubmission gives main's barrier.Not run: a real MLIP on a GPU, or this on a cluster.
#84 and #82 are merged, and this merges with them with no conflicts. All three together: 356 passed, 25 skipped; ruff is clean.