Stop giving wrong or incomplete results without saying so

README.md

@@ -66,7 +66,9 @@ wait in a queue until then ([when barriers are refined](docs/campaigns.md#when-b
 Three more commands work on finished output. `reactionflow scan` replays the saved frames through
 bond detection, with the run's settings or new ones, and tabulates every bond change and species
 ([offline scan](docs/offline-scan.md)). `reactionflow check campaign.json band.traj` tests whether
-the highest image of any band, from ReactionFlow or another code, is a saddle between its ends.
+the highest image of any band, from ReactionFlow or another code, is a saddle between its ends. It
+reports the band's residual forces and does not require them to be small, so a barrier from another
+code's band means what it says only for relaxed ends and a stationary highest image.
 `reactionflow export-electrace <pathway> --out <folder>` lays out a band as VASP frame folders and
 an [ElecTrace](https://github.com/Austin243/ElecTrace) configuration, to follow charges, critical
 points and bonds along it
@@ -75,7 +77,8 @@ points and bonds along it
 A number for `pressure_GPa` runs NPT, and the pathway is then refined with variable-cell SSNEB and
 gives an enthalpy barrier. `null` runs NVT with ordinary NEB. Each trajectory writes its frames,
 every detected occurrence, every refinement with its band and saddle check, and a
-`reactions.sqlite3` index under `output_root/<trajectory-id>/`. Nothing is overwritten, and a
+`reactions.sqlite3` index under `output_root/<trajectory-id>/`. Published events and refinements are
+never overwritten, though a rerun of `scan` or `check` replaces its own tables or verdict, and a
 resubmitted job continues each unfinished trajectory from its last checkpoint.
 
 Refining one event stops after `max_hours_per_barrier` hours, 4 by default, and records

docs/campaigns.md

@@ -236,10 +236,18 @@ marked failed refines its queue when resubmitted, then reports the failure again
 
 MD restarts exactly after a pause, and refinement reads only saved frames, so with a model that
 keeps no state between evaluations both orders give the same frames, events, and barriers. A model
-with internal state, such as ASE's EMT and its neighbor list, can differ at rounding level, as it
-does after any restart. The setting is not part of a trajectory's contract, so you can change it
+with internal state can differ, as it does after any restart: ASE's EMT and its neighbor list at
+rounding level, and tblite's xTB, which reuses its wavefunction, by a whole different path
+([exact restart](exact-restart.md)). The setting is not part of a trajectory's contract, so you can change it
 when you resubmit. A trajectory already paused for refinement finishes that refinement first.
 
+`validate`, and `run` when it starts a trajectory, warn about settings that quietly shape the result,
+on standard error for `run` and under `warnings` for `validate`: a structure with several frames, of
+which the run takes the last; momenta in the structure, which replace `temperature_K` for the
+starting velocities; constraints, which hold in MD and are removed in refinement; a cell at a pressure
+that refinement cannot use; and a run with too few observations to confirm any reaction. None of
+these stops the run.
+
 ## Campaign status
 
 `reactionflow status campaign.json` summarizes a campaign without changing it, so it is safe to
@@ -249,22 +257,30 @@ imports the MLIP.
 The first table has one row per trajectory: model profile, pressure, phase, step, detected events,
 converged pathways out of those refined, and the latest error. That error is the permanent
 failure for a `failed` trajectory, and otherwise the contents of `last-error.json`. Trajectories
-without output are listed as `not started`. A trajectory whose files cannot be read is reported
-with that error while the rest of the report still prints.
+without output are listed as `not started`. A file that cannot be read costs only what depends on it:
+the error column counts what was left out, `--json` lists it as `unreadable`, and a line under the
+table says the report is incomplete. A class whose candidate bundles cannot all be read, as in a copy
+of a run's results alone, keeps its counts and barriers in a row of its own, because classes merge
+only by their graphs. Saved frames that are gone appear in the error column and as `missing_frames`.
 Pathway counts include every refined occurrence. An event refined in steps counts each step as a
-refined pathway, and only its steps can converge.
+refined pathway, and only its steps can converge. Events still waiting to be refined are not among
+them: a line under the table counts them, and `--json` gives `queued`.
 
 The second table merges reaction classes across trajectories that share a model profile and
 pressure, because only those barriers are comparable. Classes merge by the local change: the
 changed bonds and the atoms within `--radius` bonds of them (default 2), including forward/reverse
 equivalence ([local classes](reaction-identity.md#local-classes)). Each row lists the bond changes
 with the formula of those atoms, the number of trajectories and events, converged bands out of
-those refined, validated saddles, and the minimum, median, and maximum barrier. Each step of an
-event refined in steps is an event of its own local class
+those refined, validated saddles, and the minimum, median, and maximum barrier. A barrier is measured
+from each event's own reactant, so a class that runs both ways has two barriers: the row's barrier
+is for the direction it names, and a further column, `reverse_barriers_eV` in `--json`, is for the
+other. Each step of an event refined in steps is an event of its own local class
 ([steps](pathway-refinement.md#reactant-frame-and-steps)). Barriers come only from validated
 saddles ([saddle check](pathway-refinement.md#saddle-check)). A line below the table
 totals the saddle check's verdicts across the campaign (validated, no saddle, rejected, unchecked)
 and counts separately the bands that converged before the check existed, which carry no verdict.
+Events that ended with no converged band are counted by status and message, up to five in the text
+and all of them as `no_band` in `--json`.
 NVT barriers are potential energies; NPT barriers are enthalpies. `--json` prints the same data,
 including every barrier value, the counts of each pathway, frequency and connectivity status,
 trajectory ID, and one example occurrence per class, for scripted analysis.

docs/candidate-tracking.md

@@ -31,8 +31,9 @@ Start the detector and tracker together so the tracker sees a baseline before an
 
 - The first stable topology establishes the accepted reactant basin.
 - Each changed region has its own stability window. A region is the changed atoms and the atoms
-  within `region_radius` bonds of them (2 by default), over the union of accepted, stable, and
-  provisional bonds. Regions that share an atom merge, so interacting changes stay together.
+  within `region_radius` bonds of them (2 by default, at least 1), over the union of accepted,
+  stable, and provisional bonds. Regions that share an atom merge, so interacting changes stay
+  together.
 - `region_radius=None` makes a region the whole bonded component instead. In a polymer, network,
   or slab that component holds every change in it, so unrelated changes merge and a crossing
   anywhere holds back a finished one. A campaign sets the radius as

docs/eon-search.md

@@ -222,7 +222,8 @@ exactly once, and an interrupted search is repeated with its saved seed. Keeping
 does not promise bitwise agreement across EON builds or numerical environments.
 
 The output directory holds the model/settings contract in `search-contract.json`, progress in
-`search-state.json`, and immutable checksummed records under `states/`, `processes/`,
+`search-state.json`, the error that ended the last run, if any, in `last-error.json`, and immutable
+checksummed records under `states/`, `processes/`,
 `attempts/`, and `steps/`. A process record keeps its saddle, product, barrier, prefactors, and
 bond changes; the reverse of each process is recorded for the product state. An attempt that EON
 rejected after its saddle converged keeps the saddle, both minima with their energies, and the
@@ -245,7 +246,13 @@ output directory.
   show up in the step list. `basins: "bonds"` merges states only when no bond changes between
   them; a low-barrier reaction is still a separate step.
 - A state is searched until it reaches the confidence target, with no cap on searches; `status`
-  shows the searches and confidence of the current state.
+  shows the searches and confidence of the current state, and counts the failed searches by reason
+  (`failure_reasons` in `--json`). `run` prints each new reason on standard error as it first
+  appears, up to ten. A confidence that stays at 0 while searches fail will not recover by itself.
+  The one automatic stop: if the model raises or returns a non-finite value during a search, it is
+  evaluated once on that search's starting structure. If that fails too, the run exits with status
+  1 without recording the search, and a rerun repeats it with its saved seed. If that succeeds, the
+  failure belonged to the geometry, so the search is recorded as failed and the run goes on.
 - Searches use a fixed, nonsingular cell, fully periodic or fully nonperiodic, with `FixAtoms` as
   the only constraint.
 - By default EON converges saddles and minima on the norm of the whole force vector

docs/exact-restart.md

@@ -40,13 +40,18 @@ Bitwise continuation requires the same ASE and NumPy versions, so the codec reco
 do not influence the propagated state.
 
 Calculator state is not guessed. An MLIP adapter must either serialize all mutable state or
-declare that deterministic reconstruction from pinned model/configuration data is exact. An
-adapter that cannot satisfy that contract cannot be used in strict exact-restart mode.
+declare that deterministic reconstruction from pinned model/configuration data is exact.
 
 The generic ASE calculator adapter is for deterministic, stateless inference. It binds the
 calculator factory and kwargs, model-file SHA-256 values, calculator source, and selected installed
 package versions to the checkpoint. Calculators with mutable inference state or their own RNG must
-use a custom adapter that snapshots that state explicitly.
+use a custom adapter that snapshots that state explicitly. Nothing tests this at run time: the
+adapter labels every calculator exact and gives no warning when one is not. tblite reuses its
+previous wavefunction, so each pause or kill of the xTB example changes its path: 6e-8 Å by step 100,
+growing to 6.3 Å within 20 ps, with 7 events uninterrupted, 8 with `refine_after_md`, and 10 after
+three kills, on one machine. Each is still a valid sample, but not the same trajectory. ASE's EMT and
+LJ differ at rounding level, at most 7e-15 Å over 3 ps. ANI-1xnr, AIMNet2-RXN, MACE and UMA on a CPU
+stayed bitwise exact in the cases tried; no GPU has been tested.
 
 The built-in Torch adapters, ANI-1xnr included, use deterministic PyTorch execution, disable TF32,
 and fix the cuBLAS workspace configuration. They record the model-file SHA-256 values, options,
@@ -87,7 +92,12 @@ run keeps a single runtime checkpoint.
 `state.json` records the phase, generation, global counters, pending pathway occurrence IDs,
 detector state, configuration, any durable-state failure, and the active checkpoint. Each
 generation writes one frame per observation to `segments/<generation>/trajectory.traj`; a frame is
-written before its checkpoint, so an interrupted attempt never leaves a gap. An MD state with a
+written before its checkpoint, so an interrupted attempt never leaves a gap. A segment lost later,
+to a node crash or a cut copy, cannot change the state in the checkpoint, so MD continues, and every
+`run`, `status` and `scan` then names the missing frames. `run` and `status` read each segment's
+ends, so damage inside a segment shows only in `scan`, which reads every frame. The one refusal is a
+current segment that cannot be appended to: restore it, or move it aside to continue in a new file.
+An MD state with a
 non-finite position, momentum, cell, or integrator array gets neither: the job stops with a non-zero
 exit, and the next job repeats from the last checkpoint, or starts over if there is none. A frame or
 checkpoint an earlier version wrote from such a state is not repaired.

docs/models.md

@@ -303,9 +303,10 @@ every pathway stage. For CPU runs, set `device: "cpu"` and campaign `require_gpu
 worker must see exactly one usable GPU.
 
 Each checkpoint records the model files, options, package versions, adapter/runtime source, and the
-Torch/platform/device environment. A changed model file or option refuses the resume. A changed
-version, source hash, or device continues with a warning and a line in
-`restart-environment.jsonl`, because the run is then no longer bitwise exact
+Torch/platform/device environment. A changed model file or option refuses the resume, and that
+includes the campaign's `device` option, since it is part of the trajectory's contract. A changed
+version or source hash, or other hardware behind the same option, continues with a warning and a line
+in `restart-environment.jsonl`, because the run is then no longer bitwise exact
 ([exact restart](exact-restart.md)). Keep the environment of an active campaign when you can.
 For Torch backends, deterministic algorithms are enabled and TF32 is disabled; unsupported nondeterministic
 operations fail. UMA uses fixed float32 inference without compilation or MoLE merging, so its

docs/offline-scan.md

@@ -13,16 +13,19 @@ reactionflow scan XDATCAR --pair C-O 1.6 2.0 --frame-ps 0.05
 ## Campaign runs
 
 A run saves one frame per observation in `segments/<generation>/trajectory.traj`. A new generation
-starts with the frame it resumed from, and the scan reads that frame once. It stops at the first
-frame it cannot read, such as the one a running job is writing, and it prints `"scan": null` for a
-trajectory with no saved frames yet.
+starts with the frame it resumed from, and the scan reads that frame once. It reads a segment up to
+the first frame it cannot read, such as the one a running job is writing, and passes over a segment
+that is missing, empty or cut. Frames the run has checkpointed that the scan could not read are
+listed as `missing_frames`, in the summary and on standard error, and the timing of a change across
+such a gap is not known. It prints `"scan": null` for a trajectory with no saved frames.
 
 With no options the scan uses the run's own settings from `state.json`, so it finds the candidates
 the run recorded. `candidates.csv` gives each one's occurrence ID, and in `summary.json`
 `matching_recorded` equals `recorded_by_run` for a finished run. Options change the replay and
 nothing else:
 
-- `--radius R` sets the region radius in bonds, and `--radius none` takes whole bonded components.
+- `--radius R` sets the region radius in bonds, from 1, and `--radius none` takes whole bonded
+  components.
 - `--persistence N` sets how many observations a crossing must last to become a bond change.
 - `--stability N` sets how many observations a region's new bonds must hold.
 - `--pair A-B FORM BREAK` sets the formation and break distances in Å for one element pair. It can
@@ -37,7 +40,10 @@ Any source that is not a `.json` campaign file is read as a trajectory file, in
 reads, for example an XDATCAR, an extxyz file, or a LAMMPS dump. Atoms keep the IDs the file
 carries, such as an `atom_id` column in extxyz, and otherwise get IDs 0 to N−1 in file order, so
 the file must keep its atom order. ASE sorts a LAMMPS dump by its `id` column, and it names the
-elements from an `element` or `mass` column, because types alone do not. The settings start from
+elements from an `element` or `mass` column, because types alone do not: a dump with only types is
+read as H, He, and so on, and the scan warns, on standard error and under `warnings` in the summary.
+It also warns on standard error about a file with no periodic cell, where a wrapped atom reads as a
+bond change. The settings start from
 the defaults of a campaign (thresholds from covalent radii, persistence 3, stability 3, radius 2)
 and take the same options. `--frame-ps` gives the time between frames for the time columns, which
 are empty without it. The tables go to `<file>-scan/` beside the file unless `--out` names another
@@ -51,8 +57,9 @@ directory.
   first frame.
 - `candidates.csv` has one row per candidate: the reactant, product and observed frames and the
   times of the first two, resolved or not, whether it was still changing at the end, its number of
-  atoms, its bond changes, its local class and label as `reactionflow status` names them (radius 2),
-  and the run's occurrence ID when the run recorded the same candidate.
+  atoms, its bond changes, its local class as `reactionflow status` merges them (radius 2), the
+  label of that class in this row's direction, and the run's occurrence ID when the run recorded the
+  same candidate.
 - `species.csv` counts each formula among the bonded components every `--every` frames, with the
   number of starting molecules in each component, which is the chain length in a polymer.
 - `summary.json` holds the settings and the counts, which are also printed:
@@ -62,7 +69,8 @@ directory.
     unresolved;
   - `net_changes_in_resolved` and `classes_in_resolved`, the bond changes and local classes in the
     resolved candidates;
-  - `recorded_by_run` and `matching_recorded` for a campaign run.
+  - `recorded_by_run` and `matching_recorded` for a campaign run;
+  - `missing_frames`, only when a campaign run's saved frames have gaps.
 
 These counts separate the stages between a threshold crossing and a candidate: what persistence
 filters, what is confirmed, how confirmed changes group into candidates, and how many never come

docs/pathway-refinement.md

@@ -144,7 +144,7 @@ The check ends a converged band as one of these:
 | `saddle_validated` | one mode, and the two sides relax to the band's two ends |
 | `no_saddle` | an endpoint is the highest point, or the saddle image has no mode |
 | `saddle_rejected` | more than one mode, or the sides do not reach both ends |
-| `saddle_unchecked` | the Hessian failed, or the connectivity check failed or was inconclusive |
+| `saddle_unchecked` | an image's energy was not finite, the Hessian failed, or the connectivity check failed or was inconclusive |
 
 The outcome's message says which case applies. A saddle lies above both endpoints, so when an
 endpoint is the highest point of the band, there is no saddle to test, and the outcome has no
@@ -200,7 +200,16 @@ pathway's `images.traj`, a band from another code, or an EON saddle written as t
 (minimum, saddle, minimum). The campaign supplies the model, the bond thresholds, and the pathway
 settings. `--index` picks the trajectory whose model and pressure are used; a campaign with several
 trajectories needs it, even inside a Slurm job. `--pressure` replaces that pressure. The verdict is written beside the band as `<band>.check.json`, in the form
-of a pathway's `result.json`, and its status, barrier, and message are printed.
+of a pathway's `result.json`, and its status, barrier, and message are printed. The images must hold
+the same atoms, elements, and periodicity, or the check refuses the band; images with atom IDs in
+another order are put in the first image's order, and a file without IDs is taken in its own order.
+
+The verdict rests on curvature and connectivity, and the barrier is the highest image minus the first
+image as given, so it means what it says only for relaxed ends and a stationary highest image, which
+a band from another code may not have. `largest_force_eV_A` in the verdict gives the largest atomic
+force on the first, highest, and last images, and the message of a validated band says when these
+pass `neb_fmax` or `relax_fmax`. A converged band of this package can pass `neb_fmax` slightly,
+because that tolerance applies to the band's projected forces. Neither changes the verdict.
 
 Campaign and state files written before the saddle check may name `active_radius` and
 `imaginary_frequency_cutoff_cm1`. They still open, and those two settings are ignored. A trajectory