[Montana]

Fixes #1307.

Stacked on #1321. This PR's first nightly failed Path-agreement on TIMEOUTs only: models at the edge of the 900 s cap (MODEL1006230049, 0053, 0077) lost every row once a check was added to every model. #1321 bounds each check by the model's cap and keeps a killed worker's finished rows. It is merged in here, and this PR's new checks run after every check main runs. Until #1321 merges, this PR also shows its commits.

Stacked on #1291. #1307 is blocked by #1291, and this branch is built on it. Until #1291 merges, this PR also shows #1291's commits. Only the last commit, 3c72486, is this PR's.

It touches lines its siblings also touch:

• CHECKS, also edited by differential: the analytic Jacobian's values against differences of the RHS (#1308) #1315 (differential: the analytic Jacobian's values against differences of the right-hand side, not only its pattern #1308) and differential: metamorphic transforms of the model file (#1305) #1316 (differential: metamorphic transforms of the model, the check a shared loader cannot fail together #1305);
• the runner's docstring;
• the end of test_differential_shapes.py, which differential: generated models, and each finding shrunk to a reproducer (#1306) #1317 (differential: generate models (random networks and rate laws) and shrink each finding to a minimal reproducer #1306) also appends to;
• Base.species_names, which differential: metamorphic transforms of the model file (#1305) #1316 adds with the identical lines.

Each conflict resolves by keeping both sides. Whichever of these lands after another, I'll resolve it.

What it adds

Three checks. Every other differential check runs a fresh model, by design, so none exercises reuse. Callers reuse models, though, and "the solver leaves scratch state in the user's Model, or drops state it should carry" is a family of its own (#786; #748, #705, #693). Each check is judged against #1291's base run with Base.compare.

check	what it runs	expected
restart	run_until the middle sample, then run_until the end, on one Simulator (which carries the events' trigger state and pending delays across the stop, #693). For a model without events, also a fresh Simulator given the middle's saved state	the joined trajectory is the base
reuse	a second run() on one Simulator, and a first run on a second Simulator over one Model, each after the first run is undone with snapshot()/restore()	each is the base
batch	run_batch on 4 threads against the same four points one at a time (the first sensitivity parameter ×1, 1.1, 0.9, 1.05), and the ×1 point against the base	equal

Three design decisions, each to avoid a false finding:

1. restore(), not Model.reset(), undoes the first run. reset() restores species only, by its own documentation, so a model whose event writes a parameter would read as a leak.
2. The fresh-model restart is for models without events only. A fresh Simulator can't know which triggers were already true at the stop, so it isn't asked to.
3. A batch point that doesn't integrate one at a time is SKIP, named. It's the model at the edge of its domain, 10% off its own parameter, not the batch. A point that fails only on threads is still a finding.

Base keeps its run's species names, so columns are compared by name. It's the same addition #1316 makes, with the same lines.

Over the corpus

1,353 models each: the 53 .net jobs and the BioModels, run locally.

check	PASS	SKIP	DIFF	EXCEPTION
restart	1,260	92	0	1
reuse	1,260	93	0	0
batch	1,259	94	0	0

The SKIPs are #1291's own: models whose base does not load or run, or has no reference. The exception is batch's, for four models (BIOMD0000000763, BIOMD0000001043, MODEL1112110002, MODEL2009230001), whose perturbed point fails one at a time as well as on threads.

The exception, triaged: BIOMD0000000827 under restart. It is no leak:

• Stopped at t = 50, 100 or 150 and continued, it fails soon after (t = 80, 106, 162) at a NaN in a rate law (EGFR_EGFR_formation).
• It fails the same way whether continued by run_until or by a fresh Simulator from the saved state.
• run_until's state at the stop equals run()'s exactly.
• The straight run is robust at 0.5× to 2× its tolerance, and the model has no events.

So the state is carried correctly. Restarting the integrator from it takes the run into a region the continuous run steps around. That is a model at the edge of its domain, #1297's and #1298's class.

Tests (python/tests/test_differential_shapes.py, appended)

• The shape tests pick up the three checks automatically: all 15 shapes pass each.
• Negative controls: a model mutated between runs reads DIFF, as the issue asks. Each runs on a .net (equality-dwell) and on an SBML model with an event (event-at-a-sample):
  • a parameter doubled between the two run_until legs;
  • … after set_state in the fresh-model restart (on the event-free .net);
  • … inside restore(), which reuse rests on;
  • … for one threaded batch point.
• The batch's SKIP, both ways:
  • a point that fails one at a time and on threads is SKIP;
  • one that fails only on threads is still an EXCEPTION.

Mutations, each caught:

• restart without its run_until comparison;
• reuse without the restore between runs (the shapes' own reuse rows then read DIFF);
• batch compared against itself rather than one at a time;
• a combiner that passes every part;
• a serial failure not caught.

Suites: test_differential_shapes.py and parity_checks/tests/test_differential_run.py give 222 passed, 1 xfailed (#1291's #1254 xfail). ruff 0.14.0 is clean. The branch merges cleanly with the current lanl main.

Cost

Per model:

• restart: 2–3 runs;
• reuse: 4;
• batch: 8.

Locally, the batch check alone took 240 s over the corpus on 8 workers.

Baseline

parity_checks/nightly/baselines/differential.json will be re-recorded from this PR's own nightly run, as #1307 asks, in a follow-up commit here (as for #1315).

Changes (this PR's commit, less the tests)

diff --git a/parity_checks/differential/_differential.py b/parity_checks/differential/_differential.py
index 5bf10534..5a55147b 100644
--- a/parity_checks/differential/_differential.py
+++ b/parity_checks/differential/_differential.py
@@ -101,6 +101,10 @@ CHECKS = (
     "grid",
     "sparsity_covers_jacobian",
     "backend_as_asked",
+    # State leaks (#1307): a restarted run, a reused Model or Simulator, a batch.
+    "restart",
+    "reuse",
+    "batch",
 )
 
 
@@ -209,6 +213,9 @@ class Base:
         result, sim = run(job)
         self.job = job
         self.x = np.asarray(result.species, dtype=float)
+        # The columns of x, by name: a Result's species, which leave out a
+        # parameter an event promotes into the state (Model.species_names has it).
+        self.species_names = list(result.species_names)
         self.backend = sim.codegen_backend
         self.drift = own_drift(job, self.x)
         self.limit = max(PATH_RTOL_FACTOR * job.rtol, DRIFT_FACTOR * self.drift)
@@ -521,6 +528,158 @@ def check_sparsity(base: Base) -> Verdict:
     return _guard(go)
 
 
+# State-leak checks (issue #1307). Every other check runs a fresh model, by
+# design; callers reuse one, and "the solver leaves scratch state in the
+# user's Model, or drops state it should carry" is a family of its own (#786;
+# #748, #705, #693). Each run here is compared with the base as every path is.
+
+BATCH_FACTORS = (1.0, 1.1, 0.9, 1.05)
+BATCH_THREADS = 4
+
+
+def _species(result, base: Base) -> np.ndarray:
+    """A Result's species in the base's column order."""
+    names = list(result.species_names)
+    x = np.asarray(result.species, dtype=float)
+    return x[:, [names.index(n) for n in base.species_names]]
+
+
+def _worst(verdicts: list[tuple[str, Verdict]]) -> Verdict:
+    """The verdict of several comparisons: a DIFF if any is, else the first."""
+    for _label, v in verdicts:
+        if v.status != "pass":
+            return v
+    return Verdict(
+        "pass", max(v.value or 0.0 for _, v in verdicts), "; ".join(v.note for _, v in verdicts)
+    )
+
+
+def _solver(job: Job) -> dict:
+    return {"rtol": job.rtol, "atol": job.atol, "max_steps": job.max_steps}
+
+
+def check_restart(base: Base) -> Verdict:
+    """A run to the middle sample, continued to the end, is the base. Leg by
+    leg with ``run_until`` on one Simulator, which carries the events' trigger
+    state and pending delays across the stop (#693); and, for a model without
+    events, a fresh Model given the state saved at the middle, which cannot
+    know which triggers were already true and so is not asked to."""
+    import bngsim
+
+    def go():
+        job = base.job
+        n = job.n_points
+        m = (n - 1) // 2
+        t_mid = job.t_start + (job.t_end - job.t_start) * m / (n - 1)
+        out = []
+
+        sim = bngsim.Simulator(load(job.path), method="ode", **job.sim)
+        if job.t_start != 0.0:
+            sim.set_time(job.t_start)
+        first = sim.run_until(t_mid, n_points=m + 1, **_solver(job))
+        second = sim.run_until(job.t_end, n_points=n - m, **_solver(job))
+        legs = np.vstack([_species(first, base), _species(second, base)[1:]])
+        out.append(("run_until", base.compare(legs, "run_until to the middle and on: ")))
+
+        model = load(job.path)
+        if model._core.n_events == 0:
+            sim = bngsim.Simulator(model, method="ode", **job.sim)
+            head = sim.run(t_span=(job.t_start, t_mid), n_points=m + 1, **_solver(job))
+            state = np.asarray(sim.get_state(), dtype=float)
+            fresh = bngsim.Simulator(load(job.path), method="ode", **job.sim)
+            fresh.set_state(state, time=t_mid)
+            tail = fresh.run(t_span=(t_mid, job.t_end), n_points=n - m, **_solver(job))
+            joined = np.vstack([_species(head, base), _species(tail, base)[1:]])
+            out.append(("fresh", base.compare(joined, "a fresh model from the middle's state: ")))
+        return _worst(out)
+
+    return _guard(go)
+
+
+def check_reuse(base: Base) -> Verdict:
+    """A second run is the base: a second ``run()`` on the same Simulator, and
+    a first one on a second Simulator over the same Model, each after the
+    first run is undone with ``snapshot()``/``restore()`` (species, parameters,
+    and the events' trigger state, #693). ``Model.reset()`` restores species
+    only, so an event that writes a parameter would leave it written; restore
+    is the documented way back."""
+    import bngsim
+
+    def go():
+        job = base.job
+        span = {"t_span": (job.t_start, job.t_end), "n_points": job.n_points, **_solver(job)}
+        out = []
+
+        sim = bngsim.Simulator(load(job.path), method="ode", **job.sim)
+        snap = sim.snapshot()
+        sim.run(**span)
+        sim.restore(snap)
+        again = sim.run(**span)
+        out.append(
+            ("run", base.compare(_species(again, base), "a second run() on one Simulator: "))
+        )
+
+        model = load(job.path)
+        first = bngsim.Simulator(model, method="ode", **job.sim)
+        snap = first.snapshot()
+        first.run(**span)
+        first.restore(snap)
+        second = bngsim.Simulator(model, method="ode", **job.sim).run(**span)
+        out.append(
+            ("model", base.compare(_species(second, base), "a second Simulator on one Model: "))
+        )
+        return _worst(out)
+
+    return _guard(go)
+
+
+def check_batch(base: Base) -> Verdict:
+    """``run_batch`` on ``BATCH_THREADS`` threads is the same points run one
+    at a time, and its unchanged point is the base. The points scale the
+    first parameter a sensitivity run would take (``sens_params``); a model
+    with none runs the same point four times, which still asks whether the
+    threads share anything."""
+    import bngsim
+
+    def go():
+        job = base.job
+        model = load(job.path)
+        knobs = sens_params(model)
+        if knobs:
+            p, v = knobs[0], float(model.get_param(knobs[0]))
+            points = [{p: v * f} for f in BATCH_FACTORS]
+        else:
+            points = [{} for _ in BATCH_FACTORS]
+        kw = {
+            "t_span": (job.t_start, job.t_end),
+            "n_points": job.n_points,
+            "params": points,
+            **_solver(job),
+        }
+        # One at a time first: a point that does not integrate alone (a model
+        # at the edge of its domain, moved 10% off its own parameter) is not
+        # the batch's to judge. Threads failing where it runs is.
+        try:
+            serial = bngsim.Simulator(load(job.path), method="ode", **job.sim).run_batch(**kw)
+        except bngsim.SimulationError as e:
+            first = str(e).splitlines()[0][:200]
+            return Verdict(
+                "skip", note=f"a point of the sweep does not run one at a time: {first}"
+            )
+        sim = bngsim.Simulator(model, method="ode", **job.sim)
+        threaded = sim.run_batch(num_processors=BATCH_THREADS, **kw)
+        out = [("point 0", base.compare(_species(threaded[0], base), "the batch's own point: "))]
+        for i, (a, b) in enumerate(zip(threaded, serial, strict=True)):
+            g = gap(_species(a, base), _species(b, base), PATH_RTOL_FACTOR * job.atol)
+            note = (
+                f"point {i} on {BATCH_THREADS} threads against one at a time: {g:.2e} of its peak"
+            )
+            out.append((f"point {i}", Verdict("pass" if g <= base.limit else "diff", g, note)))
+        return _worst(out)
+
+    return _guard(go)
+
+
 def comma_locale() -> str | None:
     """A comma-decimal LC_NUMERIC this machine has, or None. Asked of a child
     process, so this one's locale is never set."""
@@ -594,6 +753,9 @@ def check_all(job: Job, locale_name: str | None, checks=CHECKS) -> dict[str, Ver
         "grid": check_grid,
         "sparsity_covers_jacobian": check_sparsity,
         "backend_as_asked": check_backend,
+        "restart": check_restart,
+        "reuse": check_reuse,
+        "batch": check_batch,
     }
     for name, fn in one.items():
         if name in checks:
diff --git a/parity_checks/differential/differential_run.py b/parity_checks/differential/differential_run.py
index 02cc1645..71e82b70 100644
--- a/parity_checks/differential/differential_run.py
+++ b/parity_checks/differential/differential_run.py
@@ -7,6 +7,9 @@ run's, the FD Jacobian against the analytic one, KLU against the dense solver,
 compiled against interpreted, cc against MIR, a comma-decimal locale against C,
 and three rules for bugs whose answers are right (a sensitivity run refused, a
 sparsity pattern that misses entries, a backend other than the one asked for).
+Three more reuse what every other check makes fresh (#1307): a run stopped and
+continued, a second run on a Simulator or a Model a first one ran, and a batch
+on threads against the same points one at a time.
 One row per (model, check), its ``method`` the check:
 
     the path agrees with its reference .............. PASS

No changelog fragment: this changes parity_checks/ and tests only.