test: cover NaN lane-step counter accumulation across a MultiStart resume

[Jammy2211]

End-to-end cover for the value-NaN / gradient-NaN step counters added in PyAutoFit#1472 (n_value_nan_lane_steps / n_grad_nan_lane_steps).

Those counters restore via search_internal.get(..., 0) and are meant to be lifetime totals across a resume. That could not be demonstrated end-to-end until PyAutoFit#1474: Fitness.check_log_likelihood compared a stored log likelihood against the search's own figure of merit, so every multi-start resume raised a SearchException before reaching the fit loop. #1472 therefore shipped with the resume accumulation covered only by unit tests over hand-built search_internal dicts.

Result: no bug — the counters accumulate correctly. This locks that in.

What the script does

Kills a MultiStartAdam mid-run as a subprocess (iterations_per_full_update=2 so checkpoints are written during the run) and resumes it by re-invoking the identical search, then compares against an uninterrupted reference run.

uninterrupted : total_steps=142, value_nan=403, grad_nan=9
killed at     : total_steps=16,  value_nan=1,   grad_nan=0
killed+resumed: total_steps=142, value_nan=403, grad_nan=9

Why equality is the assertion

The load-bearing invariant is equality with the uninterrupted reference, not "the counters went up". The search is deterministic (broad starts seeded 0), so the reference is exact.

This is not a stylistic choice. Simulating the regression — resetting both counters on resume — loses the lane-step recorded before the kill and yields 402 against the reference's 403. A >= assertion accepts that happily; equality catches it. The weaker >= checks are kept alongside only so a failure reports which way it broke.

Verified by injecting that regression into PyAutoFit locally, confirming this script fails, then restoring.

Why both traps sit on the descent path

_broad_starts rejects any draw whose objective or gradient is non-finite, so every lane begins healthy by construction and a NaN trap placed at the edges of the prior is never reached — early attempts reported counters of exactly 0 for this reason. Both traps are therefore positioned where descent actually runs into them, which is also the realistic case (a pixelized likelihood going degenerate near its solution):

• gradient-NaN — |centre - 50| < 2, the truth basin. Uses the where/sqrt pattern documented in Fitness.call: inside the band the selected branch is a finite 0.0 while the unselected sqrt of a negative is NaN, and reverse-mode gives 0 * NaN = NaN. Value stays finite, only the gradient dies.
• value-NaN — sigma < 10.5. Truth sigma is 10.0, so surviving starts descend across the threshold and die there.

resurrect=False so dead lanes stay dead and keep counting every step, making the totals grow monotonically.

Guards against a vacuous pass

• the reference run must actually produce NaN lane-steps in both counters
• a mid-run checkpoint carrying a non-zero counter must be observed before the kill, or there is nothing for the resume to accumulate onto
• the kill must land before the search finishes (at_kill["total_steps"] < reference["total_steps"])

Notes

• Not added to smoke_tests.txt — it is real_search jax like the sibling MultiStartResurrect.py, too slow for the smoke gate.
• Also confirmed while writing this: a resumed run's final counters are independent of where the kill lands (steps 72 and 74 both finished at the reference totals), so the checkpoint captures the full accumulator state.

---

Generated by Claude Code