New qin lf

[cwjames1983]

This is an update featuring all goodies from Qin, but without bugs due to merging from main, and a couple of additions left over from pulling in repeaters

[profxj]

Review scope: the PR diff vs main (e85d656) — 22 files, +1900/−192. This is a big improvement over #87: the iteration.py re-merge from main is clean (I verified minimise_const_only2, the PATH machinery, the ptauw dkis2 coefficients, get_rates() masking, and the lEmax=41.84 default are all restored; Grid.update() stays deleted; grid_kwargs is now actually passed; the MeerTRAP Unicode-minus and the DSA_34 reference in zdm/scripts/slurm/ are fixed; and the new tests all pass). What remains are three small-but-fatal slips that crash exactly the configurations this PR ships, plus a few silent-behavior items and the #87 structural suggestions that are still open.

Blocking — small fixes, but they crash the shipped configurations

1. zdm/iteration.py:750-751 — half-applied rename: line 731 defines zt_tomult, but the two use sites now read ptaus *= tz_tomult / piws *= tz_tomult (main uses zt_tomult in all three). Any run with ptauw=True — the option this PR wires into MCMC_wrap — dies with NameError at the first evaluation.

2. zdm/iteration.py:1721-1743 — calc_likelihoods_2D's pwb block now references bweights/wweights, which are only ever defined in calc_likelihoods_1D (lines 544-545); this function defines zbweights/zwweights (1227-1248), which is what main used here. Any run with pwb=True on a survey with localized FRBs — exactly what both shipped slurm scripts do (--Pn --pwb) — raises NameError at line 1721.

3. zdm/MCMC.py:456 — if nthreads < 1: executes before the if nthreads is not None: resolution at 461, and the new default is nthreads=None (both in mcmc_runner and MCMC_wrap --nthreads), so every default invocation dies with TypeError: '<' not supported between 'NoneType' and 'int' before sampling. Move the guard after the None-resolution. Two adjacent issues while you're there: (a) the comment at 459-460 says numerical libraries are prevented from starting extra threads, but the old OMP_NUM_THREADS line was deleted rather than implemented, so nothing enforces it; (b) once fixed, the non-Slurm default silently becomes os.cpu_count() fork workers where the old default was 1 — with six-survey grids that's a large memory jump worth an explicit print.

4. zdm/energetics.py:321 (also ~453, ~611) — result = np.empty_like(Eth) is filled only through the four comparison masks. A NaN element of Eth fails every comparison, so its slot keeps uninitialized heap garbage. Grid thresholds can contain NaN (grid.py itself checks and warns for them), and with LF 0-3 a NaN propagates to NaN and is caught by the np.isnan(llsum) guard — with LF 4/5/6 you instead get nondeterministic, silently wrong rates. np.full_like(Eth, np.nan) restores the old failure mode.

5. papers/FitRepetition2025/slurm/run_mcmc.slurm:32,46 — this new file is a byte-for-byte copy of the pre-fix zdm/scripts/slurm/run_mcmc.slurm: it references survey DSA_34 and data/MCMC/params2.json, neither of which exists (the sibling copy was corrected to DSA in this same PR). As shipped, the paper-reproduction script fails at startup — and once fixed it hits crash #2 via --pwb.

Silent behavior changes — please confirm intent

6. zdm/MCMC.py:249-265 — when g0info is None, grid_kwargs stays {}, so calc_log_posterior's own nz/ndm/zmax/dmmax arguments (still in the signature, still forwarded by mcmc_runner) are silently ignored and the grid is built at the 500×1400 default. On main these were always passed. Either seed grid_kwargs from the function arguments in the else case or drop the dead parameters. (Also: the zmax = zvals[-1] + dz/2 reconstruction is exact only for linear grids — a zlog=True g0info would reconstruct the wrong zmax.)

7. zdm/scripts/run_slice.py:~90 — the hard-coded param_dict now sets 'lEmin': 30.0 with the previous fitted value 38.394... commented out on the same line, and the old vals2=[39.0] override loop is gone (moved to the paper script). Every slice now runs with an effective minimum energy nine decades lower — if intentional, worth a comment; if leftover experimentation, worth restoring.

8. zdm/energetics.py:58-60 — SplineMin −6→−9 and NSpline 1000→1500 move every knot of the igamma spline used by the default LF (2): existing fits re-evaluated on this branch will give slightly different likelihoods than main (no parameter changed), and each spline build gets ~70% more expensive (measured 0.14 s → 0.24 s of mpmath). Fine if deliberate — but see #9/#10, which multiply that cost.

9. zdm/energetics.py:541 (_broken_schechter_upper_gamma) — still no domain guard: the spline is tabulated on [1e-9, 1e6] and the shipped break_schechter.json priors (lEb ≥ 36, lEmax ≤ 45) sit exactly at the 1e-9 boundary; any wider prior or direct State call extrapolates the log-log cubic silently, and the np.clip(result, 0, 1) at 623 hides the garbage. Validate x against 10**SplineMin (the machinery already supports reinit=True re-tabulation), and demote the clip to a tolerance assertion.

10. zdm/MCMC.py:375 — energetics.reset() still runs unconditionally at the end of every posterior evaluation, so LF 2/6 runs rebuild the spline every step: 1500 mpmath calls ≈ 0.24 s per evaluation ≈ ~6.7 CPU-hours per 100k evaluations, even when the relevant gamma is not being sampled. Cache keyed on gamma (sparing unchanged keys in reset()), or build once in mcmc_runner.

Structural (carried over from #87, still open — fine as a follow-up PR)

• The LF integer codes are dispatched by magic number in four places that must stay in sync: grid.init_luminosity_functions (plus twelve near-identical wrapper methods; _broken_schechter_params is body-identical to _broken_power_law_params), ConvertToMeaningfulConstant (iteration.py:1933 — whose final else silently computes the gamma-function constant for any unknown code), valid_parameter_combination (MCMC.py:74), and the [1, 2, 6] Emax_boost list (grid.py:841). One registry in energetics — code → (functions, params-from-state, constraint, soft_cutoff) — collapses all four.
• The (ratio**gamma − 1)/gamma helper now exists in seven copies with inconsistent gamma≈0 tests (six use == 0, _broken_schechter_lower_integral uses np.isclose), so LFs disagree about a sampled gamma of 1e-9. One module-level helper.
• Constraint enforcement is still split: valid_parameter_combination returns −inf in MCMC, while the energetics functions raise ValueError — and the protective try/except in calc_log_posterior remains commented out (MCMC.py:230), so any non-MCMC caller (a slice scan crossing lEb, cube building) crashes mid-run instead of scoring −inf.
• The five zdm/data/MCMC/*.json configs are near-copies already drifting (Sbackproject: true in two of five, with no indication whether that's deliberate); a shared base + per-model overrides would prevent silent divergence.
• papers/FitRepetition2025/run_slice_vs_emin.py is a ~95% copy of the pre-cleanup run_slice.py, keeping the dead plot_grids/commasep, ~35 lines of commented-out param dicts, the plot_slice-inside-mkdir quirk, and a pkg_resources import that is deprecated on the very Python (≥3.12) this PR now requires. Importing init/calc_slice/plot_slice from the maintained script (or a shared module) keeps the paper reproducible without a fork. Also, passing g0info from init() into calc_log_posterior would avoid rebuilding the analytic z-DM grid for every slice point in both scripts.

Tests

test_energetics.py and test_mcmc.py are genuinely useful additions — but note that all of them pass on this branch despite crashes 1-3, because none exercise ptauw, pwb, or a default mcmc_runner call. A smoke test that runs calc_log_posterior once with Pn=True, pwb=True, ptauw=True on a tiny survey, and one mcmc_runner(..., nsteps=1) with default nthreads, would have caught all three and will protect the slurm workflows in future merges.

---

Review prepared with Claude Code: multi-agent scan of the PR diff cross-checked against the closed PR #87 review; every blocking item was verified by hand at the PR head (the undefined names, the None comparison, the mask-fill gap, and the missing data files).

[profxj]

One addendum to my review, from a late-arriving check: ConvertToMeaningfulConstant (iteration.py:1933-1967) — the new LF 4/5/6 branches pass np.array([Eref]) and so return a shape-(1,) ndarray where the LF 0 branch returns a float (callers like papers/Repeaters/Utilities/check_normalisation.py:86 get an array), and they return a normalized survival fraction P(E>Eref) while the LF 0 factor (Eref/Emin)**gamma - (Emax/Emin)**gamma is unnormalized — so 'meaningful constants' aren't on the same scale across LF codes. To be fair, both quirks are inherited from the pre-existing else branch (vector_cum_gamma), but since this PR adds three more branches on that side, it's a natural moment to unwrap the scalar (float(factor[0])) and either normalize the LF 0 branch or document the convention difference.