[Montana]

Fixes #1261.

The defect, measured on amici_sens

The issue asked for amici_sens to be measured first. I replayed the nightly's 48 models × 2 methods from reference-data-v1 (replay mode, no AMICI needed). The outcomes match the committed baseline on all 96 cases. For the 82 compared cases I saved both tensors, then zeroed, negated, or multiplied by 1000 each bngsim parameter column, one at a time.

	columns
parameter columns, across 82 compared cases	4,272
under 1e-3 of their tensor's peak	3,146 (in 74 cases)
a zeroed, negated or ×1000 column still reads PASS today	2,702, in 56 of 82 cases

So the gap fd_sens had before #1167 is larger here: about 63% of amici_sens's columns could go wrong unseen.

The fix

_compare now judges each parameter column on its own, by the rule fd_sens adopted in #1167. The code moves into _amici_sens.py so both suites share it (issue step 2):

• column_verdicts: sens_verdict per column, each column scaled to its own peak, with atol scaled alike so the atol/|p| floor is unchanged.
• judged_columns: a column is judged where its peak, the larger of the two sides', clears its atol/|p| floor by JUDGE_FACTOR = 20 (issue step 4). Otherwise it is listed in columns_not_judged, as fd_sens does.
• A row with no column to judge is BAD_TEST, rather than amici_parity: a sensitivity tensor entirely below the resolvable floor is reported PASS #328's vacuous PASS. None of the 96 is one today.
• New report fields: n_columns_judged, columns_not_judged, columns_diff. The comment adds "k/n columns judged" and names the failing columns.
• fd_sens calls the shared functions with its own difference floor and forgive mask. Its behaviour is unchanged, and its 26 tests pass as before.

After, on the same replay

columns judged	3,448
a zeroed, negated or ×1000 judged column still reads PASS	0
columns not judged (under 20× their floor), named on the row	824
×1000 applied to a not-judged column	240 now judged and DIFF; 140 still under 20× their floor even at ×1000, so not judged by design
outcome changes against the committed baseline	none (82 PASS, 10 REFERENCE_FAILED, 2 BAD_TEST, 2 UNSUPPORTED)

On the issue's remaining steps:

• Step 3 (what the AMICI side forgives): no extra forgive mask was needed. Judged alone, no column of the 82 cases failed, so there was no constant-dominant-species false DIFF to forgive.
• Step 5 (re-baseline): no outcome moves, and the baseline records outcomes, not the max_rel value. So parity_checks/nightly/baselines/amici_sens.json stands as committed, with no new DIFF to triage. verdicts.py diff against it gives the same single alert as main's replay does on this machine (BIOMD0000000498 n_jac_evals 463 → 588). That alert is my Mac against the Linux baseline runner. A verdict change cannot alter solver work.
• Dispositions: the two :sens dispositions (MODEL0910846879, MODEL2105110001) are not among the nightly's 48 models, so the change in value's meaning (now the worst column's max_rel) reaches none of them.

Changes

diff --git a/parity_checks/amici_parity/_amici_sens.py b/parity_checks/amici_parity/_amici_sens.py
index 103495d8..1100b4e0 100644
--- a/parity_checks/amici_parity/_amici_sens.py
+++ b/parity_checks/amici_parity/_amici_sens.py
@@ -847,6 +847,81 @@ def sens_verdict(
     return v
 
 
+# How far a column's peak must clear its floor to be judged: 1 / differ's
+# HARD_REL_CEILING (5%). The verdict forgives every cell inside a floor, so a
+# column only just over one would have all of its disagreement forgiven, and a
+# PASS on it would say nothing; at 20x, what a floor forgives stays under what
+# the verdict calls real. fd_sens measured it (#1167): wofsy_goldstein's K11
+# (peak 2.2x its difference's floor) passed a 30% error, and nanomolar_dimer
+# (2.5x the atol/|p| floor) 40%.
+JUDGE_FACTOR = 20.0
+
+
+def column_peaks(sx: np.ndarray) -> np.ndarray:
+    """Per parameter, the peak ``|value|`` over (time, species); a column holding
+    a non-finite cell peaks at ``inf``."""
+    t = np.asarray(sx, dtype=float)
+    if t.size == 0:
+        return np.zeros(t.shape[-1] if t.ndim else 0)
+    return np.where(np.isfinite(t), np.abs(t), np.inf).max(axis=(0, 1))
+
+
+def judged_columns(bn_sx, ref_sx, param_values, atol: float, extra_floors=None) -> list[int]:
+    """The parameter columns a verdict can judge (issue #1261).
+
+    A column's peak, the larger of the two sides', has to clear its
+    ``atol/|p|`` floor (:func:`sens_resolution_floors`), and each of
+    ``extra_floors`` if given, by :data:`JUDGE_FACTOR`. Below that the verdict
+    would forgive its disagreement anyway, so a PASS on it would establish
+    nothing; it is left out and named on the row instead.
+
+    Either side's peak counts, so a column one side gets wrong is judged
+    whichever way it is wrong: a wrongly large one on its own peak, a silent
+    zero on the other side's. A column with a non-finite cell peaks at ``inf``
+    and is always judged.
+    """
+    floors = np.asarray(sens_resolution_floors(param_values, atol), dtype=float)
+    if extra_floors is not None:
+        floors = np.maximum(floors, np.asarray(extra_floors, dtype=float))
+    peaks = np.maximum(column_peaks(bn_sx), column_peaks(ref_sx))
+    return [j for j in range(len(floors)) if peaks[j] > JUDGE_FACTOR * floors[j]]
+
+
+def column_verdicts(
+    bn_sx, ref_sx, param_values, atol: float, judged, forgive_mask=None
+) -> dict[int, dict]:
+    """:func:`sens_verdict` for each judged parameter column on its own (#1167, #1261).
+
+    A parameter column's sensitivities are in that parameter's units, so one
+    tensor-wide verdict judges every column against the largest one: ``differ``'s
+    significance gate counts a (species, parameter) column only past 1e-3 of the
+    whole array's peak, and forgives every cell under an absolute 1e-12. On the
+    nightly's amici_sens cases, 2,702 of 4,272 parameter columns, in 56 of 82
+    compared cases, could be zeroed, negated or multiplied by 1000 and still read
+    PASS.
+
+    Each column is therefore judged alone, scaled to a peak of 1 (so ``differ``'s
+    absolute floor is relative too), with ``atol`` scaled alike so the
+    ``atol/|p|`` floor is unchanged. ``forgive_mask``, shaped like the tensors,
+    names cells the caller's reference cannot adjudicate (fd_sens: a finite
+    difference's own noise and step error); amici_sens passes none.
+    """
+    bn_sx = np.asarray(bn_sx, dtype=float)
+    ref_sx = np.asarray(ref_sx, dtype=float)
+    out = {}
+    for j in judged:
+        a, r = bn_sx[..., [j]], ref_sx[..., [j]]
+        mags = np.abs(np.concatenate([a.ravel(), r.ravel()]))
+        mags = mags[np.isfinite(mags)]
+        peak = float(mags.max()) if mags.size else 0.0
+        c = 1.0 / peak if peak > 0.0 else 1.0
+        mask = None if forgive_mask is None else np.asarray(forgive_mask, dtype=bool)[..., [j]]
+        out[j] = sens_verdict(
+            a * c, r * c, param_values=[param_values[j]], atol=atol * c, forgive_mask=mask
+        )
+    return out
+
+
 def measure_warmup(include_amici: bool = True) -> dict:
     """Per-process warmup for both engines — see :func:`_amici_common.measure_warmup`."""
     return ac.measure_warmup(include_amici)
diff --git a/parity_checks/amici_parity/amici_sens_run.py b/parity_checks/amici_parity/amici_sens_run.py
index 23a2b809..48459c45 100644
--- a/parity_checks/amici_parity/amici_sens_run.py
+++ b/parity_checks/amici_parity/amici_sens_run.py
@@ -120,7 +120,7 @@ DEFAULT_RTOL = asens.DEFAULT_RTOL
 DEFAULT_ATOL = asens.DEFAULT_ATOL
 
 
-def _compare(bn, am, param_values=None, atol=None) -> dict:
+def _compare(bn, am, param_values=None, atol=None, param_ids=None) -> dict:
     """Compare one (state, sensitivity) pair. Returns a result-fragment dict.
 
     ``bn``/``am`` are ``(t, x, sx, species_names)``. The parameter axis needs no
@@ -131,7 +131,9 @@ def _compare(bn, am, param_values=None, atol=None) -> dict:
     ``param_values`` / ``atol`` enable the solver-resolution noise floor (see
     ``_amici_sens.sensitivity_noise_mask``): a sensitivity that is identically
     zero makes the relative metric saturate at 1.0 against the other engine's
-    integration noise, which nothing scale-relative can forgive.
+    integration noise, which nothing scale-relative can forgive. With them,
+    each parameter column is judged on its own scale (issue #1261), and
+    ``param_ids`` names the columns the row reports as not judged or failing.
     """
     bn_t, bn_x, bn_sx, bn_names = bn
     am_t, am_x, am_sx, am_names = am
@@ -154,12 +156,41 @@ def _compare(bn, am, param_values=None, atol=None) -> dict:
     # State parity first — it qualifies the sensitivity verdict below.
     sv = differ.deterministic_verdict(bn_x[:, bn_idx], am_x[:, am_idx])
 
-    # Sensitivity parity: the headline.
-    v = asens.sens_verdict(
-        bn_sx[:, bn_idx, :], am_sx[:, am_idx, :], param_values=param_values, atol=atol
-    )
-
+    # Sensitivity parity: the headline, judged a parameter column at a time
+    # (issue #1261). On the tensor as a whole, differ counted a column only past
+    # 1e-3 of the largest one's peak: 2,702 of 4,272 columns here could be
+    # zeroed, negated or multiplied by 1000 and still read PASS.
     n_p = bn_sx.shape[2]
+    bn_used, am_used = bn_sx[:, bn_idx, :], am_sx[:, am_idx, :]
+    not_judged: list = []
+    columns_diff: list = []
+    if param_values is None or atol is None:
+        # No floor to judge a column against: the tensor-wide verdict, as
+        # sens_verdict itself does without them.
+        v = asens.sens_verdict(bn_used, am_used)
+        verdicts = {None: v}
+    else:
+        judged = asens.judged_columns(bn_used, am_used, param_values, atol)
+        verdicts = asens.column_verdicts(bn_used, am_used, param_values, atol, judged)
+        names = list(param_ids) if param_ids is not None else list(range(n_p))
+        not_judged = [names[j] for j in range(n_p) if j not in verdicts]
+        columns_diff = [names[j] for j, cv in verdicts.items() if not cv["passed"]]
+        v = {
+            "passed": bool(verdicts) and not columns_diff,
+            "max_rel": max((cv["max_rel"] for cv in verdicts.values()), default=0.0),
+            "noise_decisive": not columns_diff
+            and any(cv["noise_decisive"] for cv in verdicts.values()),
+        }
+        for k in (
+            "n_fail",
+            "n_cells",
+            "n_hard_fail",
+            "n_soft_fail",
+            "budget_forgiven",
+            "n_noise_rescued",
+            "n_below_noise_floor",
+        ):
+            v[k] = sum(int(cv[k]) for cv in verdicts.values())
 
     # Degeneracy witnesses (issue #328). A job can be reported PASS when the
     # WHOLE sensitivity tensor lies below the magnitude either solver can
@@ -194,10 +225,27 @@ def _compare(bn, am, param_values=None, atol=None) -> dict:
     )
     if n_resolvable == 0:
         comment += "; DEGENERATE: no parameter column resolvable"
+    if None not in verdicts:
+        comment += f"; {len(verdicts)}/{n_p} columns judged"
+        if columns_diff:
+            comment += f"; DIFF in {columns_diff[:6]}"
+    status = "pass" if v["passed"] else "diff"
+    if None not in verdicts and not verdicts:
+        status = "bad_test"
+        comment = (
+            "no parameter column carries a sensitivity "
+            f"{asens.JUDGE_FACTOR:g}x over its atol/|p| floor: the comparison would "
+            f"establish nothing; {comment}"
+        )
     return {
-        "status": "pass" if v["passed"] else "diff",
+        "status": status,
         "value": v["max_rel"],
         "comment": comment,
+        # issue #1261 — the columns the verdict could not judge, by name, and
+        # the ones it failed.
+        "n_columns_judged": None if None in verdicts else len(verdicts),
+        "columns_not_judged": not_judged,
+        "columns_diff": columns_diff,
         "state_passed": bool(sv["passed"]),
         "state_max_rel": float(sv["max_rel"]),
         "n_common_species": len(common),
@@ -706,7 +754,9 @@ def _worker(spec: dict, q) -> None:
         return
 
     try:
-        res.update(_compare(bn, am, param_values=param_values, atol=p["atol"]))
+        res.update(
+            _compare(bn, am, param_values=param_values, atol=p["atol"], param_ids=shared_ids)
+        )
     except Exception as exc:
         cap = capture_exception("compare", exc)
         res["status"] = "exception"
@@ -1033,6 +1083,11 @@ def main() -> int:
                         "max_abs_sx",
                         "n_resolvable_params",
                         "state_span",
+                        # issue #1261 — which columns the per-column verdict
+                        # judged, left out (too small to resolve), and failed.
+                        "n_columns_judged",
+                        "columns_not_judged",
+                        "columns_diff",
                     )
                     if k in r
                 },
diff --git a/parity_checks/fd_sens/fd_sens_run.py b/parity_checks/fd_sens/fd_sens_run.py
index d358cac6..3d74c0a9 100644
--- a/parity_checks/fd_sens/fd_sens_run.py
+++ b/parity_checks/fd_sens/fd_sens_run.py
@@ -249,13 +249,9 @@ FD_NOISE_FACTOR = 10.0
 # parameter) column's largest step gap |D(h) - D(h/2)| is inside the error that
 # gap estimates for the reference, and is not the analytic side's to fail.
 FD_UNCERTAINTY_FACTOR = 3.0
-# How far a column's peak must clear each floor to be judged: 1 / differ's
-# HARD_REL_CEILING (5%). The verdict forgives every cell inside a floor, so a
-# column only just over one would have all of its disagreement forgiven, and a
-# PASS on it would say nothing; at 20x, what a floor forgives stays under what
-# the verdict calls real. wofsy_goldstein's K11 (peak 2.2x its difference's
-# floor) passed a 30% error, and nanomolar_dimer (2.5x the atol/|p| floor) 40%.
-JUDGE_FACTOR = 20.0
+# How far a column's peak must clear each floor to be judged. amici_sens judges
+# its columns by the same rule (#1261), so the factor lives in _amici_sens.
+JUDGE_FACTOR = asens.JUDGE_FACTOR
 
 
 def fd_noise_floor(x_max: float, value: float) -> float:
@@ -266,15 +262,6 @@ def fd_noise_floor(x_max: float, value: float) -> float:
     return FD_NOISE_FACTOR * (FD_RTOL * x_max + FD_ATOL) / (fd_step(value) / 2.0)
 
 
-def _column_peaks(tensor: np.ndarray) -> np.ndarray:
-    """Per parameter, the peak ``|value|`` over (time, species); a column holding
-    a non-finite cell peaks at ``inf``."""
-    t = np.asarray(tensor, dtype=float)
-    if t.size == 0:
-        return np.zeros(t.shape[-1] if t.ndim else 0)
-    return np.where(np.isfinite(t), np.abs(t), np.inf).max(axis=(0, 1))
-
-
 def judgeable_columns(
     analytic: np.ndarray, reference: np.ndarray, x_max: float, values, atol: float
 ) -> list[int]:
@@ -301,14 +288,13 @@ def judgeable_columns(
     signal. A column with a non-finite cell on either side peaks at ``inf`` and
     is always judged: the verdict fails a non-finite analytic cell, and the gate
     refuses a non-finite reference.
+
+    The rule is :func:`_amici_sens.judged_columns`, shared with amici_sens; the
+    difference's floor is the one this suite adds to it.
     """
-    floors = asens.sens_resolution_floors(values, atol)
-    peaks = np.maximum(_column_peaks(analytic), _column_peaks(reference))
-    return [
-        j
-        for j, v in enumerate(values)
-        if peaks[j] > JUDGE_FACTOR * max(floors[j], fd_noise_floor(x_max, v))
-    ]
+    return asens.judged_columns(
+        analytic, reference, values, atol, extra_floors=[fd_noise_floor(x_max, v) for v in values]
+    )
 
 
 def reference_settled(analytic, reference, x_max: float, values) -> np.ndarray:
@@ -333,30 +319,18 @@ def column_verdicts(analytic, reference, gap, x_max: float, values, atol: float,
     significance gate counts a (species, parameter) column only past 1e-3 of the
     tensor's peak, and in 18 of 44 manifest models 49 of 159 judged
     parameters sat under that, where zeroing or negating an analytic column still
-    read PASS.
-    Each column is therefore judged alone, scaled to a peak of 1 (so ``differ``'s
-    absolute 1e-12 underflow floor is relative too: nanomolar_dimer's tensor
-    peaks at 6e-20), with ``atol`` scaled alike for the ``atol/|p|`` floor.
-
-    On top of that floor, the cells the difference cannot adjudicate are
-    forgiven: inside its noise (:func:`reference_settled`) or its step error
-    (:func:`reference_uncertain`). A DIFF therefore rests only on cells where the
-    reference is sure of itself.
+    read PASS. The per-column verdict is :func:`_amici_sens.column_verdicts`,
+    which amici_sens uses too (#1261).
+
+    On top of the ``atol/|p|`` floor, the cells the difference cannot
+    adjudicate are forgiven: inside its noise (:func:`reference_settled`) or its
+    step error (:func:`reference_uncertain`). A DIFF therefore rests only on
+    cells where the reference is sure of itself.
     """
     unsure = reference_settled(analytic, reference, x_max, values) | reference_uncertain(
         analytic, reference, gap
     )
-    out = {}
-    for j in judged:
-        a, r = analytic[..., [j]], reference[..., [j]]
-        mags = np.abs(np.concatenate([a.ravel(), r.ravel()]))
-        mags = mags[np.isfinite(mags)]
-        peak = float(mags.max()) if mags.size else 0.0
-        c = 1.0 / peak if peak > 0.0 else 1.0
-        out[j] = asens.sens_verdict(
-            a * c, r * c, param_values=[values[j]], atol=atol * c, forgive_mask=unsure[..., [j]]
-        )
-    return out
+    return asens.column_verdicts(analytic, reference, values, atol, judged, forgive_mask=unsure)
 
 
 def _judge(res: dict, analytic, reference, gap, species, values, spec: dict) -> None:

Plus parity_checks/tests/test_amici_sens_per_column.py.

Tests (test_amici_sens_per_column.py, 10)

test	main	this PR
a column at 1e-7 of the tensor's peak, zeroed / negated / ×1000, reads DIFF (precondition asserted: the tensor-wide verdict passes it)	fails: 'pass' == 'diff'	passes
a tensor entirely under differ's absolute 1e-12 (1e-17 scale, atol 1e-30), negated column, reads DIFF	fails	passes
the failing column is named (columns_diff == ["k2"])	fails	passes
agreeing columns pass, nothing listed	fails (no field)	passes
a column under 20× its floor is named in columns_not_judged and not judged	fails (no field)	passes
no column to judge → BAD_TEST	fails	passes
without parameter values the tensor is judged whole, as before	fails (no field)	passes
fd_sens's JUDGE_FACTOR and column_verdicts are the shared ones	fails	passes

The first two rows fail on main for the reason the issue names. The rest fail on main only because the fields and functions are new.

Mutations, each caught:

mutation	caught by
drop the per-column scaling	the 1e-17 test
judge every column (JUDGE_FACTOR = 0)	the not-judged and BAD_TEST tests
judge on bngsim's peak only	three fd_sens tests and two here

Suites: parity_checks/tests gives 916 passed. The merge of this branch with #1291, which also touches fd_sens_run.py, is clean, and 932 pass on the merged tree. The one failure there checks that the build's commit is in the repository, and the merged tree is an extracted copy, not a git checkout. ruff 0.14.0 (the pinned version) is clean.

No changelog fragment: this is a parity_checks/ change only, which ci/changelog.py does not ask for, as with #1167.

Likely objections

1. "824 columns are now not judged, where before they were in the tensor." Before, they were in the tensor but invisible: under 1e-3 of its peak, differ skipped them. Now they are listed by name. Those under 20× their own atol/|p| floor are columns whose disagreement the floor would forgive anyway.
2. "The row's value changes meaning." It is now the worst judged column's max_rel, each column relative to its own peak, as in fd_sens. No disposition in the nightly's set reads it.
3. "Why not re-record the baseline anyway?" The issue asks for a nightly re-baseline, expecting PASS→DIFF rows. There are none, and the baseline does not store the value. This PR's own nightly run on pull_request is the check: it should show the outcome table unchanged.