DunedinPACE end-to-end test fails on pandas 3.0 (stale golden file + stack() NaN change)

[IsraelAfangideh]

Summary

biolearn/test/test_model.py::test_dunedin_pace_normalization fails under pandas 3.0 for two independent reasons. Both are in the test/fixture layer — the normalized values themselves are correct (they match the golden data to ~5e-16). This is distinct from #195 (the read-only ValueError), which aborts the same test earlier on older code; once that is fixed, these two surface.

Environment

• Python 3.11
• pandas 3.0.3, numpy 2.4.6

Cause 1 — stale golden-file format

pace_normalized.pkl stores CpG ids in an ID_REF column with a default RangeIndex (shape 20000 x 11):

       ID_REF  GSM1009660  GSM1009661  ...
0  cg00000029    0.555300    0.536760  ...

But dunedin_pace_normalization returns CpG ids as the index (shape 20000 x 10). The test does np.abs(actual - expected), which misaligns on index/columns and ultimately raises:

KeyError: 'cg00000029'   (self = RangeIndex(start=0, stop=20000, step=1))

Fix: set ID_REF as the index and reindex expected to actual before comparing.

Cause 2 — DataFrame.stack() no longer drops NaN in pandas >= 3.0

The mismatch count uses:

mask = np.abs(actual - expected) > 0.000001
mismatches = actual[mask].stack()      # actual[mask] is NaN where mask is False
total_mismatches = mismatches.size

In pandas < 3.0, stack() dropped NaN, so only true mismatches remained. In pandas 3.0 the new stack() keeps NaN, so every cell is counted and the assertion fails even when all values are equal. Minimal repro:

import pandas as pd, numpy as np
df = pd.DataFrame(np.ones((3, 3)))
mask = pd.DataFrame(np.zeros((3, 3), dtype=bool))   # all False
df[mask].stack().size   # -> 9 on pandas 3.0, 0 on pandas 2.x

Fix: count from the boolean mask directly (e.g. stacked = mask.stack(); locations = stacked[stacked].index).

Verification

With both fixes, the test passes; aligned |actual - expected| max is ~5.5e-16 across all 20000x10 cells, so there is no numeric change to the model output.

I have a fix ready for both and can open a PR (it depends on #195 / #206 being present so the test reaches this point).

[marcbal77]

Fixed by #210, same root cause as #209 (fixture orientation plus the stack() NaN change). Both fields addressed there.