Write the per-tile final_cat as HDF5, one dataset per column

[cailmdaley]

Closes #941. Builds on #939 and #937 (both merged).

Change

• make_cat writes the per-tile catalogue as HDF5, in one pass. The runner collects every stage's columns (SExtractor, TILE_ID/TILE_UNIQUE_ID, ngmix, per-epoch PSF slots, MASK_*) in one ordered dict and writes final_cat<num>.hdf5. Each column is one lzf dataset; vector columns are 2-D. Columns keep stage order and dtype (byte order normalised to native). The file has no attributes, because the FITS file carried nothing beyond its column cards and EXTNAME.
• WORK_DIR is removed (runner and config_tile_Mc.ini). With a single write there is nothing to stage: a real 9,226 × 174 tile writes in 0.17 s to NFS on candide.
• Readers. create_final_cat.read_data reads per-column HDF5 into the structured array the merge already writes, and still reads FITS. find_final_cat prefers .hdf5 over .fits in both layouts. FITS support stays for runs already on disk and for sp_validation's image-sims im_merge rule. merge_final_cat reads final_cat-<ID>.hdf5 and drops its unused --hdu.
• Workflow. final_cat(), tile_make_cat and astra.yaml declare final_cat-<tile>.hdf5.

The merged final_cat_<run>.hdf5 layout does not change (one compound dataset per tile).

Campaign boundary

The declared per-tile output and the tile_make_cat params pin both change. A resumed campaign finds no final_cat-<tile>.hdf5 and re-declares every tile, so start a fresh root.

Schema note

The FITS writer promoted every float to float64. HDF5 keeps SExtractor's float32, so in SExtractor-mode (image-sims) campaigns 12 merged columns become float32 with identical values (MAG_*, FLUX_*, FLUXERR_*, MAGERR_*, FLUX_RADIUS, SNR_WIN, FWHM_*). A merged row shrinks from 530 to 482 bytes. Data campaigns keep their dtypes: DR6 detection columns are already f8/i4/i8 (checked on a DR6 sexcat).

Verification

• Real tile (image-sims 1z2z_grid_3, 273.282): make_cat as FITS (make_cat: write the final catalogue once per save stage, node-local #939) and as HDF5 (this branch) gives the same 174 columns in the same order with equal values (NaN-aware); only the float32 dtypes differ. Merging each through read_data + copy_data gives identical rows.
• Tests (dev SIF shapepipe_develop-dev-20260926, candide): pytest tests -m "not slow" gives 861 passed, 1 skipped, including new tests for the per-column lzf writer, merge invariants on make_cat's writer, and read_data HDF5/FITS parity.
• uvx astra-tools@0.2.17 validate passes.

Not changed

The Gen-2 helpers scripts/python/merge_final_cat.py, get_number_objects.py, combine_runs.bash, post_proc_sp.bash and example/unions_800/config_tile_match_ext_r.ini still expect FITS. sp_validation reads only the merged file and needs no change.

Claude Opus 5.5 on behalf of Cail

🤖 Generated with Claude Code

https://claude.ai/code/session_01JkM3cmXPuuSrNAf7vJde9e

[cailmdaley]

shouldn't we also update these helper scripts to accept other fits and hdf5?

[cailmdaley]

Yes. This is done in this PR: 1139435, 249b3c2 and 8dc0a76.

One reader for both formats. The new module is shapepipe.utilities.final_cat.read_final_cat.

• It decides HDF5 or FITS per file from the content (the HDF5 signature), not from the filename.
• Either format comes back as the same structured array in native byte order, with vector columns as sub-array fields.
• A missing requested column raises a KeyError that names the file and the column.

Helpers that now read through it.

• scripts/python/create_final_cat.py, and the workflow's merge_final_cat.py through it.
• The legacy scripts/python/merge_final_cat.py (used by post_proc_sp.bash). Before this change it only collected *fits, so on a directory of HDF5 tiles it found zero files and then failed with an IndexError. A directory that mixes FITS and HDF5 tiles now merges in one pass.

Helpers that needed no change.

• These never open a final catalogue: combine_runs.bash (it symlinks final_cat-*, so either extension works), clean_tile.py, run_report.py and hdf5_reconcile.py.
• These read sexcats rather than final catalogues: get_number_objects.py and ngmix_range.py.
• The workflow's merge still finds tiles only through final_cat-<tile>.hdf5, on purpose: that path is the Snakefile's tile-done marker.
• Out of scope: example/unions_800/config_tile_match_ext_r.ini still points match_external_runner at FITS final catalogues.

Tests. I ran these in the develop SIF on nibi.

• tests/unit/test_final_cat_reader.py is new. It writes one catalogue in both formats, with scalar, vector, int64 and bool columns, and checks four things: every column reads identically, a column subset comes back in the requested order, a file whose extension names the wrong format still reads, and a missing column raises for each format.
• That file plus test_final_cat_merge_invariants.py and test_decisions.py: 70 passed.
• The import tests: 113 passed.
• A hand run of the legacy merge on one HDF5 tile and one FITS tile gave 10 rows.

— Claude (Opus, nibi chair) on behalf of Cail