You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Write the per-tile final_cat as HDF5, matching the campaign merge #941
The per-tile final_cat is FITS, while the campaign merge that concatenates the tiles writes HDF5 (hdf5_reconcile.py, lzf since #937). This issue proposes writing the per-tile file as HDF5 as well, with one dataset per column. The merge itself keeps one compound dataset per tile.
make_cat's column-adding becomes natural. In a per-column layout, adding a column writes only that column. FITS tables are row-major, so every added column rewrites the whole file. make_cat: write the final catalogue once per save stage, node-local #939 works around this by buffering and writing once.
Compression and partial reads come for free (lzf, per-column reads).
Not a speed project. The assessment behind #937–#940 found the pipeline's I/O cost came from access patterns, not FITS itself. After #939, a one-shot write of either format to node-local disk takes under a second. The gain here is consistency, plus a format that suits how make_cat builds the catalogue.
What it touches:
make_cat's output, plus merge_final_cat.py and hdf5_reconcile.py, which read per-tile files.
The completeness and params pins: the declared output changes, which makes this a campaign boundary.
Four or so test files.
Anyone who reads per-tile final_cat FITS directly.
Parquet is out of scope. If the analysis side moves to dataframe tooling, the merged HDF5 can be exported to Parquet then.
Activity