Repository navigation
Retire patch logic — ShapePipe v2 has no patches #340
Description
ShapePipe's Snakemake workflow (CosmoStat/shapepipe#852) processes a campaign — a tile list — with no intermediate "patch" (P1…P7) unit. Its merged products (CosmoStat/shapepipe#879) are one final_cat_<campaign>.hdf5 and one full_starcat-0000000.fits per campaign. sp_validation still carries the patch layer throughout, and the workflow currently has to write a legacy patches/<campaign>/<tile-ID> group into the hdf5 purely so catalog.py's reader opens it.
Where patch assumptions live today (rough grep, jackknife npatch excluded):
src/sp_validation/catalog.py— hdf5 reader hardcodespatches/{name}/<ID>src/sp_validation/catalog_builders.py—get_patches()(v1/v1.5→P1..P7/8), patch→surveymerge_cataloguesover<patch>/sp_output/shape_catalog_comprehensive_*.fitssrc/sp_validation/survey.py— patch boundary coordinatessrc/sp_validation/rho_tau.py,cosmo_val/real_space.py— patch-keyed paths/labels- scripts:
combine_results.py,merge_psf_cat.py,prepare_patch_for_spval.sh,plot_rho_stats_patches.py,survey_stats_all.sh,stats_tile_id_gal_counts.py,star_match_stats.py,check_tile_IDs_SP_LF.py,compute_area.py
Proposed end state: sp_validation takes a campaign's two merged files directly (hdf5 with a flat per-tile layout, or a single top-level group; star cat as-is), and any cross-campaign combination is expressed as "list of campaigns", not "list of patches". Scripts that only ever made sense for the P1–P7 era get deleted rather than ported. Once the reader is flat, the patches/ key in ShapePipe's final_cat_merge goes too.
— Claude (Fable) on behalf of Cail
Activity
cailmdaley commentedon Sep 26, 2026
@martinkilbinger on second thought, do we need to keep the patch logic so that older catalogs can still be read/analyzed?
cailmdaley commentedon Sep 26, 2026
and if we need to keep some backwards compatibility, how much? #343 purports to still be able to read the old layout, but also deletes some patch-based scripts...
retire patch logic (#340)
scripts: cheaper star mask cut, campaign-derived leakage labels
cailmdaley commentedon Oct 1, 2026
Answer as implemented in #343: reading old catalogues needs no patch logic. The released v1 merged catalogues are flat files (
data/data_ext, with an int8patchcolumn that just passes through), and the reader plusgrammar.adaptopen them directly. So #343 keeps only the reader and adapter for v1, and everything else speaks campaigns:JointCatmerges campaign files, and the patch-only scripts are gone (combine_results.py,compute_area.pyand the five listed above). The one thing lost is re-merging v1 from per-patch FITS intermediates, which nothing needs while the merged releases exist.One follow-up stays ShapePipe-side:
create_final_cat.pystill wraps tiles inpatches/<campaign>/. The reader walks through it, so nothing here depends on it; dropping it wants its own small ShapePipe PR at a campaign boundary.— Claude (Opus) on behalf of Cail
cailmdaley commentedon Oct 1, 2026
Reading and analysing older catalogues doesn't need the patch logic. Only rebuilding them from patch outputs does.
cut_strucFITS, and the PSF/star FITS. Patches appear only in the inputs that build them, such asP*/sp_output/shape_catalog_comprehensive_ngmix.fitsandP*/sp_output/psf_catalog_ngmix.fits.extract_infoon per-patchfinal_cat, inJointCat.get_patches/merge_catalogues(behindcreate_joint_comprehensive_cat.py), in thev1/v1.5/v1.6shortcuts ofmerge_psf_cat.py, and in the per-patch diagnostic scripts.extract_infono longer reads per-patchfinal_cat,create_joint_comprehensive_cat.pymerges campaign files, and the patch-only scripts are deleted.merge_psf_cat.pystill accepts an explicit-p P1+…+P9, but writes acampaigncolumn instead ofpatch.So #343 can go in as it is. Anyone who needs to rebuild a v1.x catalogue from patch outputs can use a checkout from before #343.
— Claude Opus 5.5 on behalf of Cail
Read ShapePipe v1 and v2 products side by side (#343)