Repository navigation
Validate the 330-tile v2 campaign (smk-p3a) and diff its shapes against v1.4.6 #405
Description
The first large ShapePipe v2 campaign, smk-p3a, finished on Nibi on 1 Oct. It covers 330 contiguous tiles (about 80 deg²) with 12.5M objects, 4.47M of them in the science selection, and passes ShapePipe's standing checks. sp_validation hasn't been run on it yet.
- Run cosmo_val on p3a through Read ShapePipe v1 and v2 products side by side #343's v2 reader. The products are under
/project/def-mjhudson/cdaley/sp-products/smk-p3aon Nibi (path inferred from the earlier campaigns; Cail can confirm). - Difference v1.4.6 and v2 shapes object by object over the same tiles, matching by sky position (
sp_catcompare.py --match sky), since v1 and v2 number objects differently.
p3a predates #922/#925 (defects, UberSeg) and #934 (PSF star box). Read the blend and PSF results with that in mind.
Claude Opus 5.5 on behalf of Cail. From the 3x2pt telecon, 6 Oct 2026.
Metadata
Metadata
Assignees
Labels
No labels
Activity
cailmdaley commentedon Oct 10, 2026
Current ShapePipe v2 products on Nibi now have one stable path:
/project/def-mjhudson/cdaley/sp-products/latest/(groupdef-mjhudson).These are the production batches from 10 Oct. They supersede smk-p3a, which ran on September code.
latest/final_cat_latest.hdf5: one structured dataset per tile underpatches/latest/<tile>, all rows, no cuts appliedfull_starcat_latest.hdf5: one dataset per exposurenexp_latest.hsp,nflagged_latest.hsp(HealSparse)psf/<exp>.tar)tiles/<shard>/<tile>/,exp/<shard>/<exp>/, as links into the source batchMANIFEST.md/MANIFEST.json: batches, code sha, counts, column conventionsCurrently: 125 tiles, 156 exposures, 5,015,937 objects (batches smk-b01–b05, code
255d4392).The path stays the same as new waves land. Each finished batch is added and the manifest updated; where a tile appears in more than one batch, the newer one is used.
N_EPOCHcounts only epochs with a validated PSF model (#952). The wave-1 files were migrated in place to that definition, and the manifest states the convention per tile.Two things persist beyond the catalogues: the PSFEx models and the PSF star catalogue. For anything else you'd want kept (SExtractor tile catalogues, star selections, …), say so here: what gets kept is a config choice, and we'd rather keep what people use.
Claude (Opus) on behalf of Cail.