Tentative: PSFEx models ~0.4% larger than stars on P3 tiles (worth exploring) #956

Activity

1. cailmdaley commented on Oct 10, 2026

   

   cailmdaley

   on Oct 10, 2026

   Contributor

   More actions

   Thanks for this. It's a clean, independent measurement. We had found the same offset in an investigation in late September, and we think we know where most of it comes from. Here is a summary.

   What we found

   The size. Comparing held-out stars to the model across smk-p3a's 127 exposures, every exposure mean of δT/T (star − model) is negative, between −1.13% and −0.23%. A forward-render proxy over the campaign gives −0.585%, consistent with your −0.58% from full_starcat_smk-p3a.hdf5.

   The mechanism: PSFEx fits through Lanczos-3 resampling, and ShapePipe evaluates the model without it.

   • When PSFEx 3.21.1 fits (psf_refine), it resamples each basis vector to the star's sub-pixel position with a sum-normalised Lanczos-3 kernel. The star pixels themselves are not resampled.
   • Lanczos-3 shrinks a profile's HSM (adaptively weighted) size by about 0.65%, averaged over sub-pixel phase. PSFEx compensates by learning a native basis that is correspondingly broader.
   • ShapePipe's interpsfex sums the polynomial-weighted native PSF_MASK images directly, with no shift and no resampling (and it ignores PSF_SAMP). So it measures the broader native model.

   Evidence.

   • Refit. On exposure 2086324 we reconstructed all 38 usable model cubes bit for bit, then refit the same training stamps after Fourier-aligning them to integer centroids. That removes the fractional Lanczos step. The native model T shrank by 0.685 ± 0.019% at held-out positions, and the residual on the 1,423 accepted training stars went from −0.72 ± 0.04% to −0.03 ± 0.04%.
   • Reimplementation. A numpy + GalSim version that doesn't use PSFEx reproduces it: the phase-averaged HSM shrink of a Moffat is 0.645%, and a least-squares fit through the operator gives a 0.649% native excess.
   • Independent check. An independent rerun of the 47 identity checks confirmed the mechanism from both codebases.

   Caveats

   • It depends on the estimator. The Lanczos-3 kernel's unweighted second moment is essentially zero at every phase. The shrink comes from its positive fourth moment acting through adaptive weighting, so its size depends on how you measure size. That may be relevant to your moments-based shear-response check.
   • The evidence is narrow. The campaign-wide claim rests on one fully refitted exposure plus the forward-render proxy.
   • A flux-dependent residual remains. After the correction, bright held-out stars sit at about +0.26% (SNR > 200) to +0.41% (SNR > 400), and faint stars go negative. So the near-zero corrected mean is partly cancellation, and the error budget beyond the mechanism isn't closed. Your per-exposure spread (rms 0.33%) and the excess per-star rms point the same way: there is more here than one global offset.

   Not fixed yet

   • A validation-only Lanczos-3 shift removes most of the mean offset, but introduces about 0.27% cosine dependence on sub-pixel phase per axis.
   • Finer sampling helps in noise-free tests, but needs a renderer that honours PSF_SAMP.
   • Our plan is a known-input PSFEx-to-renderer recovery test that constrains size, ellipticity and phase dependence together, rather than a blanket rescaling of T.

   Related

   About 10.5% of PSFEx training stamps are centred on round(X_IMAGE) rather than round(XWIN_IMAGE), 1 px from the CENTER_KEYS position. PSFEx's recentring when it loads stamps hides this. We hadn't looked at the 1-based vs 0-based polynomial origin you raise, and it's worth a check alongside it.

   Your Gaia-selected sample is independent of the PSFEx star selection, which this needed. Repeating it over more area, with the PSFEx training stars held out, would show how much of the offset survives once the Lanczos effect is accounted for.

   Claude (Opus) on behalf of Cail. Summary of the nibi PSF-size investigation, 25 Sep.
2. cailmdaley commented on Oct 10, 2026

   

   cailmdaley

   on Oct 10, 2026

   Contributor

   More actions

   @calumhrmurray: the PSFEx models and star catalogue you used from smk-p3a now have a current-code counterpart at one stable path. Details below.

   Current ShapePipe v2 products on Nibi now have one stable path: /project/def-mjhudson/cdaley/sp-products/latest/ (group def-mjhudson).

   These are the production batches from 10 Oct. They supersede smk-p3a, which ran on September code.

   What	Path under latest/
   Galaxy catalogue, all tiles	final_cat_latest.hdf5: one structured dataset per tile under patches/latest/<tile>, all rows, no cuts applied
   PSF star catalogue (star vs PSFEx model, HSM)	full_starcat_latest.hdf5: one dataset per exposure
   Exposure-count and flagged-pixel maps	nexp_latest.hsp, nflagged_latest.hsp (HealSparse)
   Per-tile catalogues; per-exposure PSFEx models (psf/<exp>.tar)	tiles/<shard>/<tile>/, exp/<shard>/<exp>/, as links into the source batch
   What's in it	MANIFEST.md / MANIFEST.json: batches, code sha, counts, column conventions

   Currently: 125 tiles, 156 exposures, 5,015,937 objects (batches smk-b01–b05, code 255d4392).

   The path stays the same as new waves land. Each finished batch is added and the manifest updated; where a tile appears in more than one batch, the newer one is used. N_EPOCH counts only epochs with a validated PSF model (#952). The wave-1 files were migrated in place to that definition, and the manifest states the convention per tile.

   Two things persist beyond the catalogues: the PSFEx models and the PSF star catalogue. For anything else you'd want kept (SExtractor tile catalogues, star selections, …), say so here: what gets kept is a config choice, and we'd rather keep what people use.

   Claude (Opus) on behalf of Cail.