Repository navigation
Migrate cosmo_val + Snakemake to the SACC writers #247
Description
Make SACC the sole native output format of the cosmology-validation pipeline. Every two-point product (ξ± coarse and fine, pseudo-Cℓ, COSEBIs, pure-E/B, ρ/τ, n(z)) is written through the sacc_io builders into single-statistic SACC part files, and the Snakemake workflow assembles those parts into one terminal SACC file per catalogue version, {version}.sacc. No bespoke writer remains on the data-vector path.
Desired end state
- Each
cosmo_valstatistic's write step is replaced by a call to the matchingsacc_iobuilder, producing a part file that holds exactly one statistic plus the self-describing n(z) tracers it references. A producer writes a part with a singlesave(s, path)call; it never assembles a terminal file and never emits a bespoke format alongside. - A dedicated Snakemake assembly rule reads the parts, concatenates them in canonical order, attaches the
BlockDiagonalcovariance (analysis blocks plus the dense per-pair fine block), and writes the single terminal{version}.sacc. The terminal file is a pure gather; anything that wants only one statistic targets the corresponding part rule, not a second terminal file. - Part filenames are DAG internals: consumers bind parts through the workflow's path helpers, never by hardcoding names.
{version}.saccis the only stable name. - Blinding rides the DAG here. Blinding is per part, at birth (PRD: SACC data format + Smokescreen blinding #241 §4; CLIs and the assembly-time assertion are delivered by Smokescreen blinding wiring (fork protocol, three theory backends, custody) #252): a
blind-initrule fires once per catalogue version, each blindable part rule (coarse ξ±, fine ξ±, pseudo-Cℓ) is followed byblind-partso only blinded parts persist on disk, COSEBIs/pure-E/B are derived from the blinded fine ξ± (born blinded), and the terminal assembly assertsblind_commitmentis identical across parts. This issue authors the rules; Smokescreen blinding wiring (fork protocol, three theory backends, custody) #252 supplies the machinery. - The bespoke
.txt/custom-FITS/.npzwriters on the data-vector path are deleted; nothing downstream reads them.
sacc_io is a fixed dependency and the single source of truth for every builder signature. Metadata keys required on every file: catalogue_version, sp_validation_version, created, npatch.
Acceptance
The workflow produces {version}.sacc per version from parts, with only blinded blindable parts persisted and the commitment assertion exercised; the deleted bespoke writers have no remaining readers; the assembled file matches the sacc_io layout contract.
— Fable on behalf of Cail.
Activity
[-]`cosmo_val` + Snakemake migration to the SACC writers[/-][+]Migrate cosmo_val + Snakemake to the SACC writers[/+]cailmdaley commentedon Jul 21, 2026
Notes on terminal-file contents and size, from the #245 review discussion — relevant when this PR decides what the final gather contains.
Size budget (5 bins, 15 pairs, production grids)
The data vectors are trivial; the covariance sets the cost.
Two consequences:
assemble_covariancenow stores covariances block-diagonally (one FITS table per block, Σ block² on disk instead of dense N²; sacc'sconcatenate_data_setspreserves the blocks throughmerge). This removes most of the storage cost of keeping products in one file. In particular, Cℓ BB/EB and ρ/τ are cheap (~17 MB even with full cross-polarisation covariance) — no storage reason to exclude them.{version}.saccand let it persist as its per-part file — Snakemake provenance covers traceability. That keeps the terminal file at tens-of-MB scale with everything scientifically load-bearing inside.Other pieces relevant here
— Claude (Fable) on behalf of Cail.
feat(blinding): wire blind-at-birth custody into the SACC migration DAG
revert(sacc): drop xi_integration from terminal file per #247 ruling
Merge sacc-4: drop xi_integration from terminal gather per #247 ruling
Merge branch 'feat/sacc-4-cosmo-val-sacc' into feat/sacc-7-sacc-like
cosmo_val + Snakemake migration to the SACC writers (PRD #241 row 4) (#…