diff --git a/.github/workflows/deploy-image.yml b/.github/workflows/deploy-image.yml index 6e8eb1206..6ecd4791e 100644 --- a/.github/workflows/deploy-image.yml +++ b/.github/workflows/deploy-image.yml @@ -120,9 +120,8 @@ jobs: docker run --rm -e HYPOTHESIS_PROFILE=ci "$IMAGE" \ pytest -rX -m "not slow" --no-cov tests/science - # The actual test suite, run inside the shipped image — replacing the - # retired conda-based suite. pytest exercises the same wheels, binaries, - # and Python (3.12) that production runs on, not a parallel environment. + # Run the full suite inside the shipped image: the same wheels, + # binaries, and Python (3.12) that production uses. # pyproject's addopts add `--cov=shapepipe`; COVERAGE_FILE is set to /tmp # in the image so it works on read-only filesystems too. The Hypothesis # profile is explicit here so CI always uses the deterministic, capped diff --git a/CLAUDE.md b/CLAUDE.md index d8011f7f3..ee33bf5e2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,15 +5,14 @@ runs the full chain from raw survey images to calibrated shear catalogues — object detection, PSF modelling, and shape measurement — and was used to produce the first UNIONS cosmic-shear release. -The project is now entering a substantial rework of the shape-measurement -pipeline, with the near-term goal of a tight loop between the pipeline and image -simulations for validation and calibration. Development is by a small team; +Development focuses on a tight loop between the shape-measurement pipeline and +image simulations for validation and calibration. Development is by a small team; reproducibility and clarity matter more than breadth. ## Environment — the container is the source of truth ShapePipe is not a stand-alone library: it needs system tools (Source Extractor, -PSFEx, WeightWatcher), MPI, and a specific scientific-Python stack. The supported +PSFEx), MPI, and a specific scientific-Python stack. The supported way to get all of that is the container. - **Dependencies** are declared in `pyproject.toml` as **floor ranges**; exact @@ -105,8 +104,8 @@ Full detail: `docs/source/installation.md` and `docs/source/container.md`. exercises exactly what ships — run them the same way, in the container. - **CI** (`.github/workflows/deploy-image.yml`): every PR and push builds the image and runs the test suite + the example pipeline + binary smokes; pushes to - `develop` / `main` / `master` additionally publish the images. API docs deploy - from `master` (`cd.yml`). + any branch additionally publish the images. See `cd.yml` for versioned API-doc + builds and deployment. - **Style**: PEP 8; numpydoc docstrings on public modules, classes, and methods. Match the surrounding code. diff --git a/Dockerfile b/Dockerfile index 400ee591f..d5682b4c1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,8 +2,8 @@ # # One image: CI tests it, canfar batch jobs run it, downstream stacks (e.g. # sp_validation) build FROM it. Everyday CLI tools and all extras (test, -# lint, doc, jupyter, fitsio, …) are baked in, so there is nothing a job -# runs that CI did not already exercise. +# lint, doc, jupyter, fitsio, …) are baked in. See +# .github/workflows/deploy-image.yml for the checks run before publication. # ---------------------------------------------------------------------- # base — system deps @@ -61,7 +61,7 @@ RUN apt-get update -y --quiet && \ # host's `mpirun` launches one container rank per slot, and the OpenMPI inside # the image wires the ranks together through PMIx. That handshake requires the # container's PMIx to be compatible with the host launcher's. Debian bookworm's -# package is OpenMPI 4.1.4 with PMIx 2.x; modern clusters (e.g. candide) now run +# package is OpenMPI 4.1.4 with PMIx 2.x; modern clusters (e.g. candide) run # OpenMPI 5.0.x with PMIx 5.x, and a PMIx 2 client cannot talk to a PMIx 5 # server — so every rank silently degrades to a standalone "rank 0 of 1" and the # job runs N independent copies instead of one N-rank job. Building OpenMPI diff --git a/README.rst b/README.rst index fc1413031..86c4aa57f 100644 --- a/README.rst +++ b/README.rst @@ -3,8 +3,8 @@ ShapePipe |CI| |CD| |python312| |release| -.. |CI| image:: https://github.com/CosmoStat/shapepipe/workflows/CI/badge.svg - :target: https://github.com/CosmoStat/shapepipe/actions?query=workflow%3ACI +.. |CI| image:: https://github.com/CosmoStat/shapepipe/actions/workflows/deploy-image.yml/badge.svg + :target: https://github.com/CosmoStat/shapepipe/actions/workflows/deploy-image.yml .. |CD| image:: https://github.com/CosmoStat/shapepipe/actions/workflows/pages/pages-build-deployment/badge.svg :target: https://github.com/CosmoStat/shapepipe/actions/workflows/pages/pages-build-deployment @@ -24,8 +24,8 @@ Quickstart ---------- ShapePipe ships as a container image, so you can run the bundled example -pipeline — a single CFIS tile through the full chain — without installing -anything: +pipeline — a chain of demonstration modules — without installing the +scientific stack on the host: .. code-block:: bash @@ -37,8 +37,8 @@ anything: The image is built, tested, and published on every push to the `GitHub Container Registry `_: -``:develop`` tracks the integration branch, release tags (e.g. ``:v1.1.0``) a -stable cut. +``:develop`` tracks the integration branch; branch tags support testing before +merge. See ``.github/workflows/deploy-image.yml`` for publication triggers. Documentation ------------- diff --git a/conftest.py b/conftest.py index f481d46d1..596f1ef28 100644 --- a/conftest.py +++ b/conftest.py @@ -3,7 +3,8 @@ Markers, environment detection, and the candide skip policy live here so every test module — wherever it sits in the tree — sees the same rules. -Markers (also declared in ``pyproject.toml`` so ``--strict-markers`` is on): +See ``pyproject.toml`` for marker declarations and pytest defaults. +Markers used by the suite include: * ``slow`` — heavy compute (minutes), not part of the fast inner loop. * ``candide`` — needs the candide cluster and/or its real on-disk data; @@ -42,10 +43,7 @@ # Candide detection # --------------------------------------------------------------------------- # -# Candide compute / login nodes are named c01, c03, n22..n36, etc. The login -# host this suite is most often driven from is ``c03``. We match the candide -# node-name families rather than a fixed list so new nodes are covered, and -# allow an explicit override for CI or odd hostnames. +# Match Candide node-name families rather than a fixed list of hosts. _CANDIDE_HOST_RE = re.compile(r"^(c\d{2}|n\d{2})$", re.IGNORECASE) @@ -54,7 +52,7 @@ def on_candide(): The check is, in order: an explicit ``SHAPEPIPE_ON_CANDIDE`` override (``1``/``0``), then the hostname against the candide node-name families - (``c0x`` login, ``nXX`` compute; whole bare hostname, so ``c6.nibi.sharcnet`` + (``cXX`` login, ``nXX`` compute; whole bare hostname, so ``c6.nibi.sharcnet`` does not match). Cheap, import-safe, no cluster calls. """ override = os.environ.get("SHAPEPIPE_ON_CANDIDE") @@ -106,9 +104,8 @@ def pytest_collection_modifyitems(config, items): def artifacts_dir(): """Directory where guardrail tests drop plots + status summaries. - The artifacts SEAM: a later GitHub Pages / status step publishes from - here. Created on demand so a clean checkout has nothing to commit until - a test actually emits. + Create ``tests/_artifacts`` when the fixture is requested. Tests write + plots and status summaries here for inspection or publication. """ from pathlib import Path diff --git a/docs/source/clusters.md b/docs/source/clusters.md index 0dd189b74..da7b6082b 100644 --- a/docs/source/clusters.md +++ b/docs/source/clusters.md @@ -29,7 +29,7 @@ protocol as the host launcher (see the candide section below). ## candide (SLURM) -candide uses **SLURM** (`sbatch`; the old `qsub`/PBS commands are gone). The repo +candide uses **SLURM** (`sbatch`). The repo ships ready job scripts in `example/pbs/` — `candide_smp.sh` (single node, parallelised with joblib) and `candide_mpi.sh` (multi-node, hybrid MPI). To run the bundled single-tile example end to end: @@ -88,14 +88,11 @@ post-processing — is documented in the [CANFAR production walkthrough](pipeline_canfar.md). ```{note} -The CANFAR production submission scripts (`scripts/sh/job_sp_canfar*.bash`) still -run under the pre-container environment and are slated for the same -container-first cleanup the candide scripts received. Treat the walkthrough as -the current-but-evolving production procedure. +For the CANFAR production submission scripts (`scripts/sh/job_sp_canfar*.bash`), +follow their environment requirements and the production walkthrough. ``` ## ccin2p3 -ccin2p3 is not yet containerised. The `example/pbs/cc_{smp,mpi}.sh` scripts target -the pre-container environment; a container-first path mirroring candide is future -work. +The `example/pbs/cc_{smp,mpi}.sh` scripts target a host-installed environment. +See their headers for machine-specific requirements. diff --git a/docs/source/container.md b/docs/source/container.md index 704465b47..e3e791f11 100644 --- a/docs/source/container.md +++ b/docs/source/container.md @@ -7,7 +7,7 @@ CANFAR), see [Running on a cluster](clusters.md). ## One image -Every tag (`:develop`, a feature branch, `:v1.1.0`, …) is the same build: +Every published branch tag (`:develop`, a feature branch, …) uses the same build: everyday CLI tools (`vim`, `tmux`, `htop`, `rg`, `fd`, `jq`, `bat`, `less`, `git-lfs`, …) and **all** Python extras (test, lint, doc, jupyter, fitsio, release) on top of core deps. CI runs the test suite inside it before @@ -110,7 +110,7 @@ This is the **single source of truth** for *what kinds of environments shapepipe is compatible with*. Edit when: - adding a new dependency (`uv add foo` will edit it for you) -- bumping a minimum after crossing a major version line +- bumping a minimum when code requires a newer API - adding/restructuring extras - changing project metadata (scripts, version, etc.) @@ -129,9 +129,9 @@ uv add 'foo>=1.2' # adds to pyproject AND uv.lock ``` Commit `uv.lock` to the repo — that's how reproducibility transfers -between machines. The Dockerfile uses `uv sync --frozen`, which fails if -`pyproject.toml` and `uv.lock` have drifted. That's the discipline -mechanism: a stale lockfile cannot ship. +between machines. The Dockerfile uses `uv sync --frozen` to install from the +lockfile without resolving dependencies or checking whether it is current. +Regenerate the lock whenever requirements change. ### `Dockerfile` — the build recipe @@ -161,18 +161,9 @@ The asymmetry is deliberate: Python deps go through pyproject + lockfile versioning). Don't `apt install` something that has a Python wheel; don't `pip install` something Debian packages directly (e.g. `source-extractor`). -## Why this shape - -- **No conda.** Earlier images double-installed packages via conda *and* - pip, ballooning the image and creating environment-resolution bugs. - Dropped in favour of plain Python + uv. -- **`uv sync --frozen`** at build time means the image is bit-exactly - reproducible from a tagged commit, and impossible to ship with a stale - lockfile. -- **Astromatic binaries from Debian** (`psfex`, `source-extractor`) - instead of source builds — Debian carries the - GCC-compatibility patches that the previous Dockerfile had to apply - inline with `sed`. -- **One image** so a canfar batch job, a downstream `FROM` clause, and CI's - test run are never three different environments — `vim`, `pytest`, and - everything else ship on every tag. +## Build choices + +The image uses plain Python and uv for Python dependencies, and Debian's +`psfex` and `source-extractor` packages for the Astromatic binaries. +See `Dockerfile` for the source-built OpenMPI version and host-compatibility +requirements. diff --git a/docs/source/dependencies.md b/docs/source/dependencies.md index 8c9afd630..776dca1c3 100644 --- a/docs/source/dependencies.md +++ b/docs/source/dependencies.md @@ -16,7 +16,7 @@ Scientific stack: |---------|------------| | [Astropy](https://www.astropy.org/) | {cite:p}`astropy:2013,astropy:2018` | | [GalSim](https://github.com/GalSim-developers/GalSim) | {cite:p}`rowe:15` | -| [ngmix](https://github.com/aguinot/ngmix) | {cite:p}`sheldon:15` | +| [ngmix](https://github.com/esheldon/ngmix) | {cite:p}`sheldon:15` | | [MCCD](https://github.com/CosmoStat/mccd) | {cite:p}`liaudat:21` | | [ModOpt](https://cea-cosmic.github.io/ModOpt/) | {cite:p}`farrens:20` | | [python-pysap](https://github.com/CEA-COSMIC/pysap) | | @@ -26,7 +26,6 @@ Scientific stack: | [Matplotlib](https://matplotlib.org/) | {cite:p}`hunter:07` | | [Joblib](https://joblib.readthedocs.io/en/latest/) | {cite:p}`joblib:20` | | [mpi4py](https://mpi4py.readthedocs.io/en/stable/) | {cite:p}`dalcin:05,dalcin:08,dalcin:11` | -| [reproject](https://reproject.readthedocs.io/) | | | [h5py](https://www.h5py.org/) | | Data access & infrastructure (CANFAR / UNIONS): @@ -36,21 +35,18 @@ Data access & infrastructure (CANFAR / UNIONS): | [vos](https://github.com/opencadc/vostools) | CADC / CANFAR VOSpace access | | [skaha](https://github.com/shinybrar/skaha) | CANFAR Science Platform sessions | | canfar | CANFAR container-job submission | -| [astroquery](https://astroquery.readthedocs.io/) | external catalogue queries | | [cs_util](https://github.com/CosmoStat/cs_util) | shared CosmoStat utilities | | [sqlitedict](https://github.com/RaRe-Technologies/sqlitedict) | on-disk pipeline state | ```{note} -`ngmix` is pinned to the -[`aguinot/ngmix@stable_version`](https://github.com/aguinot/ngmix/tree/stable_version) -fork until the fixes land upstream — do not modernise that line (see the note in -`pyproject.toml`). +Git-sourced packages and their selected refs are declared in +`pyproject.toml` under `[tool.uv.sources]`; `uv.lock` pins the resolved commits. ``` ## System Dependencies -The container also provides the non-Python tools ShapePipe calls, all from Debian -packages (no source builds), plus the MPI stack: +The container also provides the non-Python tools ShapePipe calls, from Debian +packages, plus a source-built MPI stack (see `Dockerfile`): | Package | References | |---------|------------| diff --git a/docs/source/installation.md b/docs/source/installation.md index 169a623db..8355df7c2 100644 --- a/docs/source/installation.md +++ b/docs/source/installation.md @@ -1,9 +1,8 @@ # Installation ```{attention} -ShapePipe was not designed to be a stand-alone Python library. Instead users -are expected to install the full ShapePipe environment on the system(s) where -data should be processed. +Running the pipeline requires both the Python package and system executables. +Use the container for the complete supported environment. ``` ## Container Installation (Recommended) @@ -35,7 +34,7 @@ docker pull ghcr.io/cosmostat/shapepipe:develop ``` ```{attention} -We do not currently build images for Apple Silicon/amr64; however the amd64 images should work on these systems, albeit with reduced performance. +We do not currently build images for Apple Silicon/arm64; however the amd64 images should work on these systems, albeit with reduced performance. ``` The image bundles the astromatic binaries (`source-extractor`, `psfex`), diff --git a/docs/source/testing.md b/docs/source/testing.md index 6808c07b2..f397abf13 100644 --- a/docs/source/testing.md +++ b/docs/source/testing.md @@ -7,30 +7,13 @@ the bundled example pipeline. ## The automated test suite The test suite runs with [pytest](https://docs.pytest.org/) under a single -discovery root, `tests/`, with one tier per subdirectory: - -- `tests/module/` — **unit, property-based, and integration** tests for - the analytic surface: pure helpers and geometry (coordinate round-trips, - postage-stamp shapes, the MegaCam CCD flip), tested both by example and with - [Hypothesis](https://hypothesis.readthedocs.io/) property tests where a - function has a clear invariant. Integration tests that need the astromatic - binaries (Source Extractor, PSFEx) build synthetic FITS on the fly rather - than requiring survey data. -- `tests/unit/` — **structural** tests, one parametrized case per file: every - shell script parses (`bash -n`), every example config parses, every - `shapepipe.*` submodule imports, every `[project.scripts]` entry handles - `-h`, and every `*_runner.py` carries its `@module_runner` metadata. Cheap, - broad, and good at catching the dumb-but-real regressions (a syntax error, a - broken import, a renamed entry point). -- `tests/science/` — **fast scientific guardrails** (controlled simulations - with a known answer) and `tests/cluster/` — **candide guardrails** that read - real on-disk catalogs; the latter are marked `candide` and auto-skip off the - cluster. See `tests/README.md` for the full layout and markers. - -**CI is the single gate.** On every pull request and every push to `develop`, -`.github/workflows/deploy-image.yml` builds the image and runs the suite -*inside it* — so the tests exercise exactly the environment that ships. A green -check means the container actually works. +discovery root, `tests/`. See `tests/README.md` for the tiers, markers, and +machine-specific requirements. + +`.github/workflows/deploy-image.yml` builds the image and runs tests inside it +on branch pushes and pull requests targeting the integration or release branches. +See that workflow for the suite, science checks, workflow-DAG checks, and binary +and example smoke runs. ### Running it yourself diff --git a/docs/source/workflow.md b/docs/source/workflow.md index 624a9819d..816364a29 100644 --- a/docs/source/workflow.md +++ b/docs/source/workflow.md @@ -2,8 +2,7 @@ The end-to-end CFIS production run is driven by a [Snakemake](https://snakemake.readthedocs.io) workflow that lives in the -`workflow/` directory of the repository. It replaces the older bit-coded bash -job chain (`job_sp`) that this documentation previously described. +`workflow/` directory of the repository. The workflow owns: @@ -19,6 +18,6 @@ repository. ```{note} The PSF-validation and post-processing configurations that the workflow does -not yet cover are kept separately in `example/cfis/`; see the README there and +not cover are kept separately in `example/cfis/`; see the README there and [Post-processing](post_processing.md). ``` diff --git a/example/cfis/README.md b/example/cfis/README.md index 700f31e02..5a9eae75d 100644 --- a/example/cfis/README.md +++ b/example/cfis/README.md @@ -2,7 +2,7 @@ These configuration files cover PSF validation (`Ms`, `Pl`, `MsPl`, `valjoint`) and assorted post-processing / statistics steps that are **not -(yet) covered by the Snakemake workflow**. They are kept here for reference +covered by the Snakemake workflow**. They are kept here for reference and manual use with `shapepipe_run -c `. The per-module pipeline configurations driven by the Snakemake workflow live diff --git a/example/cfis_image_sims/README.md b/example/cfis_image_sims/README.md index fa0e00ac4..283e5a1c3 100644 --- a/example/cfis_image_sims/README.md +++ b/example/cfis_image_sims/README.md @@ -1,7 +1,9 @@ # cfis_image_sims — module configs for ShapePipe on simulated tiles -This directory is the single `.ini` tree for image-simulation runs. It is -selected by `run_job_sp_canfar_v2.0.bash -t image_sims` (which sets +This directory holds the configs for bash-driven image-simulation runs. +For Snakemake runs, use `workflow/config/cfis_image_sims/` and see +[`workflow/README.md`](../../workflow/README.md). +The bash config tree is selected by `run_job_sp_canfar_v2.0.bash -t image_sims` (which sets `config_dir = example/cfis_image_sims`, `retrieve = symlink`, and forces `tile_det = sx`) and consumed stage-by-stage by `job_sp_canfar_v2.0.bash`. @@ -10,18 +12,15 @@ loops over the bits set in `-j`, initialises the tile/exposure work directories, runs each completeness check, and delegates the actual `shapepipe_run` calls to `job_sp_canfar_v2.0.bash -j `, which is where the `.ini` file for each bit is selected. A few bits are special-cased for sims *before* that delegation — -those cases are documented in the last column below and, at more length, under -[Sim special-casing](#sim-special-casing). +those cases are documented in the last column below. ## Job-bit dispatch Every row is derived from the two bash scripts. "Module(s)" is the ShapePipe runner(s) the selected `.ini` names; "`.ini` selected" is what `job_sp_canfar_v2.0.bash` picks for that bit under sim settings -(`retrieve=symlink`, `psf=psfex`, `tile_det=sx`). Bit 32 (mask exposures) is -gone: ShapePipe generates no masks (PR #847), so the bash scripts' -`star_cat_for_mask` setting and the `config_*_Ma_*.ini` configs it selected no -longer exist. +(`retrieve=symlink`, `psf=psfex`, `tile_det=sx`). There is no bit-32 masking +stage: ShapePipe generates no masks. | Bit | Stage | Module(s) | `.ini` selected (sim settings) | Sim special-casing | |----:|-------|-----------|--------------------------------|--------------------| @@ -51,22 +50,6 @@ Notes on cross-cutting conventions used above: bit 256 always runs SExtractor detection and bit 512 selects the `_sx`-suffixed vignet/ngmix/PiViVi configs. -## Sim special-casing - -Three bits diverge from the data chain, all handled in -`run_job_sp_canfar_v2.0.bash` before the per-bit delegation: - -- **Bit 2 — faked weight-uncompress.** Sim weights ship uncompressed, so instead - of running `uncompress_fits_runner`, run_job fabricates the expected Uz output - directory and symlinks `CFIS_simu_weight-.fits` in, so downstream - `last:uncompress_fits_runner` references resolve. -- **Bit 64 — placeholder.** run_job writes a placeholder completeness log and - runs nothing; the sim PSF is produced by `fake_psf_runner` inside bit 512. -- **Bit 512 — fake PSF + vignets.** run_job runs `fake_psf_runner` - (`config_exp_psfex.ini`) followed by the two `vignetmaker_runner` runs - (`config_tile_PiViVi_canfar_sx.ini`). `fake_psf_runner` consumes the sexcat - from bit 256, so bit 256 must precede it. - ## Parallelism model and the `PSF_DICT` requirement The orchestrator (the sp_validation Snakemake workflow) fans out one job per diff --git a/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini b/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini index 26564e4fa..d42a10bdd 100644 --- a/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini +++ b/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini @@ -119,9 +119,8 @@ STAMP_SIZE = 51 # Output file name prefix, file name is vignet.fits PREFIX = -# Additional parameters for path and file pattern corresponding to single-exposure -# run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for -# the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. +# Discover single-exposure output directories under $SP_EXP using the +# runner names and file patterns below. ME_IMAGE_EXP_DIR = $SP_EXP ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner ME_IMAGE_PATTERN = flag, image, weight diff --git a/profiles/candide/config.yaml b/profiles/candide/config.yaml index 2932e64eb..bc82c8ee2 100644 --- a/profiles/candide/config.yaml +++ b/profiles/candide/config.yaml @@ -12,41 +12,33 @@ default-resources: # comp nodes have TmpDisk=0, so the tile_shape group (--tmp, see tile.smk's # TILE_SLURM_EXTRA) lands on pscomp; everything else may use either. slurm_partition: "comp,pscomp" - # n09/n17/n36: excluded in the sp_validation candide profile; n23: slurm jobs - # failed there (sp_validation 2b40c99). + # n09/n17/n36 follow sp_validation's candide exclusions; n23 has measured + # SLURM job failures (sp_validation commit 2b40c99). slurm_extra: "'--exclude=n09,n17,n23,n36'" software-deployment-method: [apptainer] -# /local/scratch is where tile.smk puts the node-local tile store (TILE_LOCAL, -# ~5.6 GB per real-data tile). candide has no /local, and its node-local /scratch -# is writable only per job (/scratch/$USER/$SLURM_JOB_ID, made by the prolog), -# which this line cannot name: snakemake passes it through literally, and -# --cleanenv drops SLURM_JOB_ID inside the container. So the node's /tmp (local -# disk, ~29 GB free, sticky) is bound there instead. That holds ~5 concurrent -# tile stores per node, too few for dense image-sim tiles: such a campaign sets -# `tile_store_root:` in its run config, and bin/sp replaces this bind's source in -# the launch snapshot (see workflow/config.yaml). The PYTHONPATH pin is rewritten by bin/sp to the -# launch snapshot's src/; the value here only has to be a checkout's src/. +# Bind node /tmp (~29 GB free, local disk) to tile.smk's /local/scratch: +# candide has no /local, and its writable /scratch path requires a job ID. +# Variable expansion limits are documented in profiles/nibi/config.yaml. +# The disk holds ~5 real-data tile stores (~5.6 GB each); dense image-sim runs +# can set tile_store_root in the run config (workflow/config.yaml). +# PYTHONPATH names a checkout's src/; workflow/bin/sp rewrites the launch copy. apptainer-args: "--cleanenv --env OMP_NUM_THREADS=1 --env MALLOC_ARENA_MAX=2 --env PYTHONPATH=/n17data/mkilbing/astro/repositories/github/shapepipe/src --bind /home,/automnt,/n17data,/n23data1,/n09data --bind /tmp:/local/scratch" latency-wait: 60 # NFS: wait for outputs to appear after a job keep-going: true # a failed job poisons only its cone; siblings run on -# Transient node-level failures are the common case on a shared cluster: -# a fused tile_shape group died at joblib's fork_exec with BlockingIOError -# [Errno 11] because the node was out of process slots, killing one tile of -# 39 outright. Without this, attempt is always 1, so every -# `mem_mb = lambda wc, attempt: N * attempt` in tile.smk is dead code and a -# one-off OOM or EAGAIN is fatal. Safe for the fused group: a retry -# resubmits the same member set, tile_vignets included (tile.smk's -# "one sharp edge" note). tile_ngmix keeps its own rule-level retries: 2. +# Retry transient node failures, including measured process-slot exhaustion +# (BlockingIOError, Errno 11) in tile_shape. Retries also activate attempt-scaled +# memory requests. See tile.smk for group retry semantics; tile_ngmix sets 2 +# retries at rule level. retries: 2 rerun-incomplete: true # re-do jobs left incomplete by an unclean death show-failed-logs: true printshellcmds: true rerun-triggers: [mtime, params, code, software-env] -# The tile_shape group (tile.smk) sets its own slurm_extra (TILE_SLURM_EXTRA, -# the --tmp floor), which REPLACES the default above and so drops the node -# excludes: its jobs landed on n09 and n17. Restate both, identically on all -# four members (a group's string resources must agree). Only slurm_extra is -# overridden; the rules keep their own threads and attempt-scaled mem_mb. +# @sc [label:coupling] candide-group-slurm-extra +# tile_shape's rule-level slurm_extra replaces the profile default. Set both +# the --tmp floor and node exclusions identically on all four member rules: +# a group's string resources must agree. Leave rule threads and attempt-scaled +# memory intact. set-resources: tile_vignets: slurm_extra: "'--tmp=16000 --exclude=n09,n17,n23,n36'" diff --git a/profiles/nibi/config.yaml b/profiles/nibi/config.yaml index 76cf19e70..265c49541 100644 --- a/profiles/nibi/config.yaml +++ b/profiles/nibi/config.yaml @@ -1,14 +1,12 @@ # Snakemake profile for the Nibi cluster (Digital Research Alliance). # -# SLURM-EXECUTOR mode (PRD #848 D-profile): one SLURM job per rule instance, -# each carrying its rule's own attempt-scaled resources (cpus_per_task, -# mem_mb, runtime). Snakemake feeds the queue as jobs finish, so DR6's ~170k -# total jobs never queue at once, and multi-node scaling is inherent — this -# supersedes the earlier one-allocation/local-scheduler mode. +# The SLURM executor submits one job per rule instance or fused group, using +# the rules' resources. Snakemake feeds the queue as jobs finish, so DR6's +# ~170k jobs do not queue at once. # # Launch via `workflow/bin/sp` (loads apptainer/1.4.5, uses the /project venv). # software-deployment-method wraps every job's shell in `apptainer exec` — the -# user never types apptainer. WHICH image is resolved in the Snakefile through +# user never types apptainer. The image is resolved in the Snakefile through # workflow/scripts/container.py, whose docstring documents the layers and the # resolution order; `sp container status` prints which layer is live. @@ -28,93 +26,22 @@ default-resources: slurm_account: def-mjhudson_cpu software-deployment-method: [apptainer] -# Explicit container environment (finding 2/3): the apptainer SDM otherwise drops -# the proven-recipe env and hardcodes --home , hiding ~/.ssl/cadcproxy.pem. -# --cleanenv strict host-env isolation (APPTAINERENV_*/SINGULARITYENV_* survive) -# OMP_NUM_THREADS=1 caps OpenBLAS fork-explosion (verified: pool 32->1) -# MALLOC_ARENA_MAX=2 bounded allocator (im_sims/nibi lesson) -# --home /home/cdaley wins over the SDM's --home ; restores cadcproxy.pem for vos/vcp -# >>> EDIT THIS: THE ONE NON-PORTABLE LINE IN THIS FILE. <<< -# The apptainer-args string below hardcodes one user's home -# (--home /home/cdaley). A different user must edit it by hand. -# IT CANNOT BE FIXED HERE: YAML cannot splice an env var, and -# snakemake escapes a literal `$` before the string reaches -# any shell (see the $SLURM_TMPDIR post-mortem below, which -# cost a whole campaign). Nor can bin/sp inject it through -# --apptainer-args: that REPLACES the profile value rather -# than appending to it, so the launcher would have to restate -# this whole string -- and workflow/scripts/container.py reads -# THIS LINE to give `sp container exec` the same environment -# jobs get, so a second definition is exactly the divergence -# that mechanism exists to prevent. -# THE PYTHONPATH PIN IS THE ONE EXCEPTION, and it is not -# edited by hand: `sp run` copies this whole file into the -# campaign's code snapshot and rewrites that ONE path to the -# snapshot's src/ (bin/sp, "the launch code snapshot"). Still -# one source of truth for the flags -- the copy is generated, -# never edited -- and jobs stop reading a live checkout. -# --bind /local NODE-LOCAL NVME. $SLURM_TMPDIR on nibi is -# /local/scratch/..0, and the mount is /local -# (/dev/nvme0n1, ext4, 3.5 TB) -- binding /localscratch is -# NOT enough, the real path is under /local (verified, -# probe job 20794182). The fused `tile_shape` group job -# WRITES its 5.6 GB vignette store there and never puts it -# on scratch at all (workflow/rules/tile.smk); every other -# rule simply ignores the bind. Adding it does NOT -# invalidate anything: the `software-env` rerun trigger -# hashes only job.container_img_url (+ conda env, env -# modules) -- -# snakemake/persistence/__init__.py::_software_stack_hash, -# v9.23.1 -- and apptainer-args is not in that hash. -# -# HOW the in-container shell LEARNS its node-local dir: -# IT DOES NOT. It DERIVES the path from the tile wildcard -# (tile.smk::tile_local -> /local/scratch/sp--), and -# that is the only mechanism that survives contact with -# snakemake. Three alternatives were tried; all fail, and -# the third failed IN PRODUCTION, so read this before -# reaching for any of them again: -# * {resources.tmpdir} is NOT $SLURM_TMPDIR. Snakemake -# defers that resource to the job and evaluates it as -# tempfile.gettempdir(); nibi's Slurm does not export -# TMPDIR, so it lands on /tmp -- which on a compute -# node is a 378 GB RAM-backed tmpfs whose pages are -# charged to the job's memory cgroup. Staging 6 GB -# there would eat the job's RAM, not use the NVMe. -# (Measured, job 20794656.) -# * APPTAINERENV_SLURM_TMPDIR would have to be exported -# by something that already knows the JOB's value; -# bin/sp runs on the login node, where it is unset. -# * SPLICING "$SLURM_TMPDIR" INTO apptainer-args AND -# LETTING THE JOB'S SHELL EXPAND IT DOES NOT WORK. -# This was the shipped design for exactly one campaign -# and it killed every tile of run 20798193 in eight -# minutes: snakemake ESCAPES the `$` before the string -# reaches any shell, so the container receives the -# literal characters and never a path. NOTHING can be -# carried into the container this way -- not a tmpdir, -# not anything else. The bind below is static, and the -# VALUE is derived job-side from a wildcard instead. -# Every tile_shape member still REFUSES TO RUN if the -# derived directory is not real (tile.smk), which is the -# only reason that campaign cost minutes and not a night: -# a broken node-local path is a loud failure, never a -# silent slide back onto NFS, and never a 5.6 GB write -# into the RAM-backed /tmp tmpfs. -# PYTHONPATH SETTLED CALL 3 — pins THIS branch's src/, which is -# develop@97e16d50 plus the four commits that genuinely need -# module code: ngmix chunk fields + position-seeded RNG, -# merge_sep_cats chunk paths, the vizier star-cat helpers, -# and the cherry-picked #873 (seeded setools split). -# NOT shapepipe-prod (drifted to a PR branch mid-run, and the -# live p3-batch1 job reads it) and not the sif default -# (frozen pre-#843, and pre-#873). Production later rebuilds -# the sif at the validated commit and DROPS this --env line. -# The path written HERE is the checkout's, and is what -# `sp container exec` uses; the value JOBS see is the -# snapshot's src/, substituted per the paragraph above. So a -# running campaign holds its launch code even after this -# line (or the tree it names) changes. +# Explicit container environment: +# --cleanenv isolates host variables (APPTAINERENV_*/SINGULARITYENV_* survive). +# OMP_NUM_THREADS=1 caps thread pools (measured OpenBLAS pool: 32 -> 1). +# MALLOC_ARENA_MAX=2 bounds allocator arenas. +# --home overrides Snakemake's workdir home so vos/vcp can read cadcproxy.pem. +# A different user must edit the literal --home path below. +# PYTHONPATH selects checkout modules for `sp container exec`; `sp run` rewrites +# it in the launch snapshot. See workflow/bin/sp's launch code snapshot note. +# +# @sc [label:coupling] nibi-static-container-paths +# apptainer-args must use literal paths: YAML does not interpolate variables, +# and Snakemake escapes `$` before the shell sees it. Bind /local, the NVMe +# mount (3.5 TB ext4), so tile.smk::tile_local can derive a campaign/tile path +# beneath /local/scratch. A login-node export cannot supply SLURM_TMPDIR for a +# future job; {resources.tmpdir} resolves to /tmp on nibi, a measured 378 GB +# RAM-backed tmpfs charged to the job's memory cgroup, not to NVMe storage. apptainer-args: "--cleanenv --env OMP_NUM_THREADS=1 --env MALLOC_ARENA_MAX=2 --env PYTHONPATH=/project/def-mjhudson/cdaley/shapepipe-snakemake/src --home /home/cdaley --bind /project --bind /scratch --bind /local" latency-wait: 60 # NFS: wait for outputs to appear after a job @@ -123,35 +50,16 @@ rerun-incomplete: true # re-do jobs left incomplete by an unclean deat show-failed-logs: true printshellcmds: true -# No `keep-incomplete` here. The declared output IS the manifest, so letting -# snakemake delete a failed job's output is exactly the manifest-vs-log -# semantics this workflow wants (completeness.py's docstring argues it), and -# `show-failed-logs` above surfaces the surviving log on the console. +# No keep-incomplete: Snakemake deletes failed manifests and retains logs. +# See completeness.py for manifest/log semantics; show-failed-logs prints logs. -# rerun-triggers: the v9 default MINUS `input`. params/code/mtime fixes still -# propagate — completeness.py writes the manifest only on change, so mtimes move -# only when reality moves. -# -# `input` is dropped because reclamation needs the tile->exposure edge to be -# CONDITIONAL; with the trigger on, that conditional itself reruns every -# finished tile against a reclaimed exposure store. tile.smk's tile_finished -# commentary carries the mechanism and the fixture-t4 job counts. -# -# Nothing this workflow relied on is lost. The two genuinely data-derived input -# sets are covered another way: a changed exposure list arrives through the -# (non-ancient) find_exposures manifest's mtime, and the ngmix chunk count rides -# in params. And clean scheduling is structurally gated to bin/sp (SP_PHASE=compute -# + this profile), so a bare snakemake invocation cannot quietly recombine -# reclamation with the `input` trigger. +# Exclude the input trigger: tile.smk::tile_finished conditionally removes +# reclaimed exposure edges, and that change must not rerun finished tiles. +# Exposure-list changes propagate through find_exposures manifest mtimes; +# ngmix chunk counts are params. See tile.smk for reclaimed-edge handling. rerun-triggers: [mtime, params, code, software-env] -# NO set-threads / set-resources here, deliberately. Profile overrides REPLACE a -# rule's own values (verified snakemake 9.23), which would kill the -# attempt-scaled `mem_mb = lambda wc, attempt: ...` OOM retries and the tuned -# ngmix thread count, and would flatten the group resource composition. The -# RULES own threads and resources; this profile only sets defaults for rules -# that state nothing (default-resources above). -# -# `group:` fusion of the short rules (PRD D-profile) lives in the rule files, -# each label documented in its file's docstring: `tile_prep` (prepare.smk), -# `exp_short` (exposure.smk) and `tile_shape` (tile.smk). +# No set-threads/set-resources: overrides replace rule values (Snakemake 9.23), +# including attempt-scaled memory and ngmix threads. Rules own resource sizing +# and grouping; this profile supplies defaults only. See prepare.smk for +# tile_prep and tile.smk for tile_shape; exposure rules are ungrouped. diff --git a/pyproject.toml b/pyproject.toml index 53f0dae61..8457a38fd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,14 +11,14 @@ authors = [ license = "MIT" license-files = ["LICENSE"] requires-python = ">=3.12" -# Dependencies use FLOOR RANGES; exact versions live in `uv.lock`, the +# Dependencies use floor ranges; exact versions live in `uv.lock`, the # reproducible manifest generated by uv (never hand-edited). Renovate updates # the lock (rangeStrategy=update-lockfile, see renovate.json), so uv's resolver # owns version selection — cross-package constraints (e.g. numba capping numpy) # are resolved jointly instead of failing update PRs. Bump a floor only when # the code actually requires the newer API. dependencies = [ - "astropy>=7.0", # major 6 → 7 + "astropy>=7.0", "canfar", "cs_util>=0.2.1", "galsim>=2.8", @@ -31,8 +31,8 @@ dependencies = [ "modopt>=1.6", "mpi4py>=4.1.2", "numba>=0.59", # numpy 2 support - "numpy>=2.4.6", # major 1 → 2 - "pandas>=3.0.3", # major 2 → 3 + "numpy>=2.4.6", + "pandas>=3.0.3", "python-dateutil", "python-pysap>=0.3", "PyQt5", @@ -96,10 +96,8 @@ environments = ["sys_platform == 'linux'"] [tool.uv.sources] ngmix = { git = "https://github.com/esheldon/ngmix", tag = "v2.4.1" } -# Track cs_util's develop branch (a git ref, not a pinned release) so new -# cs_util.size / cs_util.shape helpers are available without shipping a release -# and rebuilding the image each time. shapepipe's column grammar sources size -# (T = 2*sigma^2) and ellipticity conversions from cs_util. +# The column grammar uses cs_util's size (T = 2*sigma^2) and ellipticity +# conversions. Resolve develop at lock generation; uv.lock pins the commit. cs_util = { git = "https://github.com/CosmoStat/cs_util", branch = "develop" } [build-system] @@ -120,11 +118,8 @@ script-files = [ where = ["src"] [tool.ruff] -# Ruff is the single lint + format tool, replacing black, isort, and the -# pytest-pycodestyle / pytest-pydocstyle plugins (the latter broke on the -# pytest 9.1 hookspec change). Rules below cover pycodestyle (E/W), Pyflakes -# (F), import sorting (I), and numpydoc-style docstrings (D) — matching the -# project style: PEP 8 + numpydoc. +# Ruff handles linting and formatting: pycodestyle (E/W), Pyflakes (F), +# import sorting (I), and numpydoc-style docstrings (D). line-length = 79 target-version = "py312" src = ["src", "tests"] diff --git a/scripts/python/create_final_cat.py b/scripts/python/create_final_cat.py index 14a67a95f..741f4869c 100755 --- a/scripts/python/create_final_cat.py +++ b/scripts/python/create_final_cat.py @@ -2,9 +2,13 @@ """Script create_final_cat.py -Create and update hdf5 file of all per-tile final catalogues, the output of -ShapePipe module ``make_cat_runner`` (hdf5, one dataset per column; FITS for -catalogues written before that format). Supercedes `merge_final_cat.py`. +Create or append per-tile final catalogues to an HDF5 file, one structured +dataset per tile under ``patches/``. Inputs are ``make_cat_runner`` +outputs in HDF5 (one dataset per column) or FITS format. + +Existing tile datasets are skipped, even if their sources change. For a +workflow output reconciled against campaign membership and source changes, +use ``workflow/scripts/merge_final_cat.py``. Usage: in parent dir of patches: create_final_cat.py -p ~/shapepipe/workflow/config/cfis/final_cat.param -i . -P 7 -v -m final_cat_P7.hdf5 @@ -30,17 +34,13 @@ def params_from_run_config(params, defaults): - """Fill unset paths from a workflow run config. + """Replace default-valued paths from an image-simulation workflow run config. - The workflow already knows where a campaign writes, so a manual merge - should not have to restate it. Resolution goes through the workflow's own - resolver (workflow/scripts/run_config.py), layering the run config on - workflow/config.yaml and then the input_types: and machines: tables, so - what lands here is what the rules would have used. + Use ``workflow/scripts/run_config.py`` for config resolution and precedence. + Derive image-simulation paths only; data patch naming is not handled here. - Only values still at their default are filled -- an explicit flag always - wins. Nothing is derived for the data path: its patch naming differs and - is not this function's business. + Values equal to their defaults are replaced, including explicitly supplied + flags with default values. Nondefault flag values win. """ repo = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) sys.path.insert(0, os.path.join(repo, "workflow", "scripts")) @@ -111,8 +111,7 @@ def set_params_from_command_line(args): del options - # A run config fills in whatever is still at its default - # (see the docstring); explicit flags always win. + # See params_from_run_config for default-value replacement semantics. if _params.get("run_config"): _defaults, _, _, _ = params_default() _params = params_from_run_config(_params, _defaults) @@ -178,10 +177,7 @@ def params_default(): def read_param_file(path, verbose=False): - """Read Param File. - MKDEBUG TODO: Move to cs_util. Also used in sp_val/cat.py. - - Return parameter list read from file. + """Return unique parameter names in file order. Parameters ---------- @@ -218,10 +214,8 @@ def read_param_file(path, verbose=False): print("No parameters read", end="") print(" into merged catalogue") - # Ordered dedup. list(set(...)) reordered the columns by the process's - # string hash seed, so two runs of this tool over the same inputs produced - # files whose datasets differed in column ORDER — which is part of a - # structured dtype, and therefore part of the file. + # Preserve parameter-file order while removing duplicates; column order + # is part of the output's structured dtype. param_list_unique = list(dict.fromkeys(param_list)) if verbose: @@ -406,20 +400,14 @@ def read_data(cat_file, params): def copy_data(param_list, extracted_data, dtype): - """Copy Data. + """Copy requested columns into a structured array with their stored dtypes. + @sc [label:schema] final-copy-requested-column-order + Allocate only requested columns present in the source, in parameter-file + order. Unrequested fields must not enter the output as uninitialised memory, + and source column order must not make tile dtypes incompatible at + concatenation. ``read_data`` enforces missing-column errors before this call. """ - # THE REQUESTED COLUMNS ONLY, IN THE PARAMETER FILE'S ORDER. Two things - # are being fixed here and they are easy to conflate. Allocating with the - # source's full dtype and filling only the requested columns left every - # other column as uninitialised memory — meaningless values, and different - # bytes on every run over the same inputs. And ordering the result by the - # SOURCE catalogue's columns made the output dtype a property of the - # catalogue rather than of the parameter file: two tiles written by - # different ShapePipe versions, whose catalogues order or extend their - # columns differently, then landed in one merged file with two different - # structured dtypes, which np.concatenate refuses. The parameter file is - # the schema; it says which columns AND in what order. wanted = set(dtype.names or ()) columns = [col for col in param_list if col in wanted] subset = np.dtype([(col, dtype[col]) for col in columns]) @@ -465,10 +453,8 @@ def collect_tile_ids_image_sims(patch_path): list of (tile_id, tile_path) tuples """ id_pattern = re.compile(r"^\d+\.\d+$") - # Two layouts. The Gen-2 / native runs keep tiles under the patch dir - # itself; the unified workflow (shapepipe #891) publishes to a separate - # products root, and clean_tile deletes the run-dir copies -- so on a - # reclaimed campaign /product/tiles holds the ONLY catalogues. + # Use the first existing tiles root: direct, product/, then products/. + # See workflow/scripts/clean_tile.py for persistent catalogue ownership. result = [] for sub in ("tiles", os.path.join("product", "tiles"), os.path.join("products", "tiles")): @@ -487,7 +473,7 @@ def collect_tile_ids_image_sims(patch_path): return result -# make_cat_runner writes hdf5; catalogues made before that are FITS. +# Prefer HDF5 when both supported formats exist. FINAL_CAT_SUFFIXES = (".hdf5", ".fits") @@ -588,9 +574,7 @@ def process(params): structured_data = copy_data(params["param_list"], extracted_data, dtype) - # Create a new dataset. dtype comes from the array copy_data - # built, not from the source catalogue: they differ now that - # copy_data allocates the requested columns alone. + # Use copy_data's selected-column dtype, not the source dtype. try: patch_group.create_dataset( str(id), @@ -611,7 +595,8 @@ def process(params): def single_action(params): """Single Action. - Perform single-ID action. + Check or remove one ID. ``add`` returns ``None`` and lets ``main`` run the + ordinary catalogue walk; it does not restrict that walk to the requested ID. Parameters ---------- @@ -621,7 +606,8 @@ def single_action(params): Returns ------- int - return value: 0 (success), 1 (failure), ``None`` (no action performed) + 0 after check or remove; ``None`` when no single-ID action is performed. + The check branch returns 0 even if the ID is absent. """ if params["single_op"] == "check": @@ -631,7 +617,7 @@ def single_action(params): if found == "": # ID not found res = 1 - # ID found + # This assignment also runs when the ID is absent. res = 0 elif params["single_op"] == "remove": diff --git a/scripts/python/run_breakdown_grid.py b/scripts/python/run_breakdown_grid.py index 1e6c9a12d..ad0fe4dd2 100644 --- a/scripts/python/run_breakdown_grid.py +++ b/scripts/python/run_breakdown_grid.py @@ -1,33 +1,32 @@ #!/usr/bin/env python -"""Statistically rigorous resolution x noise breakdown grid (v2). +"""Paired-shear calibration grid over galaxy resolution and image noise. -The Tier-1 companion of ``run_mbias.py``. Where that recipe reports the +This complements ``run_mbias.py``. Where that recipe reports the multiplicative bias at a single well-resolved, high-S/N operating point, this one sweeps two axes to *map* where the metacal shear calibration stays within tolerance and where it breaks down: galaxy size relative to the PSF (the resolution ratio ``r = gal_hlr / psf_fwhm``) and image noise (three S/N bands). -**What changed from v1.** v1 ran a single-signed arm per axis and reported bare -``m = //gamma - 1`` with no uncertainty; at moderate/low S/N the per-seed -shear noise gives sigma_m ~ 0.5, so v1's scary S/N trends were *unmeasured*. v2 -makes every number carry an honest bootstrap error and cancels measurement noise -by pairing signed arms that share an identical noise realisation: +Signed arms share a noise realisation to reduce measurement noise; cell +estimators carry bootstrap-over-seeds errors: * **Paired shear response.** Six arms per seed -- (g1+, g1-), (g2+, g2-), - (psf+, psf-) -- driven so the noise draw and sub-pixel shift are BIT-IDENTICAL + (psf+, psf-) -- driven so the noise draw and sub-pixel shift are identical across arms of a seed (same ``RandomState(seed+1000)`` into ``make_data``; galsim ``.shear()`` adds no rng draw). The +/- difference then cancels the - shared shape noise that dominated v1. ``m1 = ( - )/(2 gamma + shared measurement noise. ``m1 = ( - )/(2 gamma Rbar11) - 1`` with ``Rbar11`` the mean R11 over both g1 arms. * **Additive bias from a PSF-shear pair.** ``c1 = ( - )/(2 Rbar11_psf)``, leakage ``alpha = c1 / 0.05``. c2 (expected 0) is a null check. * **Bootstrap-over-seeds errors** (B=1000) on every cell estimator. -* **Degeneracy guard.** When ``|Rbar| < 5 se(Rbar)`` the m is set null with - ``degenerate: true`` rather than dividing by a response consistent with zero. -* **Legacy estimator** (v1's ratio-of-averages) carried alongside for continuity. +* **Degeneracy guard.** When ``|Rbar| < 5 se(Rbar)``, the paired m and its + error are null and ``m{axis}_degenerate`` is true. Additive estimates use the + same guard and ``c_degenerate``; single-arm estimators are not guarded. +* **Single-arm ratio-of-averages estimator**, stored in ``m*_legacy`` fields + alongside the paired results. -**Production-parity knobs (S4).** Three CLI knobs turn the clean-room grid into -production-settings ablations, each defaulting to the clean baseline: +Production-setting ablations vary the centroid source, drawing WCS and epoch +count: * ``--centroid-source {wcs,hsm}`` — "wcs" (default, as in production): the ngmix jacobian centre is the coadd centroid, the sub-pixel offset the stamp @@ -39,25 +38,11 @@ * ``--wcs-g1/--wcs-g2/--wcs-theta-deg`` — shear/rotate the drawing WCS jacobian away from a pure pixel scale (the esheldon/ngmix#72 sensitivity axis, where a mishandled jacobian produced m ~ -0.2 at wcs_g1=0.1). -* ``--n-epochs`` — pre-existing; n_epochs=2 matches the old CI operating point. - -Estimator math is exact; the finite-difference response step is hardcoded to -``do_ngmix_metacal``'s mainline value (Spec-Code Invariant: if that line moves, -``STEP`` moves with it). - -Runs ONLY inside the ShapePipe apptainer container with the live source tree -bound over the baked copy -- same invocation as ``run_mbias.py`` (two binds: the -live shapepipe src AND the live cs_util, since mainline ngmix.py imports -``cs_util.size``). Cells are embarrassingly parallel (``--jobs``); the full -24-cell production grid is driven by the orchestrator on SLURM, not here. - - apptainer exec \\ - --bind /home,/scratch,/automnt,/n17data,/n23data1,/n09data \\ - --bind /n17data/cdaley/unions/code/shapepipe/src/shapepipe:/app/src/shapepipe \\ - --bind /n17data/cdaley/unions/code/cs_util/cs_util:/app/.venv/lib/python3.12/site-packages/cs_util \\ - /n17data/cdaley/containers/shapepipe_ngmix_v2.0-dev.sif \\ - python scripts/python/run_breakdown_grid.py \\ - --output /results/baseline/breakdown_grid +* ``--n-epochs`` — epochs per object (default 1). + +See ``STEP`` for the finite-difference response coupling. Run in the supported +ShapePipe container; ``docs/source/container.md`` describes source overrides. +Cells run in parallel with ``--jobs``; the default grid has 24 cells. Emits ``/breakdown_grid_v2.json`` (meta + provenance + per-cell summaries with all errors) plus a mirror ``breakdown_grid.json`` for the status-page reader, @@ -85,9 +70,9 @@ ) from shapepipe.testing.simulate import make_data -# Metacal finite-difference step: R = (g1p - g1m) / (2*step). Hardcoded to -# do_ngmix_metacal's mainline value (ngmix.py); the Spec-Code Invariant says if -# that line moves, this must move with it. +# @sc [label:coupling] breakdown-grid-metacal-step +# STEP must match do_ngmix_metacal's finite-difference step in +# src/shapepipe/modules/ngmix_package/ngmix.py: R = (g1p - g1m) / (2*STEP). STEP = 0.01 # PSF-shear amplitude for the additive-bias (leakage) arms; c1 = dgn / (2*this). PSF_SHEAR = 0.05 @@ -102,6 +87,8 @@ N_SEEDS_DEFAULT = {"high": 100, "moderate": 400, "low": 1600} # The six signed arms: (shear, psf_shear). g1/g2 probe multiplicative response; # psf probes additive leakage. Arms of one seed share a noise realisation. +# Galaxy shear is fixed at +/-0.02 here; --gamma changes estimator +# normalisation only, not the injected arms. ARMS = { "g1p": ((0.02, 0.0), (0.0, 0.0)), "g1m": ((-0.02, 0.0), (0.0, 0.0)), "g2p": ((0.0, 0.02), (0.0, 0.0)), "g2m": ((0.0, -0.02), (0.0, 0.0)), @@ -113,11 +100,12 @@ def build_wcs(g1, g2, theta_deg): - """Uniform drawing WCS from the S4 parity knobs. + """Uniform drawing WCS from shear and rotation settings. - All-zero knobs return None -> make_data's bit-identical legacy PixelScale - path. Otherwise: shear the pixel scale (area-preserving, det = scale**2 -- - the esheldon/ngmix#72 construction), then rotate by theta_deg.""" + All-zero settings return None for make_data's PixelScale path. Otherwise, + shear the pixel scale with area preserved (det = scale**2), then rotate + by theta_deg. + """ if g1 == g2 == theta_deg == 0.0: return None jac = galsim.ShearWCS(PIXEL_SCALE, galsim.Shear(g1=g1, g2=g2)).jacobian() @@ -145,12 +133,13 @@ def one_stamp(noise, gal_hlr, psf_fwhm, img_size, n_epochs, shear, psf_shear, seed, true_noise, wcs=None, centroid_source="wcs"): """One injected-truth realisation -> per-arm record dict. - Uses ``RandomState(seed)`` for the fit rng and ``RandomState(seed+1000)`` for - make_data, so every arm of a given seed sees the IDENTICAL noise draw and - sub-pixel shift (galsim ``.shear()`` consumes no rng) -- the property the - paired estimator rests on. ``true_noise`` routes the per-pixel true-inverse- - variance path (see ``--true-noise``). ``wcs``/``centroid_source`` are the - S4 parity knobs (see module docstring).""" + @sc [label:coupling] breakdown-grid-paired-noise + Use ``RandomState(seed)`` for fitting and ``RandomState(seed+1000)`` for + drawing in every arm of a seed, so paired arms share noise and sub-pixel + shifts. GalSim ``.shear()`` consumes no random draw. + + ``true_noise`` supplies per-pixel true inverse variance; ``wcs`` and + ``centroid_source`` control the ablations described in the module docstring.""" rng = np.random.RandomState(seed) prior = get_prior(PIXEL_SCALE, rng) gals, psfs, _, weights, flags, jacobs, centers = make_data( @@ -172,10 +161,8 @@ def one_stamp(noise, gal_hlr, psf_fwhm, img_size, n_epochs, shear, psf_shear, res = do_ngmix_metacal(stamp, prior, 1.0, rng, centroid_source=centroid_source) except (BootPSFFailure, BootGalFailure): - # Production treats a bootstrap failure as a flagged object; the - # pair-drop policy drops the seed and fail_frac records the rate (a - # SYSTEMATIC failure mode still surfaces loudly there). Without this, - # one unlucky fit in ~1e5 kills the entire grid (job 808259). + # Record a failed fit rather than aborting the grid. Pair masks drop + # affected seeds from each estimator; fail_frac records the arm's rate. return dict( g1=np.nan, g2=np.nan, R11=np.nan, R12=np.nan, R21=np.nan, R22=np.nan, s2n=np.nan, T=np.nan, T_psf_orig=np.nan, @@ -213,10 +200,10 @@ def response(d, i, j): def _boot_idx(n, B, rng): """One seed-index array per bootstrap replicate. - Every quantity inside a replicate must be computed from the SAME resampled - seed set: resampling each arm independently silently destroys the ±γ - pairing and inflates the reported error to the unpaired level (the bug this - replaces read paired σ_m = 0.23 where the per-seed pairs give 0.009).""" + @sc [label:coupling] breakdown-grid-joint-bootstrap + Within each estimator, compute both arms and their response denominator + from the same resampled seed indices. Independent arm resampling destroys the ±γ pairing and inflates uncertainty: + measured σ_m is 0.23 without pairing versus 0.009 with joint resampling.""" return rng.integers(0, n, size=(B, n)) @@ -309,7 +296,7 @@ def pair_mask(a, b): summary["alpha"] = None if summary["c1"] is None else float(summary["c1"] / PSF_SHEAR) summary["alpha_err"] = None if summary["c1_err"] is None else float(summary["c1_err"] / PSF_SHEAR) - # --- legacy ratio-of-averages (v1 continuity): m = //gamma - 1 --- + # --- single-arm ratio-of-averages: m = //gamma - 1 --- for axis, (ap, ri, comp) in {1: ("g1p", "R11", "g1"), 2: ("g2p", "R22", "g2")}.items(): m = ok(ap) g, R = arr[ap][comp][m], arr[ap][ri][m] @@ -375,7 +362,7 @@ def main(): n_seeds = a.n_seeds or N_SEEDS_DEFAULT res_grid = np.geomspace(a.res_lo, a.res_hi, a.n_res) res_idx = a.res_index if a.res_index is not None else list(range(a.n_res)) - # cell_index enumerates the FULL 8x3 grid in (res, band) order so seeds are + # cell_index enumerates the full n_res x 3 grid in (res, band) order so seeds are # stable regardless of which subset a given run selects. all_cells = [(ri, bi) for ri in range(a.n_res) for bi, _ in enumerate(NOISE)] cell_of = {(ri, bi): k for k, (ri, bi) in enumerate(all_cells)} @@ -428,8 +415,8 @@ def main(): ), cells=cells, ) - # v2 JSON is primary; mirror to breakdown_grid.json so the status-page reader - # (build_status.py loads that exact name) stays green with no code change. + # breakdown_grid_v2.json is primary; build_status.py reads the mirror + # named breakdown_grid.json. for fname in ("breakdown_grid_v2.json", "breakdown_grid.json"): with open(os.path.join(a.output, fname), "w") as fh: json.dump(out, fh, indent=2) diff --git a/src/shapepipe/modules/fake_interp_runner.py b/src/shapepipe/modules/fake_interp_runner.py index bc167f924..0186cbbcb 100644 --- a/src/shapepipe/modules/fake_interp_runner.py +++ b/src/shapepipe/modules/fake_interp_runner.py @@ -7,8 +7,7 @@ (image simulations, true PSF) this runner stands where ``psfex_interp_runner`` and ``mccd_interp_runner`` stand for the real data. It writes the same ``galaxy_psf`` SqliteDict, taken from the simulation's PSF dictionary instead -of a fitted model. ``fake_psf_runner`` is the same module under its original -name, used by the legacy bash job scripts. +of a fitted model. Bash job scripts use the equivalent ``fake_psf_runner``. :Author: Martin Kilbinger diff --git a/src/shapepipe/modules/get_images_runner.py b/src/shapepipe/modules/get_images_runner.py index 28e81b7d5..024f989e9 100644 --- a/src/shapepipe/modules/get_images_runner.py +++ b/src/shapepipe/modules/get_images_runner.py @@ -27,10 +27,8 @@ def get_images_runner( """Define The Get Images Runner.""" # Read config file section - # Copy/download method. getexpanded, not get: the workflow's ini sets - # RETRIEVE = $SP_RETRIEVE (a run-config key since d64f88ed), and plain get - # returns the literal "$SP_RETRIEVE", which then fails the check below. - # RETRIEVE_OPTIONS a few lines down already reads the same way. + # Expand environment variables such as RETRIEVE = $SP_RETRIEVE before + # validating the copy/download method. retrieve_method = config.getexpanded(module_config_sec, "RETRIEVE") retrieve_ok = ["vos", "symlink"] if retrieve_method not in retrieve_ok: diff --git a/src/shapepipe/modules/make_cat_package/make_cat.py b/src/shapepipe/modules/make_cat_package/make_cat.py index e6640d006..2c3749b43 100644 --- a/src/shapepipe/modules/make_cat_package/make_cat.py +++ b/src/shapepipe/modules/make_cat_package/make_cat.py @@ -315,17 +315,15 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): plus four OBJECT/SHEAR-less per-object metadata columns (``NGMIX[m]_MCAL_FLAGS``, ``NGMIX_N_EPOCH``, ``NGMIX_MCAL_TYPES_FAIL``, ``NGMIX_NEIGHBOUR_FLAG`` — the last a blend - flag set when the coadd seg stamp held a non-central footprint, - shapepipe#776). The galaxy is the implicit default object - and carries NO ``OBJECT`` token (``NGMIX_G1_NOSHEAR``, dropping the - ``GAL`` segment carried by the pre-#761 names). The explicit PSF + flag set when the coadd segmentation stamp contains a non-central + footprint). The galaxy is the implicit default object and carries no + ``OBJECT`` token (``NGMIX_G1_NOSHEAR``). The explicit PSF objects are ``PSF_ORIG`` (the original image PSF, fit by :func:`ngmix.average_original_psf`) and ``PSF_RECONV`` (the metacal reconvolution kernel, fit by :func:`ngmix.average_multiepoch_psf`); - see those functions for what each PSF family IS. ``PSF_ORIG`` and - ``PSF_RECONV`` are independent fits of different PSFs, no longer the - single aliased value of the pre-fix code (shapepipe#749). + see those functions for the definitions of each PSF family. The two + families contain independent fits of different PSFs. Parameters ---------- @@ -366,8 +364,7 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): self._w_log.warning(warning) ngmix_mcal_types_fail = ngmix_cat_file.get_data()["mcal_types_fail"] - # Per-object blend flag (shapepipe#776): the seg stamp held a - # non-central footprint. Galaxy-only (non-moments), like N_EPOCH. + # Per-object blend flag: galaxy-only (non-moments), like N_EPOCH. ngmix_neighbour_flag = ngmix_cat_file.get_data()["neighbour_flag"] # Needed in both moments and non-moments modes (used unconditionally # below), so read them outside the branch. diff --git a/src/shapepipe/modules/mask_query_package/__init__.py b/src/shapepipe/modules/mask_query_package/__init__.py index fed9dcd92..30d2a53dd 100644 --- a/src/shapepipe/modules/mask_query_package/__init__.py +++ b/src/shapepipe/modules/mask_query_package/__init__.py @@ -2,7 +2,7 @@ This package contains the module for ``mask_query``. -:Author: Claude Fable 5, for PR #847 +:Author: Claude Fable 5 :Parent module: ``sextractor_runner`` diff --git a/src/shapepipe/modules/mask_query_package/mask_query.py b/src/shapepipe/modules/mask_query_package/mask_query.py index aff212b5a..1222ee5fd 100644 --- a/src/shapepipe/modules/mask_query_package/mask_query.py +++ b/src/shapepipe/modules/mask_query_package/mask_query.py @@ -2,7 +2,7 @@ Class to flag SExtractor detections against external healsparse masks. -:Author: Claude Fable 5, for PR #847 +:Author: Claude Fable 5 """ @@ -91,12 +91,9 @@ def process(self): ori_cat.open() data = ori_cat.get_data() - # A CCD SExtractor found nothing on is tolerated all along this chain - # (setools' ~0.2% attrition, psfex_interp's floor=0 warn), so it must - # not be an error here either. An empty LDAC table has no columns to - # index, so read the positions only when there are rows, and still - # publish an output file — a missing sexcat_ext would look to the file - # handler like a crash rather than like an empty CCD. + # Empty CCD catalogues still need an output file so the file handler + # can distinguish an empty detection result from a failed module. + # Read position columns only when the table has rows. if len(data) == 0: ra = np.zeros(0) dec = np.zeros(0) diff --git a/src/shapepipe/modules/mask_query_runner.py b/src/shapepipe/modules/mask_query_runner.py index f4bc97941..5f2290393 100644 --- a/src/shapepipe/modules/mask_query_runner.py +++ b/src/shapepipe/modules/mask_query_runner.py @@ -2,7 +2,7 @@ Module runner for ``mask_query``. -:Author: Claude Fable 5, for PR #847 +:Author: Claude Fable 5 """ diff --git a/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py b/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py index 170e746b9..b4cd9ad2d 100644 --- a/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py +++ b/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py @@ -462,8 +462,7 @@ def shapepipe_write_output( ): r"""Write ShapePipe Output. - Save interpolated PSFs dictionary to FITS file. The saved files are - compatible with the previous ShapePipe's standard. + Save interpolated PSFs and optional shapes to a SExtractor FITS catalogue. Parameters ---------- diff --git a/src/shapepipe/modules/merge_sep_cats_package/merge_sep_cats.py b/src/shapepipe/modules/merge_sep_cats_package/merge_sep_cats.py index b0be88944..03ef20e99 100644 --- a/src/shapepipe/modules/merge_sep_cats_package/merge_sep_cats.py +++ b/src/shapepipe/modules/merge_sep_cats_package/merge_sep_cats.py @@ -127,8 +127,7 @@ def process(self): cat0.open() list_ext_name = cat0.get_ext_name() - # Inupt ngmix files sometimes have not all sheared versions - # (HDUs 1 - 5 = 1M, 1P, 2M, 2P, NOSHEAR) due to IO errors + # Require all five shear HDUs (1M, 1P, 2M, 2P, NOSHEAR). if len(list_ext_name) < 6: raise IndexError( f"Input ngmix catalogue {input_file} has only" @@ -137,10 +136,8 @@ def process(self): list_col_name = cat0.get_col_names() - # Some older input ngmix catalogues have multiple of 5 HDUs - # if reprocessed and not deleted but appended. - # The following log message should be replaced by raising - # and error in future + # Appending repeated results can leave extra HDUs. Read only the + # primary HDU and the first five shear tables. if len(list_ext_name) > 6: wmsg = f"Cropping input HDUs from {len(list_ext_name)} to 5" self._w_log.info(wmsg) diff --git a/src/shapepipe/modules/merge_starcat_package/merge_starcat.py b/src/shapepipe/modules/merge_starcat_package/merge_starcat.py index 5e579f6dd..36543580c 100644 --- a/src/shapepipe/modules/merge_starcat_package/merge_starcat.py +++ b/src/shapepipe/modules/merge_starcat_package/merge_starcat.py @@ -345,8 +345,8 @@ def process(self): model_var.append(model_var_val) model_var_size.append(model_var_val.size) - # ONE ARRAY PER CATALOGUE PER COLUMN (see _stack): the per-value - # python lists this replaces cost ~10x the input bytes. + # Keep one array per catalogue per column (see _stack). + # Per-value Python lists cost ~10x the input bytes. # positions pos = starcat_j[self._hdu_table].data["GLOB_POSITION_IMG_LIST"] x.append(np.asarray(pos[:, 0])) @@ -510,7 +510,7 @@ def process(self): # Write file output.save_as_fits(data, sex_cat_path=self._input_file_list[0][0]) - # MKDEBUG; Implement this in file_io + # Record the input catalogue type in the output table header. if self._input_cat_type: with fits.open(out_path, mode="update") as hdu_list: header = hdu_list[1].header @@ -574,8 +574,8 @@ def __init__( ("HSM_RHO4_STAR", "HSM_RHO4_STAR"), ("HSM_FLAG_PSF", "HSM_FLAG_PSF"), ("HSM_FLAG_STAR", "HSM_FLAG_STAR"), ) - # Present in psfex_interp output, absent from pix2wcs-converted files - # (MKDEBUG); zero-filled when missing rather than failing the merge. + # Present in psfex_interp output, absent from pix2wcs-converted files; + # zero-filled when missing rather than failing the merge. _OPTIONAL = (("MAG", "MAG"), ("SNR", "SNR"), ("ACCEPTED", "ACCEPTED")) def _ccd_nb(self, path): @@ -587,11 +587,11 @@ def process(self): Process merging. - TWO PASSES, AND NEITHER HOLDS THE CAMPAIGN TWICE. The first reads only - the FITS HEADER of every input — NAXIS2, the row count — and never - touches a data block; the second allocates the output columns once, at - their exact final length, and fills them slice by slice. Peak memory is - therefore ONE output plus ONE input catalogue. + The first pass reads row counts (``NAXIS2``) and column dtypes from + FITS headers without touching data blocks. The second allocates each + output column at its final length and fills it slice by slice. During + the fill pass, retained arrays are the output columns and one input + catalogue; FITS serialization allocates additional copies. @sc [label:schema] psfex-starcat-columns-strict Every ``HSM_*`` column is read by name with no fallback, so the set @@ -599,33 +599,24 @@ def process(self): ``PSFExInterpolator._write_output_validation`` writes (``test_hsm_column_seams``). - What this replaces, in two steps, is instructive about the cost of the - obvious code. Accumulating each column into a python LIST OF VALUES — - ``x += list(data["X"])`` — turned 4 bytes of float32 payload into a - 32-byte object plus an 8-byte pointer, measured at ~10x the input bytes - end to end and putting a full-survey merge (~20k exposures x 40 CCDs) at - ~400 GB. Accumulating one ARRAY PER CATALOGUE and concatenating once - brought that to ~5.5x. This pass structure removes what was left of the - accumulation: there are no chunks, and no concatenate that must hold its - inputs and its result at the same time. - - ``self._input_file_list`` MUST BE ITERABLE TWICE, which the module - runner's list is. A one-shot generator is not, and would silently merge - nothing on the second pass — hence the explicit row-count check below. + Per-value Python lists measure ~10x the input bytes (~400 GB for + ~20k exposures x 40 CCDs); arrays accumulated before concatenation + measure ~5.5x. Two passes avoid retaining both concatenation inputs + and their result. + + @sc [label:operations] starcat-two-pass-input-stability + Inputs must remain unchanged across both passes, and + ``self._input_file_list`` must support repeated iteration. The final + row-count check rejects a mismatch rather than writing unfilled rows. """ self._w_log.info( f"Merging {len(self._input_file_list)} star catalogues" ) # --- pass 1: row counts and dtypes, from headers alone -------------- - # THE OPTIONAL COLUMNS ARE A PER-FILE QUESTION, NOT A PER-MERGE ONE. - # A pix2wcs-converted catalogue has no MAG/SNR/ACCEPTED while an - # ordinary one does, and a merge can be handed both. Deciding from the - # first file alone got it wrong in both directions: converted-first - # zero-filled the real values of every ordinary file behind it, and - # ordinary-first raised KeyError on the first converted one. So the - # dtype comes from ANY file that carries the column, and pass 2 asks - # each file for itself. + # Optional MAG/SNR/ACCEPTED columns can differ between inputs: + # pix2wcs catalogues lack them. Take the dtype from any file carrying + # the column; pass 2 zero-fills only files that lack it. names, dtypes, opt_dtypes, n_total = [], None, {}, 0 for name in self._input_file_list: try: @@ -634,10 +625,9 @@ def process(self): hdu = starcat_j[self._hdu_table] n_rows = hdu.header["NAXIS2"] # ColDefs.dtype describes the table without reading it. - # NOTE: it is the RAW storage dtype and ignores TSCAL/TZERO, - # so a scaled column would be allocated narrower than the - # values .data returns. Latent, not live: no validation_psf - # column is scaled. Read the dtype off .data if one ever is. + # This is the raw storage dtype, ignoring TSCAL/TZERO. + # Validation columns must be unscaled; scaled columns need + # the dtype of .data to preserve their decoded values. cols = hdu.columns.dtype if dtypes is None: dtypes = cols @@ -658,8 +648,8 @@ def process(self): data = {out: np.empty(n_total, dtype=dtypes[col]) for out, col in self._COLUMNS} for out, col in self._OPTIONAL: - # A column no file carries still gets a column, zero-filled, in the - # positional dtype the old code used for it. + # Columns absent from every file use X's dtype and are zero-filled + # in pass 2. data[out] = np.empty(n_total, dtype=opt_dtypes.get(col, dtypes["X"])) # CCD_NB is one string per catalogue, repeated over its rows; its width # is the widest CCD number in the campaign, which pass 1 already knows. @@ -682,7 +672,7 @@ def process(self): for out, col in self._COLUMNS: data[out][sl] = data_j[col] for out, col in self._OPTIONAL: - # THIS file's schema, not the merge's: zero-fill only the files + # Use this file's schema: zero-fill only the files # that actually lack the column. data[out][sl] = data_j[col] if col in have else 0 data["CCD_NB"][sl] = self._ccd_nb(name[0]) @@ -698,8 +688,7 @@ def process(self): f"merge_starcat: pass 1 counted {n_total} rows, pass 2 filled " f"{at} — is the input list iterable more than once?") - # Prepare output FITS catalogue - # MKDEBUG: SEx_cat=True -> False + # Prepare a plain FITS catalogue without copying a SExtractor HDU. out_path = f"{self._output_dir}/full_starcat-0000000.fits" output = file_io.FITSCatalogue( out_path, @@ -707,18 +696,13 @@ def process(self): SEx_catalogue=False, ) - # `data` was built by the two passes above (size stored as T = 2 - # sigma^2); every column is already an array of its final length. - - # Write file - # MKDEBUG for psf conv (pix2WCS) files do not write as SExtractorCat; - # we do not want to copy the first input data content to HDU #1. - # sex_cat_path=self._input_file_list[0][0], + # Write the final-length arrays as a plain FITS table without copying + # an input SExtractor HDU (size stored as T = 2 sigma^2). output.save_as_fits( data, overwrite=True, ) - # MKDEBUG; Implement this in file_io + # Record the input catalogue type in the output table header. if self._input_cat_type: with fits.open(out_path, mode="update") as hdu_list: header = hdu_list[1].header @@ -859,13 +843,8 @@ def process(self): ra.append(np.asarray(data_j["XWIN_WORLD"])) dec.append(np.asarray(data_j["YWIN_WORLD"])) - # PRE-EXISTING BUG, LEFT ALONE DELIBERATELY: these four REBIND the - # accumulators initialised above rather than appending to them, so - # only the LAST input file's ellipticities reach the output while - # every other column carries the whole merge. Setools is not wired - # to any workflow path today; fixing it is its own change with its - # own verification, and doing it silently inside a memory rewrite - # would bury it. + # These assignments retain only this file's ellipticities; + # the other columns accumulate rows from every input. m11, m20, m02 = self.get_moments(data_j) eps1, eps2 = self.get_ellipticity(m11, m20, m02, "epsilon") chi1, chi2 = self.get_ellipticity(m11, m20, m02, "chi") diff --git a/src/shapepipe/modules/psfex_interp_package/psfex_interp.py b/src/shapepipe/modules/psfex_interp_package/psfex_interp.py index 8ebca71a5..5e7877c9c 100644 --- a/src/shapepipe/modules/psfex_interp_package/psfex_interp.py +++ b/src/shapepipe/modules/psfex_interp_package/psfex_interp.py @@ -108,7 +108,7 @@ def _fourth_moments(image, moms, wcs=None): that motivates the sky-coordinate measurement. When ``wcs`` is ``None`` the computation is done consistently in the pixel frame instead (the axes are then the CCD axes and the result is *not* frame-invariant); this path exists - only for the legacy pixel-frame branch. + only for the pixel-frame branch. Parameters ---------- @@ -711,8 +711,7 @@ def _galcat_wcs(self): Read the CCD image WCS from the SExtractor FITS_LDAC galaxy catalogue. The original image header is stored as a card list in the ``LDAC_IMHEAD`` - HDU (HDU 1); this is the same header the (now retired) pixel-to-world - conversion reconstructed to rotate shapes. + HDU (HDU 1). Returns ------- diff --git a/src/shapepipe/modules/setools_package/setools.py b/src/shapepipe/modules/setools_package/setools.py index 7c82c715f..f2a84fe1a 100644 --- a/src/shapepipe/modules/setools_package/setools.py +++ b/src/shapepipe/modules/setools_package/setools.py @@ -666,12 +666,7 @@ def _make_rand_split(self): cat_size = len(np.where(mask)[0]) n_keep = int(np.ceil(cat_size * ratio)) - # Deterministic split, seeded from the unit's file number: the - # train/validation assignment is a pure function of the input - # catalogue, so the PSF star sample (and everything downstream - # of the PSF model) is reproducible run-to-run. An unseeded - # np.random here made the shear catalogue non-reproducible - # upstream of ngmix's own position seeding. + # See the split-seeded-by-file-number contract above. seed = int( re.sub(r"\D", "", self._file_number_string) or 0 ) % (2 ** 32) diff --git a/src/shapepipe/modules/vignetmaker_package/vignetmaker.py b/src/shapepipe/modules/vignetmaker_package/vignetmaker.py index 586dae28e..f0d7d9dd9 100644 --- a/src/shapepipe/modules/vignetmaker_package/vignetmaker.py +++ b/src/shapepipe/modules/vignetmaker_package/vignetmaker.py @@ -25,10 +25,8 @@ def get_stamps(image, positions, rad): The image is zero-padded by ``rad`` on every side and a ``(2 * rad + 1, 2 * rad + 1)`` stamp is sliced around the integer pixel - nearest each position (``numpy.round``). The stamp values are - bit-identical to the ``sf_tools.image.stamp.FetchStamps`` this replaces - for in-bounds positions; unlike ``FetchStamps``, an out-of-bounds centre - raises rather than being silently wrapped modulo the image shape. + nearest each position (``numpy.round``). An out-of-bounds centre raises + rather than wrapping modulo the image shape. Parameters ---------- @@ -309,6 +307,12 @@ def _get_stamp_me(self, image_dirs, image_pattern): Dictionary containing object id and vignets for each epoch @sc [decision:preparation.stamp_positioning_and_padding] + + @sc [label:coupling] vignette-centroid-rounding + Each epoch stores the [row, col] ``OFFSET`` and ``INT_POS`` from + extraction with its vignette. The ngmix "wcs" centroid prior must share + that rounding; ``INT_POS + OFFSET`` reconstructs the absolute exposure + centroid for diagnostics. """ cat = file_io.FITSCatalogue(self._galcat_path, SEx_catalogue=True) cat.open() @@ -445,14 +449,7 @@ def _get_stamp_me(self, image_dirs, image_pattern): output_dict[id_tmp][index]["VIGNET"] = final_list[j][1][ where_res[0] ] - # Sub-pixel [row, col] offset of the coadd centroid from - # the integer pixel this stamp was extracted around, plus - # that integer pixel itself. Both ride with the vignette - # (no sidecar file); OFFSET is consumed by the ngmix "wcs" - # (coadd) centroid source so the extraction and the - # centroid prior share one rounding (see #767), and - # INT_POS + OFFSET reconstruct the absolute exposure - # centroid for diagnostics. + # OFFSET and INT_POS: see vignette-centroid-rounding. output_dict[id_tmp][index]["OFFSET"] = final_list[j][3][ where_res[0] ] diff --git a/src/shapepipe/modules/vignetmaker_runner.py b/src/shapepipe/modules/vignetmaker_runner.py index a31d818ae..503fd7dcc 100644 --- a/src/shapepipe/modules/vignetmaker_runner.py +++ b/src/shapepipe/modules/vignetmaker_runner.py @@ -106,7 +106,7 @@ def vignetmaker_runner( # Multi-epoch exposures if config.has_option(module_config_sec, "ME_IMAGE_EXP_DIR"): - # v2.0: locate runner output dirs via the $SP_EXP tree + # Locate runner output directories in the configured exposure tree. exp_base_dir = config.getexpanded( module_config_sec, "ME_IMAGE_EXP_DIR" ) @@ -128,7 +128,7 @@ def vignetmaker_runner( ) image_dirs.append(dirs) else: - # v1: run-log based lookup via symlinked exposure run dirs + # Locate symlinked exposure output directories via the run log. modules = config.getlist(module_config_sec, "ME_IMAGE_DIR") image_dirs = [] for module in modules: diff --git a/src/shapepipe/pipeline/sqlite_store.py b/src/shapepipe/pipeline/sqlite_store.py index bc1240c9a..0d8fdf361 100644 --- a/src/shapepipe/pipeline/sqlite_store.py +++ b/src/shapepipe/pipeline/sqlite_store.py @@ -11,13 +11,13 @@ file with sqlite's ``immutable=1`` URI parameter instead, which skips all locking and change detection. -``immutable=1`` is only correct for a file that no process writes while it -is open: sqlite neither sees concurrent changes nor replays a hot journal -left by a crashed writer, so it would return uncommitted rows. Every call -site of this module reads a store that an earlier module wrote and closed -(``make_post_process`` reads the header log ``merge_headers`` wrote). As a -guard, opening a store that has a rollback journal or write-ahead log next -to it raises instead of reading it. +@sc [label:custody] immutable-sqlite-store +The writer must close the store before a reader opens it, and no process may +write it while an immutable reader is open. SQLite skips change detection +and journal recovery in this mode, so concurrent writes or an unrecovered +journal can yield invalid data. Opening rejects stores with a rollback +journal or write-ahead log beside them; this guard does not prevent a writer +from starting after the check. """ @@ -52,9 +52,8 @@ class ImmutableSqliteDict(Mapping): FileNotFoundError If ``path`` is not an existing file RuntimeError - If a ``-journal`` or ``-wal`` file sits next to ``path``: a writer is - mid-transaction or crashed in one, and an immutable read would see - uncommitted data + If a ``-journal`` or ``-wal`` file sits next to ``path``; the store + may have an active writer or require recovery before an immutable read """ diff --git a/src/shapepipe/testing/simulate.py b/src/shapepipe/testing/simulate.py index 76933cc80..aaf7bdb26 100644 --- a/src/shapepipe/testing/simulate.py +++ b/src/shapepipe/testing/simulate.py @@ -1,7 +1,7 @@ """Simulation utilities for ShapePipe validation tests. -Stable across shapepipe branches and ngmix versions — contains only galsim -and numpy, no shapepipe processing or ngmix fitting code. +Uses only galsim and numpy, keeping image generation independent of +ShapePipe processing and ngmix fitting code. Author: Axel Guinot """ @@ -59,40 +59,35 @@ def make_data( guardrail needs: the PSF carries a shape that an unbiased deconvolution must remove from a round galaxy. wcs : galsim.BaseWCS, optional - Uniform drawing WCS. Default None means ``galsim.PixelScale - (pixel_scale)`` with the legacy world-space sub-pixel shift draw — - bit-identical rng consumption and values to earlier versions of this - function. A non-trivial jacobian (rotation/shear — the ngmix#72 - sensitivity axis) switches the shift draw to PIXEL space (uniform - ±0.5 pixel per axis), guaranteeing the object lands within half a - pixel of the stamp centre — the contract that round-to-nearest-pixel - centroid logic (e.g. ngmix ``centroid_source="wcs"``) depends on. - Note ``psfs_sigmas`` are HSM pixel-unit sigmas of the *WCS-drawn* - PSF stamp; under a sheared WCS they are not circularly symmetric - measures. + Uniform drawing WCS. With None, use ``galsim.PixelScale(pixel_scale)`` + and draw shifts uniformly in world space within ±pixel_scale/2. + With an explicit WCS, draw shifts uniformly within ±0.5 pixel per + axis and transform them to world space; see the centroid contract below. + ``psfs_sigmas`` are HSM pixel-unit sigmas of the WCS-drawn model PSF + stamp, not circularly symmetric measures under a sheared WCS. return_centers : bool, optional If True, also return per-epoch true object centres as ``galsim.PositionD`` in galsim image coordinates (1-based; stamp - centre is ``(img_size + 1) / 2``). Default False keeps the legacy - 6-tuple return. + centre is ``(img_size + 1) / 2``). Default False returns six items. psf_model_fwhm_ratio : float, optional - Multiplicative size error injected into the PSF *model* stamp handed to - ngmix, relative to the *true* PSF that convolves the galaxy. Default 1.0 - (no error: the same PSF object renders the galaxy and the model stamp, - byte-for-byte identical to the legacy behaviour). A value != 1.0 makes - the model PSF the wrong size — over-sized (>1) over-deconvolves, under- - sized (<1) leaves residual smoothing — so metacal deconvolves by a PSF - that does not match the one in the data. This is the controlled-PSF- - model-error knob: it tests whether metacal's response correction (which - is computed with the same wrong model) absorbs a deconvolution error, - the one shape-measurement systematic the zero-error sim could not probe. + Model PSF FWHM divided by the true PSF FWHM. Default 1.0 adds no + size error. Values above 1 over-deconvolve; values below 1 leave + residual smoothing. This controlled mismatch tests whether metacal's + response correction absorbs an error in its deconvolution model. psf_model_shear : tuple of float or None, optional Extra ellipticity (Δg1, Δg2) added to the PSF *model* stamp on top of ``psf_shear``, relative to the true PSF. Default None (no error). A non-zero value mismodels the PSF shape, injecting a controlled additive (PSF-leakage) error into the recovered shear. When both this is None and ``psf_model_fwhm_ratio`` is 1.0 the model PSF *is* the true PSF object, - so the output is byte-for-byte the legacy result. + so there is no model mismatch before stamp noise is added. + + @sc [label:frame] simulation-subpixel-centres + Shifts stay within half a pixel of the stamp centre on each pixel axis, + as required by round-to-nearest-pixel centroid logic. Both drawing paths + consume random shifts in y-then-x order so seeded simulations are reproducible. + An explicit WCS transforms pixel shifts to world coordinates; drawing a + world-space box under a sheared WCS can exceed the half-pixel bounds. Returns ------- @@ -111,9 +106,7 @@ def make_data( wcs = galsim.PixelScale(scale) def draw_shift(): - # Legacy world-space draw — these exact lines (incl. dy-then-dx - # order) keep rng consumption and values bit-identical to the - # pre-wcs version of this function. + # World-space shifts for PixelScale; see the centroid contract. dy, dx = rng.uniform(low=-scale / 2, high=scale / 2, size=2) return dx, dy, dx / scale, dy / scale @@ -122,10 +115,7 @@ def draw_shift(): transform = np.array([[jac.dudx, jac.dudy], [jac.dvdx, jac.dvdy]]) def draw_shift(): - # Pixel-space draw (same dy-then-dx order): under a sheared - # jacobian a world-space ±scale/2 box maps to a pixel-space - # parallelogram exceeding ±0.5 pixel, which would silently break - # rounded-centroid logic with off-by-one-pixel centres. + # Transform bounded pixel shifts; see the centroid contract. dpy, dpx = rng.uniform(low=-0.5, high=0.5, size=2) dx, dy = transform @ (dpx, dpy) return float(dx), float(dy), dpx, dpy @@ -151,11 +141,7 @@ def draw_shift(): ), ).shift(dx, dy) - # The PSF *model* stamp handed to ngmix. By default it is the very same - # object that convolved the galaxy (zero model error, byte-for-byte). A - # size and/or shape error makes it differ from the true PSF, so metacal - # deconvolves by the wrong PSF — the systematic the zero-error sim could - # not exercise. + # Reuse the true PSF when neither model-error parameter is set. if psf_model_fwhm_ratio == 1.0 and psf_model_shear is None: psf_model = psf else: diff --git a/src/shapepipe/utilities/cfis.py b/src/shapepipe/utilities/cfis.py index 205d44a6f..27cbabd3b 100644 --- a/src/shapepipe/utilities/cfis.py +++ b/src/shapepipe/utilities/cfis.py @@ -601,8 +601,7 @@ def get_tile_number_list(tile_name_list): return nix_list, niy_list -# Survey-wide object ID: ``tile_id * TILE_UNIQUE_ID_BASE + NUMBER``, with -# ``tile_id = RRR * 1000 + DDD``. Assumes fewer than 10**6 objects per tile. +# Radix of the object-ID encoding; see get_tile_unique_id. TILE_UNIQUE_ID_BASE = 10**6 @@ -634,6 +633,12 @@ def get_tile_unique_id(tile_id, number): Return the survey-wide object ID ``tile_id * 10**6 + number``. + @sc [label:schema] tile-object-id-encoding + Tile IDs and object numbers must each be integers in ``[0, 10**6)``. + Object numbers must be unique within a tile: encoding pairs with this + radix then gives distinct survey-wide IDs and an exact inverse through + :func:`split_tile_unique_id`. + Parameters ---------- tile_id : int diff --git a/src/shapepipe/utilities/field_corners_extractor.py b/src/shapepipe/utilities/field_corners_extractor.py index a9b4e6c40..39445170c 100644 --- a/src/shapepipe/utilities/field_corners_extractor.py +++ b/src/shapepipe/utilities/field_corners_extractor.py @@ -297,12 +297,13 @@ def process_single_header(args): def get_done_ccds(self): """Get Done CCDs. - Read the set of CCD IDs already present in the output file. Resume is - keyed on individual CCD IDs, not exposure numbers: a CCD counts as done - only if its own row is present. This keeps resume correct when a write - was interrupted mid-exposure (the missing CCDs are filled in) and when - a rerun uses an expanded ``--ccd_list`` (the newly requested CCDs are - added), and it never duplicates a row. + @sc [label:schema] footprint-resume-ccd-ids + Resume is keyed on individual CCD IDs, not exposure numbers, so missing + CCDs in a partially written exposure and an expanded ``--ccd_list`` + remain eligible. A row's first column marks its CCD as done; the rest + of the row is not validated, so an interrupted row write can make resume + skip an incomplete CCD: remove any incomplete trailing row before + resuming. If reading the IDs fails, return an empty set. Returns ------- @@ -371,9 +372,7 @@ def run(self, args=None): print(f"{n} header files found") - # On resume, read the CCD IDs already written so we can skip them - # per-CCD (not per-exposure): a partially written exposure is completed - # rather than skipped, and no row is ever duplicated. + # Skip IDs returned by get_done_ccds; see its resume contract. done_ccds = set() if resume: done_ccds = self.get_done_ccds() diff --git a/src/shapepipe/utilities/final_cat.py b/src/shapepipe/utilities/final_cat.py index 644236377..b00a8039e 100644 --- a/src/shapepipe/utilities/final_cat.py +++ b/src/shapepipe/utilities/final_cat.py @@ -5,11 +5,10 @@ * HDF5, as ``make_cat.write_final_cat`` writes it: one dataset per column, in column order, a vector column as a 2-D dataset with one row per object; -* FITS, a binary table (``RESULTS``, HDU 1), as catalogues made before the - HDF5 format are. +* FITS, a binary table (``RESULTS``, HDU 1). The format is decided per file from its content (the HDF5 signature), not its -name, so a campaign or a hand-made directory that mixes the two reads as one. +name, so callers can read a directory containing both formats. """ diff --git a/src/shapepipe/utilities/header_downloader.py b/src/shapepipe/utilities/header_downloader.py index 21b9726b2..2b031b6cd 100644 --- a/src/shapepipe/utilities/header_downloader.py +++ b/src/shapepipe/utilities/header_downloader.py @@ -186,6 +186,10 @@ def get_fits_header(self, expnum, client): bool True if successful, False otherwise + @sc [label:custody] header-download-publication + Copy to a ``.part`` file in the destination directory and rename only + after success: resume treats an existing destination as complete. + A caught failure removes the partial file; interruption can leave it. """ vospace_path = self._params["vospace_path"] output_dir = self._params["output_dir"] @@ -213,10 +217,7 @@ def get_fits_header(self, expnum, client): print(f"Could not create symlink from {link_source}: {e}") # Fall through to download if symlink fails - # Download from VOSpace atomically: copy to a temp file in the same - # directory, then rename on success. An interrupted transfer leaves - # only the temp file behind, so resume never treats a partial download - # as complete. + # Publish by rename; see header-download-publication. tmp_dest = f"{dest}.part" try: client.copy(source, tmp_dest, head=True) diff --git a/src/shapepipe/utilities/mask_query.py b/src/shapepipe/utilities/mask_query.py index d0920f27e..3a7e91bd1 100644 --- a/src/shapepipe/utilities/mask_query.py +++ b/src/shapepipe/utilities/mask_query.py @@ -1,46 +1,33 @@ """MASK QUERY. -The one healsparse lookup in ShapePipe. +Shared healsparse lookup at object sky positions. -ShapePipe does not generate or rasterize masks. Sky-fixed masks are supplied -as healsparse maps and are consumed by *querying them at object positions*: -every object gets its mask value(s) as catalogue columns, and rejection happens -at the catalogue level, never at the pixel level. The only mask that still -reaches pixels is the per-exposure instrument flag image delivered with it. - -Two callers share the primitive defined here: - -* ``make_cat`` writes one ``MASK_`` column per configured map, carrying - the map value verbatim (no interpretation, no filtering); -* ``mask_query`` writes a single integer ``MASK_EXT`` column onto the exposure - SExtractor catalogue, combining the configured maps into "clean (0) or - flagged (nonzero)" so that ``setools`` — whose expression language has no - bitwise operators — *could* cut on ``MASK_EXT == 0``. The shipped selection - does not; the column is carried for measurement (see that module). +``query_map`` returns map values verbatim; ``flag_positions`` combines maps +into a clean (0) or flagged (nonzero) integer. Catalogue column definitions +and selection policy belong to the callers in ``modules/make_cat_package`` +and ``modules/mask_query_package``. Partial reads ------------- -Never read a whole map. The UNIONS products are ~550 MB each and ``mask_query`` -runs PER CCD, so a full read would cost ~22 GB of I/O per 40-CCD exposure to +Read only the coverage pixels needed by the catalogue. The UNIONS products +are ~550 MB each and the mask-query runner operates per CCD, so a full read +would cost ~22 GB of I/O per 40-CCD exposure to answer questions about a 0.06 deg² footprint. Instead the coverage index is read first (``HealSparseCoverage``, a few hundred kB), the coverage pixels the catalogue actually touches are computed with ``hpgeom.angle_to_pixel`` at the map's ``nside_coverage``, and only those get loaded. Every queried position falls inside exactly one coverage pixel and all of them -are requested, so the padding by ``hpgeom.neighbors`` is insurance rather -than necessity — at ``nside_coverage=128`` a coverage pixel of the star map -is ~23 kB, so the padding costs under a megabyte and buys immunity to any -edge convention we did not think of. ``test_partial_read_matches_full`` is -what actually holds the two paths equal. - -Measured on the DR6 star-body map — the bit-2 rung of the ladder the configs -name ``mask_ugriz_nside131072_n4.hsp``, run against the staged single-band copy -``mask_r_nside131072_n4.hsp``, 583 MB, ``nside_coverage=128`` — for 2000 +are requested. Padding with ``hpgeom.neighbors`` guards against edge +conventions; at ``nside_coverage=128`` a coverage pixel of the star map is +~23 kB, so this costs under a megabyte for a CCD-sized query. +``test_partial_read_matches_full`` checks equality with a full-map read. + +Measured on the DR6 star-body map's single-band copy +``mask_r_nside131072_n4.hsp`` (583 MB, ``nside_coverage=128``), for 2000 positions in one CCD-sized box, one process each: partial 0.102 s / 157 MiB -peak RSS, full 12.8 s / 3364 MiB, identical values. Per 40-CCD exposure that -is ~4 s against ~8.5 min of map reading, and the query adds ~0.15 GB to a rule -already asking for 16 GB. +peak RSS, full 12.8 s / 3364 MiB, identical values. Extrapolated to a 40-CCD +exposure, map reading takes ~4 s rather than ~8.5 min. Coverage -------- @@ -49,22 +36,16 @@ ``-1`` for integer maps. ``make_cat`` passes that sentinel through verbatim, which is the documented off-map flag for the final catalogue. -Coverage is reported from the COVERAGE MASK, not ``valid_mask=True``. For a +@sc [label:schema] mask-coverage-index +Coverage is reported from the coverage mask, not ``valid_mask=True``. For a boolean map — which is what the UNIONS per-bit products are — healsparse stores only the ``True`` pixels, so ``valid_mask`` returns the value itself and cannot tell "inside the footprint and clean" from "outside it entirely". -The coverage mask can, at ``nside_coverage`` resolution, which is the scale the -question is asked at anyway: does this map reach this exposure at all? +The coverage mask distinguishes them at ``nside_coverage`` resolution. -``flag_positions`` treats off-coverage as **not flagged**, for both map kinds. -This makes the integer case agree with the boolean case (whose sentinel is -literally ``False``) rather than diverge from it, and it keeps a map whose -coverage does not reach an exposure from silently rejecting every star on it. -That is also why the all-off-coverage case is logged as a WARNING: it is -indistinguishable from "nothing is masked here" in the output column, so it has -to be distinguishable in the log. +See ``flag_positions`` for the off-coverage flag convention and warning. -:Author: Claude Fable 5, for PR #847 +:Author: Claude Fable 5 """ @@ -235,9 +216,6 @@ def flag_positions(paths, ra, dec, bits=None, w_log=None): (``value & bits``); ``0`` where the value is zero or the position is outside coverage. - Contributions are combined with a bitwise OR, so the returned flag is zero - for a clean object and carries the union of the bits that fired otherwise. - Parameters ---------- paths : list @@ -272,8 +250,7 @@ def flag_positions(paths, ra, dec, bits=None, w_log=None): contribution = values.astype(np.int64) else: integer = values.astype(np.int64) - # Off-coverage: the sentinel, negative by healsparse convention. - # Zeroed rather than OR-ed in, see this module's docstring. + # Exclude negative sentinels from the OR; see the flag contract. integer = np.where(integer < 0, 0, integer) if bits is not None: integer &= bits diff --git a/workflow/CONTRACTS b/workflow/CONTRACTS index fe2e16c36..4a5b51873 100644 --- a/workflow/CONTRACTS +++ b/workflow/CONTRACTS @@ -22,7 +22,7 @@ tests/workflow/test_dag.py::test_products_use_products_dir_and_run_name. Each input type's `config/*/final_cat.param` is the merged catalogue's exact column list, in order: `final_cat_merge` writes those columns and no others, and the reader raises on a listed column a tile lacks. So a name belongs there -only if `make_cat` writes it on EVERY tile. A per-epoch family (`EXP_ID_n`, +only if `make_cat` writes it on every tile. A per-epoch family (`EXP_ID_n`, `CCD_n`, `HSM_*_PSF_n`) qualifies only with a fixed slot count, since `make_cat` otherwise sizes it from the sexcat's maximum geometric `N_EPOCH`. Enforced by tests/unit/test_final_cat_merge_invariants.py (the merge) and @@ -44,8 +44,8 @@ Regenerate tests/workflow/params_pin.json only at that boundary with @sc [label:custody] clean-exposure-waits-on-persist-iff-psf `clean_exposure` waits on the exposure's persisted PSF products exactly when -`PERSISTS_PSF` (`psf_model != "fake"`, which is `psfex`: `mccd` is refused at -parse time by `refuse_unpersistable_psf`): on the `exp_persist` manifest for a +`PERSISTS_PSF` is true (`psf_model != "fake"`; `mccd` is refused at parse time +by `refuse_unpersistable_psf`): on the `exp_persist` manifest for a live store, and on the tar (a leaf), or nothing, for a reclaimed one. Dropping the edge under psfex lets reclamation delete the scratch store before its PSF products reach `products_dir`; keeping it under `fake` makes every clean wait on diff --git a/workflow/README.md b/workflow/README.md index a415e1b55..0b6baab59 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -1,10 +1,8 @@ # ShapePipe Snakemake orchestration -Snakemake workflow that orchestrates real-data ShapePipe runs. It replaces the -`curl_canfar_local.sh → run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash` -bash layers and the per-site sbatch reimplementations. **Module code is -untouched**: rules call `shapepipe_run -c ` on the existing config -chains. Design and rationale: +Snakemake workflow that orchestrates ShapePipe data and image-simulation runs. +Rules call `shapepipe_run -c ` on the module configuration chains. +Design and rationale: [CosmoStat/shapepipe#848](https://github.com/CosmoStat/shapepipe/issues/848) (the design document). @@ -26,8 +24,7 @@ uv pip install 'snakemake>=9,<10' 'snakemake-executor-plugin-slurm>=2.7,<3' # the campaign's name; workflow/config.yaml's input_types: and machines: tables supply the rest. # `psf_model` is `psfex` for data (`fake` for image sims). `mccd` is refused -# until `persist_exp.py` and `merge_star_cat.py` read MCCD products. PSFEx is -# exercised by smk-g4 through smk-g6. +# until `persist_exp.py` and `merge_star_cat.py` read MCCD products. # Both `tile_detection` values run SExtractor on the tile image. Under # `unions_catalogue` (the input_types default for data) the UNIONS per-tile # catalogue at `inputs.catalogues` is fetched and the detections are joined @@ -44,11 +41,9 @@ workflow/bin/sp cancel # scancel this workflow's jobs workflow/bin/sp container status # which image the jobs will run ``` -Installed and pinned versions on nibi (`/project/def-mjhudson/cdaley/snakemake-env`, -queried 2026-07-30): `snakemake==9.23.1`, `snakemake-executor-plugin-slurm==2.7.1`. -Pin range: `snakemake>=9,<10`, `snakemake-executor-plugin-slurm>=2.7,<3`. The -v8→v9 breaks matter here: `--use-singularity` became `--sdm`, executors became -plugins, and full `rerun-triggers` became the default. +Supported ranges: `snakemake>=9,<10`, `snakemake-executor-plugin-slurm>=2.7,<3`. +The profiles use Snakemake 9's `--sdm` software deployment and executor plugins; +see `profiles/` for the rerun triggers. Anything other than `run`, `report`, `container`, `cancel` passes straight through to snakemake with the workflow's profile and state dir — the direct command path for @@ -141,9 +136,7 @@ only exists if you ask for one: The Snakefile's `container:` is that resolution, in one order shared by the CLI and the workflow: **sandbox → cached SIF → the `container:` path in -`config.yaml`**. With an empty cache — the normal case — that lands on the -shared `/project` `.sif` the workflow has always used, so this changes nothing -until you opt in. +`config.yaml`**. With an empty cache, resolution falls back to the configured shared image. ```bash sp container status # layers present, active one, revision vs HEAD @@ -174,9 +167,9 @@ therefore two snakemake invocations over one Snakefile: 1. **PREPARE** — `snakemake prepare_all_tiles`: per-tile static DAG (`Git_vos → Uz → Fe`), `keep-going` so tile failures are independent. A - nonzero exit here is not fatal to the run — tiles that lost their - exposure list are dropped at the compute parse — but it is a warning: - `SP_MISSING_THRESHOLD` (default 0.0) is the real gate. + nonzero exit still allows COMPUTE to run, but makes `sp run` fail. + Tiles without exposure lists are dropped at the compute parse, subject to + `SP_MISSING_THRESHOLD` (default 0.0). 2. **COMPUTE** — `snakemake all`: this invocation's *parse* builds the tile↔exposure index (`build_index.py`, imported at parse time, not a DAG node) and runs the full tile/exposure compute chain. The index @@ -208,11 +201,11 @@ the other verbs run out of the existing snapshot, `sp container` excepted (it is about the image you are working with now, not about a campaign). The mechanism and its rationale live in one place: `bin/sp`. -## Execution mode: one SLURM job per rule +## Execution mode: SLURM rules and groups -The profile (`profiles/nibi/config.yaml`) sets `executor: slurm`. Every rule -instance becomes its own SLURM job carrying that rule's own attempt-scaled -resources (`cpus_per_task = threads`, `mem_mb`, `runtime`) and runs inside +The profile (`profiles/nibi/config.yaml`) sets `executor: slurm`. Ungrouped rule instances become individual SLURM jobs; grouped rules share a job. +Rules define attempt-scaled resources (`cpus_per_task = threads`, `mem_mb`, +`runtime`) and run inside `apptainer exec` via the profile's software-deployment method — the workflow never calls apptainer directly. Snakemake feeds the queue as jobs finish, so the full campaign's job count never needs to queue at once, and multi-node @@ -226,10 +219,8 @@ attempt-scaled `mem_mb = lambda wc, attempt: ...` OOM retries and the tuned ngmix thread count. Rules own their own resources; the profile only supplies defaults for rules that state nothing. -`group:` labels that fuse short rules (uncompress, merges) into their chunky -neighbours (queue-latency amortization, per the PRD) are **not yet wired**: -they require labels in `workflow/rules/*.smk`, out of scope for this -profile-only pass. +The rule files define the groups and their resource constraints. +See `workflow/rules/prepare.smk`, `exposure.smk`, and `tile.smk` for their composition. ## Layout @@ -245,7 +236,7 @@ workflow/ scripts/ build_index.py prepare-phase run_index.sqlite builder (plain script) build_forest.py per-tile exposure symlink forest (group-compatible shell) - completeness.py the ported count table (shared by sp_rule + run_report) + completeness.py count table shared by sp_rule + run_report run_report.py standalone report (NOT a DAG node; run_report hooks call it) container.py image layers + the resolution order behind `sp container` (stdlib-only) persist_exp.py ONE exposure's keepable PSF products -> one tar on products_dir (the exp_persist rule) @@ -258,7 +249,7 @@ workflow/ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; keep-going ``` -## How it works +## Products and lifecycle - **The atom is one rule == one `shapepipe_run` on one unit.** Its single declared output is that unit's manifest @@ -279,57 +270,28 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee - **Completeness is an exact-count check, not a taxonomy.** After a run, `sp_rule.py` counts products per mandatory runner against `completeness.py`'s expected count and exits nonzero below it. A shortfall - below `expect` fails unless `warn` is set. No 3-class taxonomy, no - error-signature whitelist. `--keep-going` isolates a failure to its own + below `expect` fails unless `warn` is set. Failure signatures do not exempt shortfalls. `--keep-going` isolates a failure to its own DAG cone. - **Stores are sharded.** Every tile/exposure runs its own `shapepipe_run` in `tiles/<2-char prefix>//` or `exp///`. Configs are committed under `workflow/config/cfis/` and version with the rules that set the env vars they interpolate — there is no `config_src` knob, and no per-unit config symlink; `$SP_CONFIG` points straight at the committed directory. -- **There is no masking stage, on either side.** ShapePipe generates no masks - (PR #847). The one mask that reaches pixels is the instrument flag image - delivered with each exposure, which `exp_split` splits per CCD beside image - and weight and SExtractor reads as `IMAFLAGS_ISO`. Everything else — star - halos, manual masks, per-band coverage, MaxiMask — is supplied as sky-fixed - healsparse maps and QUERIED once per object: the `mask_query` module writes a - `MASK_EXT` column onto each CCD's detection catalogue (inside `exp_psf`), and - `make_cat` writes one `MASK_` column per band onto the final catalogue - (inside `tile_make_cat`). Neither is cut on in the pipeline: instrument flags - mark corrupted measurements and are the only masks that reject anything here, - while the healsparse masks are location flags and every decision about them is - downstream. On exposures the query ships off — `MASK_PATHS` is commented out, - making `mask_query` a no-op pass-through — so turning it on is a config edit, - not a chain edit. Map paths are config, not - code, so regenerated products cost a config edit. Nothing is fetched from a - catalogue server, staged, or rasterized, which is why the old - `star_catalogue` / `exp_star_cat` / `exp_mask` rules and their cache root are - gone. -- **Two HealSparse maps come out of the exposures.** At the mask ladder's - resolution (nside 131072), `nexp_.hsp` counts per sky pixel the - exposures whose CCD with a valid PSF model covers it, and - `nflagged_.hsp` counts the flagged CCD pixels (bad columns, saturated - pixels, bleed trails, cosmics) of those exposures that fall in it, ~74 CCD - pixels filling one sky pixel. The flag image is the one mask that otherwise - never leaves the pixel domain, so a footprint built from CCD corners could not - subtract it (#878). `exp_maps` writes one fragment per exposure beside its PSF - tar, from the split's `DATASEC` (the overscan border is neither coverage nor - defect) and the CCDs `exp_persist` packed a PSF for; `exposure_maps` sums the - campaign's fragments. Nothing in the workflow reads either map. `nexp` is - what randoms need to match an `N_EPOCH` cut; `nflagged` is a diagnostic of - where the detector is bad, not a cut (it does not count exposures lost to - the defect veto). -- **External masks are wired, on the tile side only (data runs).** `inputs.masks` - is a third input root beside tiles and exposures, set per machine in the - `machines:` table, exported as `$SP_INPUT_MASKS` and - pointing at the UNIONS DR6 ugriz bit ladder: one boolean healsparse map per - bit, nside 131072, `True` = masked. `config_tile_Mc.ini` names all 11 of them - in `MASK_EXT_PATHS` under a `_` label, so `make_cat` - writes `MASK_1_Faint_star_halos` … `MASK_2048_z2` and `final_cat.param` - carries the matching 11 names; the config holds the label table. - `MASK_2048_z2` is True where there is *no* Pan-STARRS z data, so an OR over - every column masks everything. ShapePipe cuts on none of them; the mask - choice is sp_validation's. +- **ShapePipe generates no masks.** `exp_split` splits the delivered instrument + flag image per CCD; SExtractor reads it as `IMAFLAGS_ISO`. + External sky masks are queried per object, not rasterized. + Exposure queries are disabled by the commented-out `MASK_PATHS` in + `config_exp_psfex.ini`; see `src/shapepipe/modules/mask_query_package/` + for query semantics and `star_selection.setools` for star-mask selection. +- **Exposure maps describe coverage and detector defects.** `exposure_maps` + merges the `exp_maps` fragments into `nexp_.hsp` (valid-PSF exposure + coverage) and `nflagged_.hsp` (flagged detector-pixel counts). + Neither map is a pipeline cut. See `workflow/scripts/exp_maps.py` and + `merge_exposure_maps.py` for resolution, footprint, and counting semantics. +- **Data tiles carry external mask flags, without cuts.** `inputs.masks` is + exported as `$SP_INPUT_MASKS`. The label table and map semantics live in + `config/cfis/config_tile_Mc.ini`; `final_cat.param` lists the output columns. + Mask selection belongs to downstream analysis. - **The index is parse-time data, never a rule input.** Appending tiles changes which jobs exist without invalidating completed work. - **Exposure products are not `temp()`.** Exposures overlap tiles, so @@ -354,65 +316,25 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee trigger reads that cut as a reason to rerun the very tiles it protects. Know the consequence — `--forcerun` on a tile whose `final_cat` exists will not rebuild its reclaimed exposures. Delete the `final_cat` first. -- **PSF products leave scratch before the purge does.** `exp_persist` packs - the products named by `persist_exp:` in `config.yaml` from the exposure's - scratch store into ONE uncompressed tar, - `/exp///psf/.tar` (inodes, not bytes, bind - on /project), and writes ONE manifest beside it recording the patterns, the - members and their sizes. The - threat it answers is the /scratch purge, not `clean_exposure` — the store goes - in 60 days whether or not the workflow reclaimed it — so it runs even with - `clean: false`, requested directly by `rule all`. `clean_exposure` takes its - manifest as an input, so reclamation can never overtake the copy. It is a - rule of its own rather than a `cp` on the end of `exp_psf` because the keep - list rides on `params`: adding a pattern reruns seconds of packing, not four - hours of PSF fitting per exposure. A pattern that matches nothing is a - recorded warning (setools rejects sparse CCDs); matching nothing at all is a - failure. A `localrule`, by the same arithmetic as `clean_exposure`. -- **The star catalogue's inputs are always kept; `persist_exp:` is what you - keep on top.** `exp_persist` packs `psf_validation` — the psfex_interp - validation catalogue, one per CCD — for every exposure whatever the config - says, because `star_cat_merge` stacks exactly those into the campaign's - `full_starcat`. They are that catalogue's provenance, and they are what keeps - appending a tile next month cheap rather than a rebuild from VOS. About 2 MB - per exposure: ~40 GB and ~40k inodes at DR6 scale, against a ~1 M-inode group - quota. `persist_exp:` is purely additive, and an empty list is legal — the tar - then holds the merge's inputs and nothing else. -- **The keep list names products, not globs.** Entries are names from a - catalogue in `workflow/scripts/persist_exp.py`, which is the single source of - truth for what each one means and what keeping it buys - ([#844](https://github.com/CosmoStat/shapepipe/issues/844)); `config.yaml`'s - block is that catalogue rendered, and `persist_exp.py --list-products` prints - it. Sizes are per exposure, 40 CCDs, measured on smk-m2. - - | product | glob | per exposure | what it buys | - |---|---|---|---| - | `psf_model` | `*.psf` | 2.8 MB | re-interpolate the PSF anywhere later, no rebuild | - | `psfex_cat` | `psfex_cat-*.cat` | unmeasured | which stars PSFEx's outlier rejection clipped | - | `star_selection` | `star_selection-*.fits` | 24.5 MB | which stars the selection cuts rejected, and why | - | `star_train` | `star_split_ratio_80-*.fits` | 19.9 MB | the 80% sample PSFEx fitted | - | `star_test` | `star_split_ratio_20-*.fits` | 7.1 MB | the 20% sample `psf_validation` corresponds to | - | `star_stats` | `star_stat-*.txt` | unmeasured | setools' per-CCD counts, density and FWHM cuts | - - The default is `psf_model`. `psf_validation` is in the catalogue too but needs - no naming; naming it anyway is harmless. **Retention is additive**: an - existing tar is a floor, so shrinking the list adds nothing and removes - nothing. Dropping a product is a deliberate act on `products_dir`, not a - config edit — otherwise editing a config would delete products from the - backed-up filesystem whose scratch originals are long gone. A raw glob is still accepted as an - escape hatch — anything with a glob metacharacter or a dot is read as one — - and an unknown *name* is a parse-time error listing the valid ones. The list - is exposure-side only; tile-side retention is #844 follow-up. +- **PSF products are persisted independently of scratch cleanup.** `exp_persist` + writes `/exp///psf/.tar` and a manifest, + even with `clean: false`. See `workflow/scripts/persist_exp.py` for retention + guarantees, measured storage costs, and packing details; see + `workflow/CONTRACTS` for cleanup ordering. +- **Configure optional retention with `persist_exp:`.** Product names and their + meanings live in `workflow/scripts/persist_exp.py`; `persist_exp.py --list-products` + prints them. The default is `psf_model`, and `psf_validation` is mandatory + regardless of the list. Retention is additive: shrinking the list does not + remove saved products. Remove products explicitly from `products_dir`. - **The campaign ends in two merged catalogues, and the workflow makes both.** Everything above is per unit; the two products downstream analysis actually - opens are per *campaign*, and until these rules existed each was a manual pass - after the run. + opens are per *campaign*. `star_cat_merge` collects every exposure's every CCD's `psf_validation` into `/full_starcat_.hdf5`, one dataset per exposure at `exposures/` — the rho/tau statistics input. It reads the members straight out of the per-exposure tars (`tarfile`; unpacking ~800k files to merge them would defeat the tar's whole purpose), keeps their native dtypes, - and stores `CCD_NB` as an int. sp_validation still opens the old flat FITS + and stores `CCD_NB` as an int. sp_validation opens the flat FITS name, `full_starcat-0000000.fits`; its readers move to this file under [sp_validation#340](https://github.com/CosmoStat/sp_validation/issues/340), the same migration that retires the `patches/` key on the tile side. The rule @@ -420,19 +342,17 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee **Two writers, one schema.** The module runner still emits the flat FITS table through `MergeStarCatPSFEX`, and this rule emits the hdf5; they are separate implementations on purpose, because only one of them reads tars, - keeps native dtypes and reconciles. Their 22 COLUMN NAMES must not drift - apart, and nothing else would notice if they did — a column added to one - writer would just be missing from the other's product. `tests/unit/` - `test_star_cat_columns.py` is what holds them together. + keeps native dtypes and reconciles. Their column names must agree; + `tests/unit/test_star_cat_columns.py` enforces that coupling. `final_cat_merge` collects every ready tile's `final_cat-.hdf5` into `/final_cat_.hdf5`: one dataset per tile under a group - named for the campaign, the `final_cat.param` columns, an `n_tiles` attribute. + named `patches/`, the `final_cat.param` columns, an `n_tiles` attribute. That schema is what sp_validation's reader opens, so it is fixed; the column extraction reuses `scripts/python/create_final_cat.py` while the file is written here, because that script's own discovery walks a directory layout this workflow does not have. The run config's `run:` names both files and the group. - BOTH RECONCILE, through one shared module (`hdf5_reconcile.py`) so the + Both reconcile through one shared module (`hdf5_reconcile.py`) so the campaign's two products cannot disagree about what an output owes its inputs. Each adds the units that have no dataset, drops datasets whose unit left the campaign, re-reads one whose source changed (every dataset records its @@ -457,7 +377,7 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee absorbed manifests out of `cleaned.json`, so a reclaimed exposure keeps its per-runner counts and blocks no tile. The logs go with the manifests — a log claiming `complete` for a store that is gone would contradict the unbuilt - chain the DAG must now see, and its content duplicates the manifest anyway. + chain the DAG must see, and its content duplicates the manifest anyway. The `exp_psf` benchmark tsv lives beside both dirs, not inside either, so reclamation does not eat the memory-sizing data. - **Failure is a report, not a gate.** `run_report.py` disk-scans the trees diff --git a/workflow/Snakefile b/workflow/Snakefile index 20780e1be..1037a79d9 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -2,33 +2,31 @@ Design / rationale: CosmoStat/shapepipe#848, D1-D5. -`sp run` is TWO snakemake invocations over this one Snakefile: +`sp run` uses two Snakemake invocations over this Snakefile: SP_PHASE=prepare snakemake prepare_all_tiles # Git -> Uz -> Fe, per tile SP_PHASE=compute snakemake all # everything else They are two because the exposure job set is *data-derived*: it comes from the tiles' find_exposures output, and a Snakemake DAG is fixed at parse time. The -join between them is the tile<->exposure index, built HERE at parse time of -invocation 2 (build_index.build(), imported — there is no `sp index` verb) and -loaded into plain dicts. The index is never a rule input, so appending tiles and -rebuilding it changes which jobs exist without invalidating completed work; it -ACCUMULATES across invocations, so a later clean_exposure (S5) sees every -consuming tile of the whole campaign, not just this tile list. - -Neither invocation reads the checkout: `sp run` snapshots the code into the -campaign's state dir and points snakemake at THAT tree, so `workflow.basedir` -below is the snapshot and mid-campaign edits are inert. The mechanism and its -rationale live in bin/sp. - -The atom (D2): one rule == one `shapepipe_run` on one unit; its single declared -output is its MANIFEST (`/manifests/.json`), written by -`completeness.py check`. Product files are not declared — a missing CCD is often -legitimate, and at DR6 scale per-CCD declaration means millions of paths. - -Failure evidence rides on the rule's `log:` (`/logs/.json`): -the manifest says "this stage succeeded", the log says "here is what happened" -(the contract is argued in completeness.py's docstring). +join between them is the tile<->exposure index, built by build_index.build() +at parse time of the compute invocation. The head process loads it into dicts; +job parses query only the edges they need. The index is not a rule input, so +rebuilding it changes the job set without itself invalidating completed work. +It accumulates across invocations so clean_exposure sees every consuming tile +in the campaign, not just the current tile list. + +`sp run` points Snakemake at a campaign code snapshot, so `workflow.basedir` +below is that snapshot. See bin/sp for snapshot creation and reuse. + +ShapePipe stage rules run one `shapepipe_run` on one unit (D2). Most declare a +success manifest (`/manifests/.json`) written by +`completeness.py check`, rather than per-CCD products: missing CCDs can be +legitimate, and per-CCD declarations would mean millions of paths at DR6 scale. +See rules/tile.smk for temporary chunk outputs and final-catalogue publication. + +Stage logs (`/logs/.json`) retain failure evidence separately +from success manifests; see scripts/completeness.py for the verdict contract. """ import fnmatch @@ -41,18 +39,12 @@ import sys from pathlib import Path from snakemake.exceptions import WorkflowError -# Explicit rather than relying on the name snakemake injects into this -# namespace: the one place we log at parse time is a branch that only a -# non-default keep list reaches, and a NameError there would be found by a user. +# Import the logger explicitly for parse-time warnings. from snakemake.logging import logger -# Resolved relative to THIS file, not the working directory: snakemake runs with -# --directory on /scratch (bin/sp) so .snakemake/ state never lands on /project. -# -# config.yaml is always loaded; SP_RUN_CONFIG (bin/sp) is merged on top, then -# the `input_types:` entry for input_type and the `machines:` entry for -# (machine, input_type) fill whatever is still unset, the machine entry winning -# (scripts/run_config.py, shared with bin/sp and container.py). +# Resolve paths relative to this file; bin/sp sets a separate working directory. +# Load config.yaml, then overlay SP_RUN_CONFIG. See scripts/run_config.py for +# the defaults shared with bin/sp and container.py. sys.path.insert(0, str(Path(workflow.snakefile).parent / "scripts")) import run_config # noqa: E402 @@ -148,12 +140,9 @@ if INPUT_TYPE == "image_sims": # --- paths ----------------------------------------------------------------- -# RUN_DIR is the scratch root: bulk intermediates, sized so a batch finishes -# inside the purge window. PRODUCTS_DIR is the persistent root -# for the durable, low-volume products — the final catalogues, the index, the -# report. Snakemake's own state is the one durable-looking thing that stays on -# scratch (bin/sp explains why: tens of thousands of small, hot metadata writes -# against a backed-up, file-count-limited filesystem). +# RUN_DIR holds bulk intermediates on scratch; PRODUCTS_DIR holds durable +# products. INDEX_DB has its own configured path. See bin/sp for placement of +# Snakemake's execution state. INPUTS = config["inputs"] OUTPUTS = config["outputs"] RUN_DIR = Path(OUTPUTS["run_dir"]) @@ -162,16 +151,12 @@ RUN_DIR = Path(OUTPUTS["run_dir"]) # false; refuse_one_root_cleaners enforces it. PRODUCTS_DIR = Path(OUTPUTS.get("products_dir") or RUN_DIR) INDEX_DB = Path(OUTPUTS["index_db"]) -# The campaign's NAME is the run config's `run:` — the same name `$run` expands -# to in the paths above. The two campaign-level merges label their output with -# it (`final_cat_.hdf5` and the group inside it holding the per-tile -# datasets, `full_starcat_.hdf5`). REQUIRED in run_config.py, so an unset -# `run:` has already refused above. +# One campaign name for all merged products and their internal groups. +# See workflow/CONTRACTS: campaign-name-is-run. CAMPAIGN = config["run"] SCRIPTS = Path(workflow.basedir) / "scripts" -# The config chain is the repo's committed directory (D2). The configs and -# rules that set their environment variables must be versioned together. There -# is no separate `config_src` setting; input_type picks between the two dirs. +# input_type selects the committed config chain. Version the configs together +# with the rules that supply their environment variables. CONFIG_DIR = Path(workflow.basedir) / "config" / { "data": "cfis", "image_sims": "cfis_image_sims"}[INPUT_TYPE] # `sp run`'s code snapshot (bin/sp), a sibling of workflow.basedir inside @@ -197,17 +182,11 @@ try: except ValueError as err: raise WorkflowError(str(err)) from None -# SP_PHASE is set by bin/sp and NOWHERE else: `prepare`/`compute` on the two -# invocations of `sp run`, `passthrough` on direct commands (`sp --unlock`, `sp -# --dag`, `sp exp_psf ...`). It gates the two parse-time side effects — the index -# build and the report hooks — so that a passthrough parse never mutates durable -# state or dies on a threshold it was not asked about. -# -# Requiring it to be SET is the point of the check below: a bare `snakemake` -# would otherwise parse as a passthrough, find `rule all` gated on an index it -# never built, and exit 0 on an empty DAG. Jobs re-parse this file under the -# slurm executor and inherit the head process's environment, so a submitted job -# always carries the launching phase. +# bin/sp exports prepare/compute for `sp run`, or passthrough for direct +# commands such as `sp --unlock`. Only compute builds the index and installs +# report hooks. Requiring an explicit phase prevents a bare Snakemake invocation +# from succeeding with an empty DAG and an unbuilt index. Submitted jobs inherit +# the launching phase when the SLURM executor re-parses this file. PHASE = os.environ.get("SP_PHASE", "") if not PHASE: raise WorkflowError( @@ -218,12 +197,8 @@ if PHASE not in ("prepare", "compute", "passthrough"): raise WorkflowError( f"SP_PHASE={PHASE!r} is not one of prepare, compute, passthrough.") -# DEDUPED, order preserved. The tile list is appended to by hand across a -# campaign, so a tile can appear twice — harmless for a per-tile target, which -# is the same path requested twice, but not for the campaign-level merges: their -# fingerprint counts what is in this list and the job derives a deduped set from -# the same file (build_index.campaign_tiles), so a duplicate line would make the -# two disagree about a set they must name identically. +# Deduplicate while preserving order, matching build_index.campaign_tiles so +# merge fingerprints and job-time discovery count the same tiles. with open(config["tile_list"]) as f: TILES = list(dict.fromkeys(ln.strip() for ln in f if ln.strip())) @@ -241,21 +216,14 @@ scattergather: NGMIX_CHUNKS = workflow._scatter["ngmix"] # --- parse-time index build + load (D1) ------------------------------------ -# The COMPUTE invocation's parse IS the index build, and it runs UNCONDITIONALLY -# there — no "some Fe output exists" guard. That guard used to make a -# totally-failed prepare produce an empty index, an empty DAG and a green exit 0; -# with it gone, zero Fe outputs means a missing fraction of 1.0, which trips the -# SP_MISSING_THRESHOLD gate. -# -# In every other phase (prepare, or unset for a passthrough invocation) the parse -# builds NOTHING and only loads whatever index is already on disk. +# The compute head process builds the index unconditionally: zero Fe outputs +# gives a missing fraction of 1.0 and trips the configured missing threshold +# unless it allows complete failure. The ready-tile check below also rejects an +# empty compute DAG. Prepare and passthrough only read an existing index. # -# is_main_process matters as much as PHASE: the SLURM executor re-invokes -# snakemake inside every job, that re-invocation parses this file again, and it -# inherits SP_PHASE=compute from the submitting environment. Without the guard, -# every one of the run's jobs re-runs the build — hundreds of concurrent sqlite -# writers on Lustre, which previously caused "database is locked" errors -# (2026-07-31). Job parses only LOAD the index below. +# Submitted jobs inherit SP_PHASE=compute but must not rebuild the index: +# hundreds of concurrent SQLite writers on Lustre fail with "database is locked". +# workflow.is_main_process keeps the build in the head process. if PHASE == "compute" and workflow.is_main_process: build_index.build( TILES, RUN_DIR, INDEX_DB, @@ -264,15 +232,10 @@ if PHASE == "compute" and workflow.is_main_process: # EXP: exposure base-id -> original name (2605805 -> 2605805p; the name goes # verbatim into the fabricated per-unit exp_numbers list so get_images matches # .fits.fz in the store). TILE_EXP: tile -> [exp_ids]. -# EXP_TILES is the inverse edge — the CAMPAIGN-WIDE consumer set clean_exposure -# is keyed on (D5). -# -# SLURPED ONLY IN THE HEAD PROCESS. The executor re-parses this file inside every -# job (~800 per run), and each of those parses needs at most its own unit's edges -# — reading the whole campaign's tile_exposures table there is pure waste that -# grows with the campaign rather than with the job. Job parses get the same three -# lookups backed by memoised single-row queries instead (the table's primary key -# indexes both), so every input function returns exactly what it would have. +# EXP_TILES is the campaign-wide consumer set used by clean_exposure (D5). +# Load the full index only in the head process; job parses use cached queries +# for their own units. The primary key indexes tile lookups; exposure-consumer +# lookups scan the table and are needed only by head-process cleanup. EXP, TILE_EXP, EXP_TILES = {}, {}, {} if INDEX_DB.exists() and workflow.is_main_process: # timeout=60: Lustre lock handoffs are slow; the default 5 s trips on @@ -292,7 +255,7 @@ if workflow.is_main_process: return EXP[exp] def tile_exposures(tile): - """This tile's exposure base-ids, sorted as the index stores them.""" + """This tile's exposure base-ids in table-scan order.""" return TILE_EXP.get(tile, []) def exp_consumers(exp): @@ -316,9 +279,7 @@ else: def tile_exposures(tile): if _JOB_CON is None: return [] - # ORDER BY rowid, not by exp_id: the eager slurp above is a table scan, - # so its lists are in INSERTION order, and the two paths must hand the - # DAG the same list and not merely the same set. + # Match the head process's table-scan order, not exposure-id order. return [r[0] for r in _JOB_CON.execute( "SELECT exp_id FROM tile_exposures WHERE tile_id = ? ORDER BY rowid", (tile,))] @@ -384,47 +345,35 @@ def exp_manifest(exp, stage): return f"{exp_dir(exp)}/manifests/{stage}.json" def prod_exp_dir(exp): - """The exposure's dir on the PERSISTENT root — where exp_persist writes. - - Sharded identically to the scratch one, so the two trees read as the same - campaign seen from two filesystems, exposure side as well as tile side.""" + """The exposure's persistent directory, with the same shard as on scratch.""" return f"{PRODUCTS_DIR}/exp/{exp[:2]}/{exp}" def prod_exp_manifest(exp, stage): - """A manifest that must SURVIVE reclamation, so it is not in the exposure's - scratch manifests/ dir (clean_exposure deletes that wholesale).""" + """A manifest outside the scratch tree deleted by clean_exposure.""" return f"{prod_exp_dir(exp)}/manifests/{stage}.json" def forest_dir(tile): return f"{tile_dir(tile)}/exp_forest" def final_cat(tile): - """The campaign's science product, and its tile-finished marker. - - It lives on the PERSISTENT root, not the scratch one, and both halves of - that sentence are load-bearing. As a product: it is what the campaign is - for, and a 60-day purge must not eat it. As a marker: tile_finished() keys - on this path to cut a finished tile's exposure edges (D5), so if a purge - could remove it, finished tiles would re-declare inputs against exposure - stores that reclamation deleted — the rerun avalanche the cut exists to - prevent, arriving by way of the purge instead. + """The tile's durable science product and tile_finished() marker. + + Keep it outside scratch: losing the marker to a purge would reconnect a + finished tile to reclaimed exposure stores. See rules/tile.smk's + tile_finished() for the dependency cut (D5). """ return f"{PRODUCTS_DIR}/tiles/{tile[:2]}/{tile}/final_cat-{tile}.hdf5" def unit_num(unit): """$SP_UNIT_NUM: ShapePipe's image-number convention, dot -> dash, leading dash (tile ``210.282`` -> ``-210-282``; exposure ``2605805`` -> ``-2605805``). - The RULES do this transform; the configs just interpolate $SP_UNIT_NUM into - NUMBER_LIST (the `set_config_number_list` mechanism that replaced the retired - -e/--exclusive flag, #746).""" + The rules transform the number; configs interpolate $SP_UNIT_NUM into + NUMBER_LIST via set_config_number_list.""" return "-" + unit.replace(".", "-") -# Content hash of completeness.py, computed once at parse time and carried as a -# param on every rule: the default rerun-triggers' `code` trigger hashes only the -# rule's own shell string, NOT external scripts it calls — without this, a fix to -# the count table silently leaves stale manifests in place (bitten live). Scoped -# to completeness.py alone, the one script every shell line runs; build_forest.py -# gets its own hash, on the forest rule only. +# External scripts are not covered by the rule's shell-code rerun trigger. +# Carry their content hashes in params so script edits invalidate their callers; +# completeness.py is shared by the ShapePipe stage rules. def script_hash(name): """The 12-hex fingerprint of one script under workflow/scripts/.""" return path_hash(SCRIPTS / name) @@ -433,21 +382,13 @@ def script_hash(name): def path_hash(path): """The 12-hex fingerprint of any file the rules depend on but do not own. - A rule whose behaviour comes from more than its own script needs all of it - in one trigger. final_cat_merge is the case: what it writes is decided by - scripts/python/create_final_cat.py (the column extraction) and by - CONFIG_DIR's final_cat.param (which columns), and NEITHER is under - workflow/scripts/ nor a declared input. Without them in the hash, this PR's - own edits to both would have left every finished campaign's hdf5 untouched - and nothing would have said so. - - A MISSING FILE IS NOT A PARSE ERROR. This runs at module level, so raising - here kills EVERY invocation of the workflow — `sp --unlock`, `sp report`, - a dry run — over a file only one rule needs. Worse, it kills them with a - bare FileNotFoundError, which is exactly the diagnosis merge_final_cat.py - already carries and would print if the job were allowed to reach it. So the - hash degrades to a sentinel and says so once; the parse survives, the rule - still exists, and the job fails with the message written for it. + Include external implementation and configuration files in params hashes: + final_cat_merge depends on create_final_cat.py and final_cat.param as well + as its workflow script. + + An unreadable file returns a sentinel and warns in the head process rather + than blocking every workflow invocation at parse time. The job that needs + the file supplies its own diagnostic when it runs. """ path = Path(path) try: @@ -475,40 +416,23 @@ MERGE_FINAL_HASH = ":".join(( path_hash(Path(workflow.basedir).parent / "scripts" / "python" / "create_final_cat.py"), path_hash(CONFIG_DIR / "final_cat.param"))) -# ngmix_range.py earns a hash for a stronger reason than the others. What it -# emits is not a stale RESULT but a stale BOUNDARY, and a tile's eight chunks are -# a PARTITION of its catalogue rows: resume a tile across an edit to the split and -# the chunks that already succeeded keep the old ranges while the reruns take the -# new ones, so within one tile some objects are measured twice and others by -# nobody. merge_sep_cats concatenates whatever it is handed, so the tile -# completes green with a corrupt catalogue and no error anywhere. -# -# ACROSS TIME IS ALL THIS GUARDS NOW. The within-group version of the same -# corruption — eight sibling chunks each computing the split and disagreeing — is -# structurally closed: the partition is materialised ONCE per tile, by -# tile_vignets, and the chunks only look their row up (tile.smk's -# TILE_NGMIX_RANGES, which also keeps the incident transcript). So this hash buys -# REPRODUCIBILITY of the file across attempts and resumes, not agreement between -# siblings. It therefore fingerprints the script that PRODUCES the file, and -# is used by both tile_vignets and tile_ngmix because either rule can produce -# the file. -# -# Land this hash, and any later edit to the split, at a campaign boundary on a -# fresh root: a resume across it re-measures the tile, and the failure modes a -# mid-campaign params change can reach are catalogued at tile.smk's range_hash. +# Chunk ranges must remain one partition across attempts: mixing boundaries +# can omit objects or measure them twice without a merge error. tile_vignets +# writes the partition once; tile_ngmix only reads its row. Both carry this +# hash so a splitter edit also schedules the producer, not just its readers. +# See rules/tile.smk's TILE_NGMIX_RANGES and range_hash notes. +# Change params only at a campaign boundary on a fresh root; see +# workflow/CONTRACTS: unit-pre-changes-at-campaign-boundary. NGMIX_RANGE_HASH = script_hash("ngmix_range.py") # --- exposure reclamation (D5, S5) ----------------------------------------- def flag(value, default=False): - """Truthiness for a config value that may arrive as a STRING. + """Parse boolean config values, including strings from --config. - `--config clean=false` delivers the string "false", and every non-empty - string is truthy in Python — a plain bool() read that as ON and scheduled - the deletions the user had just switched off. YAML booleans pass through - unchanged; only strings are parsed, and an unparseable one is an error, not - a guess. + Python treats the non-empty string "false" as true, so strings need explicit + parsing to avoid enabling deletions. Reject unrecognised strings. """ if value is None: return default @@ -522,19 +446,13 @@ def flag(value, default=False): return bool(value) -# `clean:` in config.yaml gates the whole mechanism. Off => the rule generates no -# jobs at all (nothing requests a tombstone); flipping it on later reclaims -# RETROACTIVELY, because the exposures already cleaned are exactly the ones with -# a tombstone, so the missing tombstones schedule exactly the clean jobs. -# Only ever active under SP_PHASE=compute: the prepare and passthrough parses -# read no index of their own and must schedule no deletions. +# Request cleanup tombstones only when clean is enabled in the compute phase. +# Enabling it also requests cleanup for eligible exposures from earlier runs. CLEAN = flag(config.get("clean", False)) and PHASE == "compute" -# Reclamation REQUIRES the `input` rerun-trigger to be off (see tile.smk's -# tile_finished cascade note). The gate is structural rather than asserted: CLEAN -# needs SP_PHASE=compute, only bin/sp sets it, and bin/sp always launches with -# profiles/nibi — so the incompatible combination is unreachable. A runtime -# assertion is impossible (workflow.dag_settings is None until the DAG is built). +# Reclamation requires the `input` rerun trigger to be off; see rules/tile.smk's +# tile_finished note and the profiles selected by bin/sp. DAG settings are not +# available at this parse-time site, so the requirement is not asserted here. # Tiles that must not pin an exposure's store. See config.yaml: a permanently # failed tile otherwise holds every exposure it touches (~80) forever, because @@ -544,8 +462,10 @@ CLEAN_IGNORE_TILES = set(config.get("clean_ignore_tiles") or []) def tombstone(exp): - """The clean_exposure output. Lives BESIDE manifests/ and logs/, not inside - either: the clean job deletes both wholesale, and `sp report` scans it.""" + """The clean_exposure output, outside the directories cleanup deletes. + + sp report reads this tombstone after manifests/ and logs/ are removed. + """ return f"{exp_dir(exp)}/cleaned.json" @@ -557,25 +477,15 @@ def clean_consumers(exp): def clean_targets(): - """Which exposures this invocation may clean. - - An exposure is eligible only when every consuming tile is either in this - run's scope or has already produced its vignets on disk. Without that test, - requesting a tombstone for an exposure shared with a LATER batch would drag - that batch's whole tile chain into this DAG through the clean rule's input — - cleanup would otherwise expand the scope. Ineligible - exposures are simply skipped; the invocation that finishes their last - consumer picks them up. Deferral, never loss. - - Consumer sets are the IGNORE-FILTERED ones (clean_consumers), so a tile in - `clean_ignore_tiles` neither gates eligibility nor appears in the job's - input — which is the purpose of that list. - - HEAD PROCESS ONLY. This feeds `rule all` at module level, so it runs on every - parse — including the ~800 per-job re-parses under the slurm executor, none of - which can ever schedule `all`. Each of those was stat()ing the whole - campaign's tile_vignets manifests for nothing. The empty list a job parse - returns costs it exactly the target it could not have used anyway. + """Cleanup targets that do not pull out-of-scope tiles into this DAG. + + Every non-ignored consumer must be ready in this invocation or already have + its vignets manifest. Otherwise defer the exposure to an invocation that + finishes its remaining consumers. clean_ignore_tiles is filtered by + clean_consumers for both this test and the clean rule's inputs. + + Only the head process schedules all; job parses skip this campaign-wide + walk and its filesystem checks. """ if not CLEAN or not workflow.is_main_process: return [] @@ -583,9 +493,7 @@ def clean_targets(): for exp, raw in EXP_TILES.items(): if not raw: continue - # An empty set AFTER filtering means every consumer is ignored: nothing - # is left that could ever read this exposure, so it is eligible now. - # all() of an empty set is True, so this case is eligible. + # All consumers ignored: all([]) makes the exposure eligible. tiles = clean_consumers(exp) if all(t in READY_SET or Path(tile_manifest(t, "tile_vignets")).exists() for t in tiles): @@ -593,15 +501,9 @@ def clean_targets(): return sorted(out) # --- persisted exposure products (D5) -------------------------------------- -# The keep list is config, not a rule input, and it is READ HERE so that exactly -# one place converts it into the form the rule carries. -# -# OPTIONAL RETENTION, and only that. What star_cat_merge needs — every CCD's -# psf_validation — is packed by exp_persist whatever this list says -# (persist_exp.py's ALWAYS argues why: provenance for the merged catalogue, and -# a cheap tile append later). So an EMPTY list is a coherent instruction and not -# a switch that turns persistence off: the tar then holds the star catalogue's -# inputs and nothing else, and exp_persist still runs for every exposure. +# This list controls optional retention, not whether exp_persist runs. +# An empty list still retains the required star-catalogue inputs; see +# scripts/persist_exp.py's ALWAYS for the retention policy. PERSIST_EXP = list(config.get("persist_exp") or []) # Whether exposures have PSF products to persist at all. psf_model=fake (the @@ -617,11 +519,9 @@ PERSISTS_PSF = PSF_MODEL != "fake" def refuse_unpersistable_psf(psf_model): """Refuse a PSF model whose products the persistence path cannot read. - exp_persist packs PSFEx products (persist_exp.py maps psf_model to - *.psf) and star_cat_merge reads PSFEx's per-CCD validation table in HDU 2 - (merge_star_cat.py). MCCD writes fitted_model-.npy and one - exposure-wide table in HDU 1, so an MCCD campaign would run every exposure - and then fail at star_cat_merge. Refusing at parse time costs nothing. + See persist_exp.py for supported product patterns and merge_star_cat.py for + the per-CCD PSFEx validation reader. MCCD's exposure-wide products require a + different persistence table and reader, so reject that model before jobs run. """ if psf_model == "mccd": raise WorkflowError( @@ -633,12 +533,8 @@ def refuse_unpersistable_psf(psf_model): refuse_unpersistable_psf(PSF_MODEL) -# The keep list names PRODUCTS (`psf_model`), not globs (`*.psf`); the -# catalogue that maps one to the other lives in persist_exp.py, which is also -# what the rule runs, so there is one definition and not a copy here. -# UNKNOWN NAMES DIE AT PARSE TIME, listing the valid ones — a typo in a keep -# list would otherwise be a silently-empty keep or a per-exposure failure an -# hour into a campaign. +# Resolve keep entries through persist_exp.py's product catalogue. Reject +# unknown entries at parse time rather than failing once per exposure. import persist_exp as _persist # noqa: E402 for _entry in PERSIST_EXP: @@ -649,24 +545,11 @@ for _entry in PERSIST_EXP: def persist_targets(): - """Which exposures this invocation must pack PSF products off scratch for. - - `rule all` requests these DIRECTLY rather than reaching them only through - clean_exposure. Persistence and reclamation are different concerns — the - /scratch purge takes the store whether or not `clean:` is on — and hanging - the copy off the clean rule alone would mean a campaign run with clean:false - persists nothing and loses everything at the purge. + """Head-process persistence targets for the ready tiles' live exposures. - Scope is the ready tiles' exposures, which `all` already builds through the - tile chain, so nothing new is pulled into the DAG by asking. - - EXCEPT AN EXPOSURE WHOSE STORE IS GONE. Its exp_psf manifest is not there, - so requesting its persist manifest would make the DAG rebuild the whole - exposure chain from VOS — the avalanche tile.smk's reclaimed-edge cut exists - to prevent, arriving through a new target instead. exp_store_reclaimed() - below is that test, and what it does NOT test is the tombstone. - - HEAD PROCESS ONLY, for the same reason as clean_targets() above. + all requests persistence independently of cleanup because a scratch purge + can remove stores even with clean disabled. Reclaimed stores are excluded + by exp_store_reclaimed() to avoid rebuilding their exposure chains. """ if not workflow.is_main_process: return [] @@ -675,15 +558,10 @@ def persist_targets(): @functools.lru_cache(maxsize=1) def persist_manifests(): - """persist_targets() without the head-process guard, memoised. - - The guard on persist_targets() is a cost decision, not a correctness one: - `rule all` reads it at MODULE level, so a job parse would pay a whole - campaign's index walk for a target it can never schedule. star_cat_merge - reads the same list through an INPUT FUNCTION, which snakemake evaluates - only for parses that actually build that job — the head process, and the one - merge job's own re-parse under the slurm executor, which genuinely needs it. - So this half carries no guard and the memo keeps either parse to one walk. + """Memoised persistence manifests, available in head and job parses. + + Unlike persist_targets(), this helper has no head-process guard so an input + function can use it during a submitted job's parse without losing edges. """ return sorted(prod_exp_manifest(e, "exp_persist") for e in psf_exposures() if not exp_store_reclaimed(e)) @@ -699,105 +577,52 @@ def psf_exposures(): def exp_store_reclaimed(exp): - """True when this exposure's PSF products exist ONLY on the persistent root. - - The one condition both the persist target list and star_cat_merge's edge - choice turn on, and it deliberately does NOT read the tombstone. - - THE TOMBSTONE ALONE IS NOT THE EVIDENCE. It says clean_exposure ran, and - clean_exposure is only one of the two ways a scratch store disappears. - - WHAT THE TEST HAS TO SEPARATE is a store that is GONE from one that has not - been BUILT yet, and no single file says that. This runs at parse time, before - any exp_psf job of a fresh campaign has run, so "the exp_psf manifest is - missing" alone would skip every exposure of a new campaign and persist - nothing at all. The question is therefore: is there evidence this exposure - once had a store? Two files carry it, and either will do: - - * the TOMBSTONE — clean_exposure ran, so the store was built and reclaimed, - and whatever was going to be packed was packed before it went; - * a PERSISTED MANIFEST with no exp_psf manifest beside it — exp_persist - ran, so the store existed, and it is not there now. This is the purge - case, and it is the one the tombstone cannot see: /scratch is purged on - a 60-day window whether or not this workflow reclaimed anything, and it - leaves nothing behind. Keying on the tombstone alone meant that after a - purge, or on any campaign run with `clean: false`, every exposure looked - live, its persist manifest was requested, its exp_psf manifest was not - there, and snakemake rebuilt the entire exposure chain from VOS. - - An exposure with a LIVE store and a manifest is not reclaimed and is still - asked for, which is what lets an edit to `persist_exp:` re-pack in seconds - rather than be silently ignored — the whole reason exp_persist is a rule of - its own. An exposure with neither file is asked for too: either it has not - run yet, or it was purged having saved nothing, and only the DAG can tell - those apart by trying. + """True when there is evidence that a built scratch store is reclaimed. + + A cleanup tombstone is sufficient. A persisted manifest with no scratch + exp_psf manifest also detects a purge, which leaves no tombstone. Missing + exp_psf alone is insufficient: an unbuilt exposure also lacks it. + + Live stores remain eligible for repacking when persist_exp changes. With + neither tombstone nor persisted manifest, the DAG must try to build the + exposure; this test cannot distinguish an unbuilt store from a purged, + unsaved one. It does not verify that the persisted products still exist. """ return (Path(tombstone(exp)).exists() or (Path(prod_exp_manifest(exp, "exp_persist")).exists() and not Path(exp_manifest(exp, "exp_psf")).exists())) # --- the campaign-level merges --------------------------------------------- -# Two rules, one job each per campaign, both writing to the persistent root, and -# both the LAST link of a chain whose per-unit half the workflow already had: -# the exposure side ends in one `full_starcat_.hdf5` (every CCD's PSF -# validation catalogue — the rho/tau statistics input) and the tile side in one -# `final_cat_.hdf5` (every tile's final catalogue — the shear -# catalogue sp_validation reads). Until they existed the workflow's product set -# was two files short of what the old `combine_runs.bash` + `create_final_cat.py` -# chain delivered, and every campaign ended with a manual merge. -# -# NEITHER IS A LOCALRULE, and the arithmetic runs the opposite way from -# exp_persist's. Those rules are ~20k jobs of seconds each, so submitting them -# costs more in scheduling latency than the work; these are ONE job each over the -# whole campaign — ~800k catalogues stacked in memory, or ~20k catalogues read -# end to end at DR6 scale. That is a compute job, and it belongs on a node. +# One submitted job per campaign writes full_starcat_.hdf5 (PSF validation +# for rho/tau statistics); another writes final_cat_.hdf5 (shear catalogue). +# Both stream per-unit inputs to the persistent root. Campaign-wide I/O belongs +# on a compute node, unlike the short per-exposure persistence jobs. # -# NEITHER PUTS ITS INPUT PATHS IN ITS SHELL. `{input}` at DR6 scale is ~20k paths -# in a single argv entry, an order of magnitude over Linux's 128 KiB -# MAX_ARG_STRLEN, and the job would die on exec. So each rule's `input` is the -# DAG EDGE (what must exist first) and each script rediscovers the same set from -# the tile list and the index; what travels is a FINGERPRINT of that set's unit -# ids, on `params`, which is what makes the merge rerun when the set changes and -# not otherwise. -# The scripts' docstrings argue the rediscovery — it is also what lets the merges -# cover exposures whose scratch stores reclamation has since taken. +# Input paths order the DAG but do not travel in shell arguments: ~20k paths +# can exceed Linux's 128 KiB MAX_ARG_STRLEN limit for one argument. Each merge +# script discovers its inputs from the tile list and index; see its docstring. +# A params fingerprint of unit IDs detects changes to the set without reacting +# to manifest-to-product edge changes after reclamation. def unit_fingerprint(units): - """A short digest of a set of UNIT IDS, for a merge rule's `params`. - - `params` is a rerun trigger and a set of ids is not: appending a tile grows - the set, moves the digest and reruns the merge, while a rerun over the same - set leaves it alone. Sorted before hashing because the ORDER is not part of - what changed. - - IDS RATHER THAN THE RULE'S `input` PATHS. A path can change while the set - does not — star_cat_merge's edge for one exposure flips from its manifest to - its tar when the store is reclaimed — and a merge that reruns over identical - content on every reclamation pass is a rerun trigger firing on bookkeeping. - The ids are also exactly what the job derives on its own side, so both - halves agree on the set and on how it is named. + """Count and digest of unit IDs for a merge rule's params trigger. + + Sort before hashing so order does not matter. Callers supply unique IDs; + this function does not deduplicate them. See the campaign-level merge note + above for why the trigger uses IDs rather than input paths. """ joined = "\n".join(sorted(str(u) for u in units)) return f"{len(units)}:{hashlib.md5(joined.encode()).hexdigest()[:12]}" -# NO GATE ON THE KEEP LIST. star_cat_merge used to exist only when -# `persist_exp:` named something validation_psf-shaped, which made the -# campaign's star catalogue an opt-in and a typo away from silently absent. -# exp_persist now packs psf_validation unconditionally, so the merge is -# requested whenever the campaign has a persisted exposure at all, and -# star_cat_targets() below is the only condition left. - - def full_starcat(): """The campaign's merged star catalogue — the rho/tau statistics input. - hdf5, one dataset per exposure, named for the campaign exactly as the shear - catalogue beside it is. The old flat FITS table it replaces was called - `full_starcat-0000000.fits` and sp_validation still opens that name; - CosmoStat/sp_validation#340 moves its readers to this file, the same - migration that retires the `patches/` key on the galaxy side.""" + HDF5, one dataset per exposure. sp_validation reads it once + CosmoStat/sp_validation#340 lands; until then it opens the flat + full_starcat-0000000.fits. + """ return f"{PRODUCTS_DIR}/full_starcat_{CAMPAIGN}.hdf5" @@ -814,37 +639,17 @@ def prod_exp_tar(exp): @functools.lru_cache(maxsize=1) def star_cat_inputs(): - """What star_cat_merge waits for: every exposure of TILES_READY whose PSF - products are on the persistent root, live and reclaimed alike. - - THE SAME SET merge_star_cat.py derives at job time, and that equality is - load-bearing — the fingerprint on `params` is taken over THIS list, so - anything the job stacked that was not in it would be rows no rerun trigger - could see. The job states the rule from its own side: same tile list, same - index, exp_persist manifest present on the persistent root. By the time it - runs, every exposure below has one. - - RECLAIMED EXPOSURES BELONG IN THE STAR CATALOGUE. Carrying their PSF - products off scratch is exactly what exp_persist is for, and a merge that - dropped them would shrink the campaign's star catalogue every time - reclamation ran. But their exp_psf manifest is gone, so REQUESTING their - exp_persist manifest rebuilds the whole exposure chain from VOS — the - avalanche persist_targets() drops them to avoid. - ancient() DOES NOT HELP: it suppresses the timestamp comparison, not the - missing input, and snakemake schedules the chain anyway. Measured on smk-g6 - with one reclaimed exposure given a manifest by hand: the dry run grew - exp_get_images, exp_split, exp_psf and exp_persist jobs. - - So a reclaimed exposure is depended on through its TAR instead. The tar is - not a declared output of any rule (exp_persist declares only its manifest, - deliberately — persist_exp.py says why), so a tar that exists is a DAG leaf: - snakemake requires it and builds nothing. A live exposure keeps its manifest - edge, which is what orders the merge after the packing; its tar does not - exist yet, so it could not serve as the edge. - - An exposure reclaimed by a workflow PREDATING exp_persist has neither tar nor - manifest and is in no set at all. Nothing short of rebuilding its chain from - VOS recovers it; the merge reports how many exposures it found. + """Star-merge dependencies for live and reclaimed PSF exposures. + + Live stores use exp_persist manifests to order packing before the merge. + Reclaimed stores use existing tars as undeclared DAG leaves; requesting + their manifests would rebuild the exposure chain. ancient() suppresses + timestamp comparisons, not missing-input dependencies: a dry run measured + on a nibi run still scheduled retrieval, splitting, fitting and persistence. + + See durable_edge() for absent-product handling and merge_star_cat.py for + job-time discovery. The params fingerprint uses star_cat_exposures(), + not these dependency paths. """ return [p for exp in psf_exposures() for p in durable_edge(exp, "exp_persist", prod_exp_tar(exp))] @@ -863,15 +668,11 @@ def durable_edge(exp, stage, product): @functools.lru_cache(maxsize=1) def star_cat_exposures(): - """The exposure IDs star_cat_merge stacks — what its fingerprint is taken - over. - - THE IDS, NOT THE PATHS, and the difference is a rerun. An exposure's edge - FLIPS from its manifest to its tar the moment its store is reclaimed, so a - fingerprint over paths moves on every reclamation pass and reruns the merge - over content that did not change. The ids move only when the set does, which - is what the trigger is for. It is also what merge_star_cat.py derives on the - job side, so the two agree on the set AND on how it is named. + """Exposure IDs for the star merge's rerun fingerprint. + + Include live exposures whose persistence jobs can still run, and reclaimed + exposures with persisted manifests. See the campaign-level merge note for + why the fingerprint excludes dependency paths. """ return [e for e in psf_exposures() if Path(prod_exp_manifest(e, "exp_persist")).exists() @@ -922,62 +723,24 @@ def exposure_maps_targets(): # --- sizing the two merges (D4) --------------------------------------------- -# MEASURED, not guessed, and measured as a SLOPE rather than a single number: -# these are the only two rules whose one job's footprint grows with the whole -# campaign, so a constant is wrong by however much the campaign is not the one -# it was tuned on. -# -# Both slopes were measured on this login node, inside the campaign container, -# against synthetic tars for the star side and against smk-g6's real -# catalogues for the tile side. Peak RSS is getrusage(RUSAGE_CHILDREN). -# -# STAR SIDE, AND IT IS FLAT IN THE CAMPAIGN. The merge writes one hdf5 dataset -# per exposure and reads one exposure at a time, so it is sized on the LARGEST -# exposure's members — ~2 MB — not on the campaign's. What follows is the -# history of how that came to be true, because the numbers are the argument. -# -# The FITS full_starcat this replaced was one flat table, so the job held the -# whole campaign. Two fixture points, 20 and 80 exposures of 40 CCDs x 400 stars -# (1.6 MB of members per exposure, against the 2.0 MB measured on smk-m2), -# across the rewrites this PR made to MergeStarCatPSFEX: -# -# input members python lists arrays+concat two passes -# 32.3 MB 383 MB 238 MB 221 MB -# 129.0 MB 1313 MB 740 MB 661 MB -# slope 10.1x 5.5x 4.8x -# -# The tenfold was one python float object (32 bytes) plus a list pointer (8) per -# 4 bytes of float32 payload. Arrays per catalogue removed that; counting rows -# from the headers and filling a preallocated array removed the rest. What -# remained at 4.8x was the OUTPUT: file_io writes every float column as FITS 1D, -# so float32 became a float64 table astropy then buffered. -# -# Per-exposure hdf5 removes the term entirely rather than shrinking it — and -# with it the ~240 GB a DR6-scale flat table would have wanted. Those -# improvements stay upstream regardless: the module runner still merges to one -# FITS table, and they are its fix. -# -# TILE SIDE, and it is the reassuring one. Two points against real smk-g6 -# catalogues, 2 tiles (73.9 MB in, largest 39.6 MB) and 6 tiles (235.5 MB in, -# largest 47.7 MB): peak RSS 129 MB and 139 MB. FLAT IN THE NUMBER OF TILES — -# the merge holds one catalogue at a time — so it is sized on the LARGEST tile, -# not the total, at ~3x it plus the interpreter. -# THE CEILING ON ANY REQUEST, and it is not a formatting nicety: a mem_mb above -# the partition maximum is a job SLURM will never schedule and snakemake will -# never diagnose — it sits PENDING with a reason nobody reads while the campaign -# looks alive. The two merge formulas grow with the campaign, so at some size -# they WILL cross it; capping turns "silently never runs" into "runs on the -# biggest node there is, and possibly dies with a diagnosable OOM". +# Both merges stream one unit at a time, so size memory from the largest unit, +# not the campaign total. The star side reads ~2 MB of validation members per +# exposure; STAR_MEM_FACTOR reserves six times that payload plus the base. +# For the tile side, measurements on a nibi run inside the campaign container +# give peak RSS (getrusage(RUSAGE_CHILDREN)) of 129 MB for 2 tiles (73.9 MB total, +# 39.6 MB largest) and 139 MB for 6 tiles (235.5 MB total, 47.7 MB largest). +# This supports ~3x the largest tile; FINAL_MEM_FACTOR reserves 4x plus the base. # -# Nibi's standard compute node is 766 GB (192 cores, 4 GB/core); 750000 leaves -# room for the OS and the slurm accounting overhead. Override with `max_mem_mb:` -# for a cluster with smaller nodes, or to reserve headroom. +# Cap requests at a schedulable node size: over-limit SLURM jobs can stay pending +# indefinitely. A capped job may OOM, but that failure is diagnosable. +# Nibi's standard node has 766 GB; 750000 MB leaves OS and accounting headroom. +# Set max_mem_mb for smaller nodes or additional headroom. MAX_MEM_MB = int(config.get("max_mem_mb", 750_000)) _capped_warned = set() def capped_mem(mb, rule): - """min(mb, MAX_MEM_MB), and say so ONCE at parse time when it bites.""" + """Cap memory at MAX_MEM_MB and warn once per rule in the head process.""" mb = int(mb) if mb > MAX_MEM_MB: if rule not in _capped_warned and workflow.is_main_process: @@ -993,14 +756,12 @@ def capped_mem(mb, rule): STAR_MEM_BASE_MB = 500 # interpreter + astropy + h5py, rounded up -STAR_MEM_FACTOR = 6 # x the LARGEST exposure's members +STAR_MEM_FACTOR = 6 # x the largest exposure's members FINAL_MEM_BASE_MB = 800 -FINAL_MEM_FACTOR = 4 # x the LARGEST tile; ~3 measured -# What one unit costs when its product is not on disk yet to be measured — a -# fresh campaign sizes its merge before anything has been packed or made. The -# exposure figure is psf_validation's alone (the only members the star merge -# reads), not a whole tar's; both are the measured medians in config.yaml's -# persist_exp block and the D5 notes. +FINAL_MEM_FACTOR = 4 # x the largest tile; ~3 measured +# Fallback payload sizes before products exist: validation members only for +# exposures, final catalogues for tiles. See config.yaml's persist_exp notes +# and the D5 measurements in CosmoStat/shapepipe#848. EXP_BYTES_DEFAULT = 2_000_000 # The product whose members star_cat_merge stacks — named once, here and in # merge_star_cat.py, and resolved through persist_exp.py's catalogue. @@ -1018,32 +779,17 @@ def _size(path, default): def star_cat_max_bytes(): - """The LARGEST exposure's psf_validation members — what sizes the merge. - - The star merge holds ONE exposure at a time now that its output is hdf5 - with a dataset per exposure, so its memory is flat in the campaign exactly - as the tile side's is. Sizing on the total would ask a node for a campaign's - worth of memory to hold ~2 MB. - """ + """Largest exposure's validation payload, or the default for an empty set.""" return max(_star_cat_exposure_bytes() or [EXP_BYTES_DEFAULT]) def star_cat_bytes(): - """Total bytes of the members the star merge will actually read. - - THE TAR'S SIZE IS THE WRONG NUMBER, and increasingly wrong as the keep list - grows: the merge reads the psf_validation members and nothing else, while - the tar also holds whatever `persist_exp:` retains. With the default - retention that is 2.4x too much, and with the star_* products on it is ~36x - — a memory request that misses by more than an order of magnitude, and one - that would jump the moment an exposure got packed, since an unpacked one - contributed the per-exposure default instead. So the MANIFEST is read and - only the psf_validation members are counted; persist_exp records the product - each member came from, exactly so this is answerable without opening a tar. - - One json parse per exposure at DAG build, and only for the parse that builds - this job. An exposure not yet packed has no manifest and contributes the - measured default, which is the psf_validation figure and not the tar's. + """Total validation-member bytes, using defaults for unreadable manifests. + + Count members through persistence manifests, not tar sizes: optional + retention inflates the tar to 2.4x the validation payload with the default + keep list, or ~36x with star_* products. These bytes describe I/O volume; + star_cat_max_bytes() supplies the largest-unit memory estimate. """ return sum(_star_cat_exposure_bytes()) @@ -1059,10 +805,8 @@ def _star_cat_exposure_bytes(): continue try: body = json.loads(manifest.read_text()) - # By product name or, for a manifest written before that field - # existed or by a raw-glob keep list, by file name — the same test - # merge_star_cat.is_member() applies, so the sizing counts exactly - # the members the job will read. + # Match by product or filename, including manifests without a + # product field. See merge_star_cat.is_member() for the reader test. out.append(sum(f["bytes"] for f in body["files"] if f.get("product") == STAR_CAT_PRODUCT or fnmatch.fnmatch(f["name"], STAR_CAT_PATTERN))) @@ -1072,7 +816,7 @@ def _star_cat_exposure_bytes(): def final_cat_max_bytes(): - """The LARGEST tile catalogue the hdf5 merge will read — what sizes it.""" + """Largest tile catalogue size, with a fallback for unavailable products.""" return max([_size(final_cat(t), TILE_BYTES_DEFAULT) for t in TILES_READY] or [TILE_BYTES_DEFAULT]) @@ -1080,12 +824,9 @@ def final_cat_max_bytes(): def star_cat_targets(): """`full_starcat` when there is anything to stack into it, else nothing. - One way to get nothing, and it is a state rather than an error: every - exposure in scope is already tombstoned — a - campaign resumed after reclamation, whose exposures were cleaned by a - workflow that predates exp_persist and therefore left neither tar nor - manifest to read. A rule with an empty input list would still be a JOB, and - it would write an empty star catalogue over a good one. + Skip the merge when no dependencies are available, including fake-PSF + runs and reclaimed exposures without tars. An empty-input merge could + overwrite a good star catalogue with an empty one. """ if not workflow.is_main_process: return [] @@ -1099,13 +840,9 @@ def final_cat_targets(): return [final_cat_hdf5()] # --- tile reclamation (D5) -------------------------------------------------- -# A separate flag from `clean:` (config.yaml carries the full -# argument): exposure reclamation costs nothing but a rebuild if a tile is -# appended later, while tile reclamation destroys the per-tile AUDIT TRAIL that -# the campaign's cost model was derived from. They are not the same decision and -# must not be made by the same switch. Gated on SP_PHASE=compute exactly as -# CLEAN is, and read through flag() for the same reason (`--config -# clean_tiles=false` arrives as the string "false"). +# Tile cleanup has a separate switch because it removes the per-tile audit +# trail; see config.yaml for retention policy. Like CLEAN, it is compute-only +# and parses string booleans through flag(). CLEAN_TILES = flag(config.get("clean_tiles", False)) and PHASE == "compute" @@ -1140,24 +877,13 @@ def tile_tombstone(tile): def clean_tile_targets(): - """Which tiles this invocation may clean: the in-scope ones, and no test. - - No eligibility test, because a tile's store has no consumer outside that - tile (clean_tile.py argues the asymmetry with clean_targets()). Every ready - tile is eligible the moment its own final_cat exists, and the rule's input - is exactly that. - - The restriction to TILES_READY is therefore scope containment, not - eligibility, and it runs the opposite way from the exposure case: an - OUT-OF-SCOPE tile's tombstone would drag that tile's final_cat, its - tile_make_cat job and the whole fused tile_shape group into this DAG, where - the `params` rerun-trigger can find a finished tile out of date and rerun it - against reclaimed exposure stores (see the TILE_LOCAL warning in tile.smk). - In scope the edge is free: `rule all` already requests the same final_cat. - Nothing is lost by skipping an out-of-scope tile — the tile list only ever - grows, so a later invocation has it in scope. - - HEAD PROCESS ONLY, for the same reason as clean_targets() above. + """Head-process cleanup targets restricted to ready, in-scope tiles. + + The clean rule waits on final_cat; no other tile consumes this store. + Requesting out-of-scope tombstones would pull those tiles' shape groups + into the DAG, risking params-triggered reruns against reclaimed exposures. + See scripts/clean_tile.py for the cleanup boundary and rules/tile.smk's + TILE_LOCAL warning for rerun hazards. """ if not CLEAN_TILES or not workflow.is_main_process: return [] @@ -1174,50 +900,24 @@ _THREAD_CAPS = " ".join( def unit_pre(stage, unit, *, exp_name=None, forest=None, env=None, pre_run=()): - """The unit-furniture + environment prologue, as bash. - - Returned as a rule ``params`` value, NEVER inlined into the ``shell:`` - string: snakemake formats a shell string ONCE, so a ``{output}``/``{threads}`` - placeholder inside a params value would survive literally — and, conversely, - the literal ``${SP_NGMIX_CHUNK}`` braces this prologue needs would blow up - that formatting if they lived in the shell string. Params values are - substituted after formatting, so both hazards go away together. - - What it materialises (the proven v2.0 isolation-by-work-dir-content, NOT - -e/--exclusive): - * ``output/``, ``manifests/`` and ``logs/`` — the last two are a SIBLING - pair, not one dir with two naming conventions, because clean_exposure - deletes both wholesale (clean_exposure.py says why). The benchmark tsv - is deliberately outside both: it measures the job, not the store, so - reclamation must not eat it. - * tile: ``tile_numbers.txt`` (dot format — what get_images reads); - * exposure: a fabricated pseudo-Fe ``exp_numbers-000-000.txt`` holding the - ORIGINAL exposure name from the index (``2605805p``), so get_images - matches ``.fits.fz`` in the store — the bare base id matches - nothing. Written UNCONDITIONALLY: an exists-guard once pinned a stale - pre-fix file with the bare id. - There is no per-unit ``cfis`` symlink any more: $SP_CONFIG points straight at - the committed config dir. - - It also exports the configured input roots as ``SP_INPUT_TILES``, - ``SP_INPUT_EXPOSURES`` and (data only) ``SP_INPUT_MASKS``, the retrieve - mode as ``SP_RETRIEVE`` and the PSF choice as ``SP_PSF`` for the committed - ini chain (plus ``PSF_DICT`` for the image-simulation true PSF). - - Every line here is part of each rule's ``params.pre`` and so of the - ``params`` rerun trigger: a line added for every rule reruns every finished - unit of a campaign on its next ``sp run``, so lines are added only at a - campaign boundary. The image-simulation lines are conditional, so a data - prologue does not carry them. - - ``tile_numbers.txt`` carries the number in the form the INPUT files are - named with, because get_images substitutes it verbatim into - INPUT_FILE_PATTERN: dot format for the survey tiles (``CFIS.210.282.r``), - dash format for the simulations (``CFIS_simu_image-210-282``). - - Finally it ``rm -rf``s this stage's own fixed run dir — ShapePipe's - FileHandler raises on an existing run dir, and it is how a rerun never sees - stale products (D2: the job clears its run dir at start). + """Build the unit's directory and environment prologue as bash. + + Return it as params.pre rather than embedding it in shell: Snakemake + substitutes params without recursively formatting their contents, preserving + literal bash braces. Rule placeholders such as {output} belong in sp_shell. + + Create output/, manifests/ and logs/, then rewrite the image-number list: + tiles use dots for survey filenames and dashes for simulation filenames; + exposures use the original indexed name (2605805p), not the base ID. + SP_CONFIG points to the committed config chain. Export input roots, retrieval + and PSF choices, plus any caller-supplied environment and pre-run commands. + + WARNING: emitted lines are params rerun triggers. Change them only at a + campaign boundary on a fresh root; see workflow/CONTRACTS: + unit-pre-changes-at-campaign-boundary. + + Finally remove this stage's fixed run directory so FileHandler can create + it and a rerun cannot read stale stage products (D2). """ # The stage table already knows both halves of "where this stage writes": # its level and its run dir. Taking them from there rather than from an @@ -1264,25 +964,16 @@ def unit_pre(stage, unit, *, exp_name=None, forest=None, env=None, def sp_shell(stage, config_name, *, check_args="", post=""): """The rule's shell string: prologue, one shapepipe_run, one completeness check. - EVERY rule that runs shapepipe_run goes through here, including the two that - need a little more than the plain body — they pass it rather than restating - it, so the rc composition below has exactly one definition: - * ``check_args`` — extra flags for the completeness check. tile_vignets - points it at the NODE-LOCAL run root (``--run-dir "$SP_LOCAL"``) and keeps - the manifest's unit field the tile ID rather than the local root's - basename (``--unit``), which also keeps the manifest content independent - of an ephemeral path the mtime rerun-trigger reads. - * ``post`` — bash between the check and the ``exit``. tile_make_cat - publishes its final_cat there, guarded on ``rc``. - - ``{threads}``, ``{output}`` and ``{log}`` are placeholders HERE and nowhere - else (see unit_pre). ``-b {threads}`` makes SMP fork width and cpus_per_task - one number by construction (D4). - - The rc is CAPTURED rather than ``&&``-ed onto, so the check still runs when - shapepipe_run failed, and it is passed back as ``--job-rc "$rc"`` so the - verdict composes the count checks with shapepipe_run's own exit status. Both - halves are argued in completeness.py's docstring. + check_args supplies extra completeness flags; tile_vignets uses it to check + the node-local root while identifying the manifest by tile ID. post supplies + bash after the check; tile_make_cat publishes final_cat there, guarded on rc. + + Format rule placeholders here, not inside params.pre (see unit_pre). + -b {threads} matches ShapePipe's SMP batch size to the rule's CPU request. + + Capture ShapePipe's exit status so completeness still runs after failure. + Pass it as --job-rc to combine process status with product counts; see + scripts/completeness.py's docstring for the verdict contract. """ return ( "{params.pre}\n" @@ -1306,17 +997,10 @@ include: "rules/tile.smk" # they would be ~20k (clean_exposure) or ~23k (clean_tile) sbatch submissions at # DR6 scale for work shorter than the scheduling latency. # -# Both are DAG LEAVES, so neither constrains a `group:` label. (A mid-chain -# localrule would: a local job cannot be fused into a submitted group. The old -# star-catalogue rules were exactly that, and they are gone with the internal -# mask generation.) -# -# exp_persist joins them for the same arithmetic — one tar of a few MB per -# exposure, ~20k of them at DR6 scale, each far shorter than the scheduling -# latency that would submit it (exposure.smk argues the placement in full). It -# sits mid-chain between exp_psf and clean_exposure, but both of those are -# outside every group already (exp_psf is heavy, clean_exposure is local), so it -# adds no new grouping constraint. +# Both clean rules are terminal jobs outside submitted groups. A local job +# cannot be fused into a submitted group. +# exp_persist is also local because packing a few MB is shorter than submission +# latency. See rules/exposure.smk for its placement outside groups. localrules: all, prepare_all_tiles, clean_exposure, clean_tile, exp_persist @@ -1338,10 +1022,8 @@ rule prepare_all_tiles: [tile_manifest(t, "tile_find_exposures") for t in TILES] # --- report hooks ----------------------------------------------------------- -# run_report is NOT a DAG node (a descendant of every job would be poisoned by -# any hard failure — the exact case it exists for). It is a standalone script, -# emitted automatically at the end of the COMPUTE invocation, runnable any time -# via `sp report`. +# Run the report outside the DAG so job failures cannot block it. Compute-phase +# success and error hooks invoke it; `sp report` also runs it independently. def _report(status): shell(f"SP_TILE_DETECTION='{TILE_DETECTION}' " f"python {SCRIPTS}/run_report.py --run-dir {RUN_DIR} " diff --git a/workflow/bin/sp b/workflow/bin/sp index cb6754108..e75370849 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -8,14 +8,11 @@ # Snapshots the code into the state dir first (see "the # launch code snapshot" below) and runs out of the copy, so # editing the checkout mid-campaign cannot reach the jobs. -# Two snakemake invocations over one Snakefile: -# 1. PREPARE snakemake prepare_all_tiles -# 2. COMPUTE snakemake all <- its PARSE builds the index -# ARGS (--jobs, -n, --forcerun, ...) pass through to BOTH. -# FILE is a user-specified config file that can be used -# to add and/or overwrite entries in the default config -# file workflow/config.yaml (read automatically) -# sp report [ARGS...] emit run_report.json now (mid-run is fine). +# Two Snakemake invocations: prepare_all_tiles, then all. +# See workflow/Snakefile for index construction between phases. +# ARGS (--jobs, -n, --forcerun, ...) pass through to both. +# FILE overrides workflow/config.yaml (read automatically). +# sp report [ARGS...] emit run_report.json (mid-run is fine). # sp container VERB manage the image every job runs inside: pull, status, # sandbox, exec, resolve. `sp container --help` documents # the layers and the resolution order. @@ -23,17 +20,14 @@ # Anything else is passed straight through to snakemake with the same profile and # state dir (the escape hatch: `sp --unlock`, `sp exp_psf ...`, `sp --dag`). # -# It also loads the apptainer module (snakemake resolves `apptainer` via PATH at -# job runtime) and activates the snakemake venv on the shared /project FS (the -# executor re-invokes it inside jobs, so it cannot live on a node-local path). -# One entry point, so a fresh tmux or a restart after a crash always launches -# with the right state. +# It ensures apptainer is on PATH and activates the Snakemake venv on shared +# /project storage: the executor re-invokes Snakemake inside compute jobs. # # Typical use, from the repo root: # # SP_PROFILE=candide workflow/bin/sp run -c ~/my_run.yaml # a campaign # SP_PROFILE=candide workflow/bin/sp run -c ~/my_run.yaml -n # dry run -# SP_PROFILE=candide workflow/bin/sp report -c ~/my_run.yaml # status now +# SP_PROFILE=candide workflow/bin/sp report -c ~/my_run.yaml # status # # Run config: -c FILE, also --config-file. Read after workflow/config.yaml and # merged on top; anything still unset comes from the machines: entry for @@ -62,13 +56,9 @@ VENV="${SP_SNAKEMAKE_ENV:-/project/def-mjhudson/cdaley/snakemake-env}" SCRIPTS="$HERE/scripts" PROFILE="${SP_PROFILE:-nibi}" -# -c is consumed here, not forwarded: snakemake's own --configfile merges -# differently, and the jobs and the snapshot need it as SP_RUN_CONFIG. The scan -# stops at `--` and at the command after `container exec`, so a -c meant for -# the delegated command (`sp container exec python -c ...`) or for snakemake's -# own escape hatch (`sp -- ...`) is left alone. `--config` is NOT sp's: keeping -# it free lets snakemake's own `--config k=v` (what the Snakefile's flag() -# reads) pass through untouched. +# Consume -c as SP_RUN_CONFIG for shared config resolution and snapshotting. +# Stop at -- or container exec so delegated commands keep their own -c. +# Pass --config k=v through to Snakemake unchanged. RUN_CONFIG="${SP_RUN_CONFIG:-}" _sp_args=() while [ $# -gt 0 ]; do @@ -81,9 +71,7 @@ while [ $# -gt 0 ]; do -c=*|--config-file=*) RUN_CONFIG="${1#*=}"; shift ;; container) - # Everything from `exec`'s own command on is the delegated command's, - # not sp's: `container.py` reads SP_RUN_CONFIG, so its own -c is never - # sp's run config. + # container.py reads SP_RUN_CONFIG; exec arguments belong to its command. _sp_args+=("$1"); shift [ "${1:-}" = exec ] && { _sp_args+=("$@"); break; } ;; @@ -103,12 +91,9 @@ fi [ -d "$REPO/profiles/$PROFILE" ] || { echo "sp: no profile $REPO/profiles/$PROFILE (SP_PROFILE=$PROFILE)" >&2; exit 2; } -# snakemake resolves `apptainer` via PATH at job runtime, so all this has to -# do is put it there. candide already has it in /usr/bin and ships no -# modulefile for it, so probing for one only produces a scary ERROR line; -# skip the module entirely when apptainer is already usable. Both streams are -# dropped because the message arrives on stderr under environment-modules and -# can arrive on stdout under Lmod. +# Load a module only if apptainer is absent from PATH; candide supplies it in +# /usr/bin without a modulefile. Suppress both module output streams because +# environment-modules and Lmod report errors on different streams. if ! command -v apptainer >/dev/null 2>&1; then module load apptainer/1.4.5 >/dev/null 2>&1 || true command -v apptainer >/dev/null 2>&1 || { @@ -127,9 +112,8 @@ fi # Snakefile resolves it. cfg() { python "$SCRIPTS/run_config.py" "$HERE/config.yaml" "${SP_RUN_CONFIG:-}" "$1"; } RUN_DIR="$(cfg outputs.run_dir)"; INDEX_DB="$(cfg outputs.index_db)" -# `sp container`, `sp cancel` and the help forms (bare `sp`, -h, --help) need -# neither a campaign nor its state dir; every other verb does. Checked here, before the state dir is created from -# RUN_DIR: with `run:` unset, RUN_DIR still holds a literal `$run`. +# Container, cancel and help need no campaign state. Check other commands +# before mkdir: an unset run leaves a literal $run in RUN_DIR. case "${1:-}" in container|cancel|""|-h|--help) NEEDS_CAMPAIGN=0 ;; *) NEEDS_CAMPAIGN=1 ;; esac if [ "$NEEDS_CAMPAIGN" = 1 ]; then if [ -z "$(cfg run)" ]; then @@ -140,50 +124,33 @@ if [ "$NEEDS_CAMPAIGN" = 1 ]; then fi fi -# Snakemake state (.snakemake: metadata, locks, incomplete markers) lives NEXT TO -# THE RUN on /scratch, never on the persistent root — the one exception to D5's -# "durable, low-volume products go on /project". It is neither: one small, hot -# metadata file per output, rewritten on every job, which at DR6 is >170k files -# of churn against a backed-up filesystem with a 1M-inode group quota. It is also -# reconstructible — losing it costs a re-parse, not a re-run. (A hard 27/27 TiB -# group quota killed a metadata write mid-run, live. The quota has since eased; -# the placement is right on its own merits.) -# --directory only moves state: all data paths are absolute, and the Snakefile +# Default state location: beside the run on /scratch, not the persistent root. +# DR6 produces >170k frequently rewritten metadata files; keeping that churn +# off backed-up /project avoids pressure on its ~1M-inode group quota. +# --directory moves state, not data: data paths are absolute, and the Snakefile # resolves its own configfile. STATE_DIR="${SP_STATE_DIR:-${RUN_DIR}-state}" [ "$NEEDS_CAMPAIGN" = 0 ] || mkdir -p "$STATE_DIR" # --- the launch code snapshot ---------------------------------------------- -# THE ONE HOME for this concept; everything else points here. +# SLURM jobs re-parse the Snakefile and read scripts/configs when they start, +# potentially hours after launch. sp run copies code to $STATE_DIR/code so +# checkout edits cannot give jobs in one campaign different implementations. # -# WHY. A campaign is not one process. The SLURM executor re-invokes snakemake on -# every job's node, so each job RE-PARSES the Snakefile, and each shell reads -# workflow/scripts/*.py and workflow/config/cfis/* when it STARTS -- hours after -# launch, out of whatever the checkout contains THEN. Editing the checkout -# mid-campaign therefore fed different code to different jobs of the same run -# (the 186.307 partition incident: an edit landed four seconds into a tile split -# it into two disagreeing halves). +# @sc [label:custody] launch-snapshot-self-contained +# The snapshot includes workflow/, src/, scripts/ and profiles/. Dereference +# config symlinks into example/ so jobs need no live checkout. Workflow paths +# resolve from workflow.basedir, including the path-loaded final catalogue +# script; rewrite the profile's PYTHONPATH to the snapshot's src/. Copy the +# external run config too, so every job parse reads the launch configuration. # -# WHAT. `sp run` copies the code it is about to launch into $STATE_DIR/code and -# runs the campaign entirely out of that copy: the Snakefile, the rules, the -# scripts, the ini chain (symlinks DEREFERENCED -- workflow/config/cfis points -# into example/, and the copy must be self-contained), src/, the repo's own -# scripts/ (final_cat_merge loads scripts/python/create_final_cat.py by path -- -# it is a script, not an installed module, and the hdf5 layout it defines must -# be pinned to the campaign like everything else here), and the profile. -# Every workflow-internal path hangs off `workflow.basedir`, which IS the -# snapshot, so they all follow it for free; the profile's PYTHONPATH pin is the -# one that cannot (YAML splices nothing) and is rewritten below. -# -# CONSEQUENCE, and the reason the old "never edit workflow/ while a campaign -# runs" warnings are gone: mid-campaign edits to the checkout are inert. A change -# takes effect on the next `sp run`, and not before. -# -# LIFECYCLE: one snapshot per campaign state dir, REFRESHED WHOLESALE on every -# `sp run` -- a new `sp run` IS the relaunch, and snakemake's params/code rerun -# triggers decide what the new code invalidates. Nothing accumulates, and the -# path is stable, so a later `sp --unlock` reaches the same workflow that took -# the lock. +# @sc [label:hazard] launch-snapshot-refresh-boundary +# Every sp run refreshes the same snapshot path wholesale. Relaunch only after +# prior jobs using that state directory have stopped: queued or running jobs +# can otherwise read the refreshed code. Snakemake rerun triggers determine +# which finished products the relaunch invalidates (see workflow/CONTRACTS, +# unit-pre-changes-at-campaign-boundary). Other verbs reuse the existing copy, +# so sp --unlock reaches the workflow that took the lock. SNAPSHOT="$STATE_DIR/code" snapshot_code() { mkdir -p "$SNAPSHOT" @@ -196,12 +163,8 @@ snapshot_code() { find "$SNAPSHOT" -name __pycache__ -type d -prune -exec rm -rf {} + fi - # The profile's `apptainer-args:` hardcodes one checkout's src/ in its PYTHONPATH - # pin and CANNOT interpolate anything (snakemake escapes a literal `$` before the - # string reaches any shell -- the post-mortem is on that line). So the pin is - # rewritten HERE, in the snapshot's copy, and `sm` points --profile at that copy. - # The checked-in profile keeps its literal checkout path and stays the source of - # truth for the FLAGS; exactly one path is substituted, by one line of sed-work. + # Rewrite only PYTHONPATH in the generated profile; the checked-in profile + # owns the flags. Variable expansion limits are documented in profiles/nibi. python - "$SNAPSHOT/profiles/$PROFILE/config.yaml" "$SNAPSHOT/src" <<'PY' import pathlib, re, sys f, src = pathlib.Path(sys.argv[1]), sys.argv[2] @@ -213,20 +176,10 @@ if not n: f.write_text(text) PY - # `tile_store_root:` in the run config moves the node-local tile store - # (tile.smk's tile_local(): the vignette + WCS stores of each tile_shape group) - # off the profile's `/local/scratch` bind, for this campaign only. It is the - # same substitution as the pin above -- the BIND SOURCE of `/local/scratch` in - # the snapshot's profile -- and deliberately not a path in tile_local(): that - # function's output is a params rerun trigger, so a path there would reschedule - # every finished tile of a resumed campaign (its warning block says how badly), - # while a bind is not a trigger at all (a dry run after changing it schedules - # only the missing tiles). Why a campaign wants it: candide binds the node's - # 31 GB /tmp, which a few dense image-sim tile stores or failed groups' orphans - # fill, after which every later group on that node fails in seconds. A shared - # directory (NFS) trades ngmix read latency for room; the campaign-unique store - # names (LOCAL_TAG) keep concurrent campaigns apart in one root. Profiles with - # no `X:/local/scratch` bind (nibi's `--bind /local`) get one appended. + # Move the tile store by rewriting the bind source, not tile_local()'s params: + # params changes invalidate finished tiles (workflow/CONTRACTS). This bind is + # not a rerun trigger. Append it if the profile has no X:/local/scratch bind. + # See workflow/config.yaml for tile_store_root's capacity/latency trade-off. python - "$SNAPSHOT/profiles/$PROFILE/config.yaml" "$(cfg tile_store_root)" <<'PY' import os, pathlib, re, sys f, root = pathlib.Path(sys.argv[1]), sys.argv[2] @@ -282,24 +235,13 @@ PY # declared tiles allowed to be missing their exposure list (default 0.0). export SP_MISSING_THRESHOLD="${SP_MISSING_THRESHOLD:-0.0}" -# WHICH code a verb runs. `sp run` refreshes the snapshot and runs out of it; -# every other verb and the escape hatch run out of the EXISTING snapshot when the -# state dir has one -- `sp --unlock` and `sp report` must speak for the campaign -# that is actually on disk, not for a checkout that has moved since. Before the -# first `sp run` there is no snapshot and the live checkout is all there is. -# (`sp container` is the exception: it is interactive tooling about the image, -# not about a campaign, and stays on the checkout.) +# Use the campaign snapshot when present; otherwise use the checkout. +# Container tooling bypasses this helper because it is not campaign-specific. code_root() { [ -f "$SNAPSHOT/workflow/Snakefile" ] && echo "$SNAPSHOT" || echo "$REPO"; } -# --snakefile pins the workflow to one tree: sp must work from any cwd (an sbatch -# head job starts in the submission directory, not the repo), and the SLURM -# executor re-invokes snakemake with this same path inside every job -- which is -# exactly what makes the snapshot stick. -# -# SP_PHASE is REQUIRED by the Snakefile (a bare `snakemake` would build an empty -# DAG and exit 0). `sp run` sets prepare/compute on its two invocations; every -# other verb and the escape hatch fall through to `passthrough`, which parses the -# index without building it and schedules no side effects. +# Absolute --snakefile works from any cwd and pins executor job parses to the +# same tree. Set the phase required by the Snakefile: run uses prepare/compute; +# other commands default to passthrough (see the Snakefile's SP_PHASE gate). sm() { local root; root="$(code_root)" # The run config the Snakefile reads: the snapshot's copy when this run has @@ -318,11 +260,8 @@ case "$cmd" in run) shift snapshot_code - # PREPARE failing is NOT fatal to the run: keep-going means a failed tile - # poisons only its own cone, and the tiles that lost their exposure list are - # dropped at the compute parse. The real gate is SP_MISSING_THRESHOLD, which - # the compute parse's index build enforces over the WHOLE tile list — so we - # record the failure and go on rather than letting `set -e` abort here. + # Continue after prepare failure so compute can apply SP_MISSING_THRESHOLD + # to the whole tile list. Retain prepare's failure for the final exit code. prep_rc=0 SP_PHASE=prepare sm prepare_all_tiles "$@" || prep_rc=$? if [ "$prep_rc" -ne 0 ]; then @@ -344,10 +283,7 @@ case "$cmd" in fi ;; report) - # Out of the snapshot when there is one, so a manual report reads the trees - # with the same script the campaign's own onsuccess/onerror hooks use (they - # call it through the Snakefile's SCRIPTS, which is the snapshot). Read-only - # either way -- this is consistency, not safety. + # Use the same snapshot script as the campaign's automatic report hooks. shift SP_TILE_DETECTION="$(cfg tile_detection)" \ python "$(code_root)/workflow/scripts/run_report.py" \ @@ -355,21 +291,13 @@ case "$cmd" in --status manual "$@" ;; container) - # The image layer, outside snakemake entirely: the script is stdlib-only so - # it runs on the bare host, and the apptainer module is already loaded above. - # Deliberately the CHECKOUT's copy, not the snapshot's: `sp container` is - # about the image you are working with now (status compares against HEAD, and - # the snapshot has no .git), and `exec` should hand you the environment your - # NEXT run will have. It reads the checked-in profile's apptainer-args, so a - # running campaign whose snapshot predates an edit to that line gets the old - # value while `exec` shows the new one -- the only place the two can differ, - # and it differs in the direction that matters (jobs stay pinned). + # Run host-side container tooling from the checkout: status compares against + # HEAD, and exec uses the checked-in profile rather than a campaign snapshot. shift python "$SCRIPTS/container.py" "$@" ;; cancel) - # Kept only because it is two lines: scancel this workflow's jobs by name - # before an --unlock. Not part of the design surface. + # Cancel the user's jobs whose names match the supplied regular expression. run="${2:?usage: sp cancel }" squeue --me --noheader --format='%i %j' \ | awk -v r="$run" '$2 ~ r {print $1}' | xargs -r scancel diff --git a/workflow/config.yaml b/workflow/config.yaml index 99f96daf9..4741dd3ec 100644 --- a/workflow/config.yaml +++ b/workflow/config.yaml @@ -13,20 +13,20 @@ # Main keys, global for all machines -# input_tyle: allowed are data or image_sims +# input_type: allowed are data or image_sims input_type: data # Run name, available as `$run` in the paths below; it also names the # campaign's merged catalogues. Required, and set by the run config. -# run: smk-g6 +# run: my_campaign # Default settings per input_type, can be overridden by a user-defined run config file. # - tile_detection: both run SExtractor on the tile. `unions_catalogue` (data) # also fetches the UNIONS per-tile catalogue and joins the detections to it, # taking its NUMBER (needs the DR6 tile image); `sextractor` keeps # SExtractor's own (required for image_sims) -# - psf_model: allowed are `psfex` (default), `mccd`; `fake` (image_sims only, -# PSF read from psf_dict) +# - psf_model: `psfex` (data default), `fake` (image_sims default, read from +# psf_dict). The Snakefile refuses `mccd`; see refuse_unpersistable_psf. input_types: data: # @sc [decision:detection.tile_detection] @@ -57,7 +57,7 @@ input_types: # - outputs.products_dir: path to final products (default run_dir; one root # needs clean: false and clean_tiles: false) # - outputs.index_db: path to bookkeeping index sqlite file -# - container: TBD +# - container: image fallback; see workflow/scripts/container.py for resolution machines: nibi: @@ -98,98 +98,21 @@ machines: psf_dict: /home/hervas/fhervas/workdir_skills/input/psf_files/Full_psf_dict.pickle container: /n17data/cdaley/containers/shapepipe_im_sims-runtime.sif -# OPTIONAL per-exposure retention: what to carry onto the persistent root ON TOP -# OF the star catalogue's own inputs, before the scratch store goes -# (`exp_persist`, exposure.smk). -# -# WHAT IS ALWAYS KEPT, AND IS NOT A CHOICE HERE: psf_validation, the psfex_interp -# validation catalogue, one per CCD. `star_cat_merge` stacks every one of them -# into /full_starcat_.hdf5, so they are that -# catalogue's PROVENANCE — a merged star catalogue with no per-exposure inputs -# beside it cannot be audited, re-cut, or recomputed after a purge — and they are -# what keeps APPENDING TILES CHEAP, since a tile added next month brings -# exposures whose catalogues must join the existing stack. ~2 MB per exposure: -# ~40 GB and ~40k inodes at DR6 scale, against a ~1 M-inode group quota. That is -# the price of being able to say where the number came from, and it is paid. -# -# SO THIS LIST IS PURELY ADDITIVE, and an empty one is a coherent instruction: -# the tar then holds the star catalogue's inputs and nothing else. -# -# Entries are PRODUCT NAMES — not globs. The catalogue below is the rendering of -# workflow/scripts/persist_exp.py's PRODUCTS table, which is the single source of -# truth for what each name means and what keeping it buys -# (CosmoStat/shapepipe#844); print it any time with +# Optional per-exposure retention, in addition to mandatory psf_validation. +# Entries name products in workflow/scripts/persist_exp.py's PRODUCTS table; +# that file owns product meanings, sizes, glob syntax and additive retention. +# Print the catalogue with: # # workflow/bin/sp container exec python workflow/scripts/persist_exp.py --list-products # -# product glob size/exposure -# -------------- -------------------------- ------------- -# star_selection star_selection-*.fits 24.5 MB -# setools' PRE-SPLIT selection. The only file that answers which -# stars the selection cuts rejected and why; the split samples have -# already lost the rejects. -# star_train star_split_ratio_80-*.fits 19.9 MB -# the 80% TRAINING sample, the stars PSFEx actually fitted. Rows -# duplicate star_selection. -# star_test star_split_ratio_20-*.fits 7.1 MB -# the 20% VALIDATION sample — the positions psf_validation's rows -# correspond to. Rows duplicate star_selection. -# star_stats star_stat-*.txt unmeasured -# setools' per-CCD STAT block: star counts, FWHM mode and cuts. -# The selection's summary without its catalogue. -# psf_model *.psf 2.8 MB -# the PSFEx model itself. Keeping it means the PSF can be -# re-interpolated at ANY position later without rebuilding the -# exposure chain — the single most capability-adding entry here. -# psfex_cat psfex_cat-*.cat unmeasured -# PSFEx's own output catalogue (FITS_LDAC): the per-star FLAGS_PSF -# and CHI2_PSF, i.e. WHICH stars outlier rejection clipped. Not -# recoverable from anything else — the .psf header keeps only the -# LOADED/ACCEPTED counts. -# psf_validation validation_psf-*.fits 2.0 MB -# the psfex_interp validation catalogue, one per CCD: the input to -# the rho/tau statistics, and to the star_cat_merge rule that stacks -# them into the campaign's full_starcat. -# -# A RAW GLOB IS STILL ACCEPTED, as an escape hatch for a file the catalogue does -# not name yet: anything carrying a glob metacharacter or a dot is taken as a -# glob rather than a name (`*.psf` is a glob, `psf_model` is the name for it). -# An unknown NAME is a parse-time error listing the valid ones. -# -# Matches are packed, flat, into ONE uncompressed tar per exposure: -# /exp///psf/.tar, with a manifest listing the -# members (and the product each came from) beside it. One tar rather than loose -# copies because inodes, not bytes, bind on /project. FITS members read straight -# from the tar: fits.open(io.BytesIO(tarfile.open(t).extractfile(m).read())). -# -# WHY COPY RATHER THAN EXEMPT THESE FROM CLEANUP. Reclamation is not the threat. -# run_dir is /scratch and is PURGED on a 60-day window whether or not -# clean_exposure ever ran; products_dir is /project, backed up and not purged. -# The only way a per-exposure product outlives its campaign is to leave the -# filesystem. (Ordering is free: clean_exposure takes the exp_persist manifest -# as an input, so a store is never reclaimed before its keepers are written.) -# -# EDITING THIS LIST IS CHEAP. It rides on exp_persist's `params`, so a change -# reruns the packing (seconds) and NOT exp_psf (four hours per exposure). That -# separation is the whole reason exp_persist is a rule of its own. -# -# THE DEFAULT is psf_model (2.8 MB per exposure on top of psf_validation's 2.0): -# the model that lets the PSF be re-interpolated at any position later without -# rebuilding the exposure chain from VOS, which is the single most -# capability-adding thing an exposure can keep. Add psfex_cat for a production -# run if you want to know which stars PSFEx clipped; the star_* products are for -# selection studies and cost an order of magnitude more. -# -# THIS LIST IS EXPOSURE-SIDE ONLY. Tile-side retention is not configurable: the -# only tile product that persists today is final_cat, written by tile_make_cat -# straight to products_dir. A tile keep list is #844 follow-up. +# The default psf_model preserves the model for interpolation at other positions. +# Add psfex_cat to inspect PSFEx's rejected stars, or star_* products for +# selection studies. Changing this list re-packs live stores without refitting +# PSFs; reclaimed stores cannot supply additional products (see exposure.smk). # -# NOTE ON products_dir DEFAULTING TO run_dir (a fixture or smoke test): the tar -# then lands beside the store on the same filesystem and buys nothing, and -# final_cat and the persist manifests sit inside the trees clean_tile and -# clean_exposure delete. A one-root run is for fixtures and smoke tests only and -# must set clean: false and clean_tiles: false; the Snakefile refuses it -# otherwise. +# @sc [label:hazard] shared-run-products-root-requires-no-clean +# A one-root fixture (products_dir == run_dir) must disable both clean flags: +# otherwise cleanup deletes persistent products. The Snakefile enforces this. persist_exp: - psf_model @@ -205,20 +128,17 @@ clean_tiles: true # Their exposures are rebuilt from scratch if the tile is retried. clean_ignore_tiles: [] -# The ceiling on any rule's mem_mb: nibi's, whose standard compute node is -# 766 GB (192 cores at 4 GB/core), so 750000 leaves room for the OS and slurm's -# own overhead. A candide run never reaches it. A request above the partition -# maximum is a job SLURM never schedules and snakemake never diagnoses: it sits -# PENDING while the campaign looks alive. The two -# campaign-level merges size themselves from the campaign's bytes and will cross -# this at survey scale — the cap turns "never runs" into "runs on the biggest -# node there is", with a parse-time warning saying which rule was capped. +# Memory-request ceiling: 750000 MB leaves OS/SLURM headroom on nibi's +# 766 GB standard nodes. An above-partition request stays PENDING indefinitely. +# Snakefile::capped_mem limits merge requests and warns when sizing exceeds the +# cap; it does not guarantee that the job fits in the capped memory. max_mem_mb: 750000 # ngmix chunks per tile. ngmix_chunks: 8 -# Node-local tile store (vignets, WCS). Unset: the profile's /local/scratch -# bind (nibi NVMe, candide node /tmp). Dense image-sim campaigns on candide -# want a shared directory instead. +# Tile store (vignets, WCS). Unset: the profile's /local/scratch bind +# (nibi NVMe, candide node /tmp). Dense image-sim campaigns on candide can use +# a larger shared directory, trading ngmix read latency for capacity. sp rewrites +# the snapshot's bind source, not the tile path; see workflow/bin/sp. # tile_store_root: /n23data1//shapepipe/tilestore diff --git a/workflow/config/cfis/config_MCCD.ini b/workflow/config/cfis/config_MCCD.ini index a195d5241..64c33b504 100644 --- a/workflow/config/cfis/config_MCCD.ini +++ b/workflow/config/cfis/config_MCCD.ini @@ -71,7 +71,7 @@ GLOBAL_POL_INTERP = False # stars will be rejected. ie 9 is a conservative rejection. # USE_SNR_WEIGHTS : Boolean to determine if the SNR weighting strategy will # be used. -# For now, it needs the SNR estimations from SExtractor. +# Requires the SNR estimates from SExtractor. # PREPROCESSED_OUTPUT_DIR : (Required) Must be a valid directory to write the # preprocessed input files. # OUTPUT_DIR : (Required) Must be a valid directory to write the output files. diff --git a/workflow/config/cfis/config_exp_mccd.ini b/workflow/config/cfis/config_exp_mccd.ini index 3e1edfcdc..cdfdd9de9 100644 --- a/workflow/config/cfis/config_exp_mccd.ini +++ b/workflow/config/cfis/config_exp_mccd.ini @@ -234,8 +234,7 @@ NUMBERING_SCHEME = -0000000 [MCCD_PLOTS_RUNNER] -# Now MCCD has created a focal-plane PSF model, including all CCDS per images, -# thus single-exposure files +# The focal-plane MCCD model combines all CCDs into one file per exposure. NUMBERING_SCHEME = -0000000 # PSF model type read by mccd_plots_runner; the other option is "psfex" diff --git a/workflow/config/cfis/config_tile_Mc.ini b/workflow/config/cfis/config_tile_Mc.ini index 695654523..18429216c 100644 --- a/workflow/config/cfis/config_tile_Mc.ini +++ b/workflow/config/cfis/config_tile_Mc.ini @@ -83,7 +83,7 @@ SHAPE_MEASUREMENT_TYPE = ngmix # Save per-epoch PSF shapes and epoch identity (HSM_*_PSF_n, EXP_ID_n, # CCD_n). The campaign merge needs one column schema across all tiles, so # the slot count is fixed rather than the sexcat's geometric max(N_EPOCH)+1. It must -# exceed the survey's maximum epochs per object with margin (smk-g7: max 6 +# exceed the survey's maximum epochs per object with margin (maximum 6 # over 8 inspected tiles); an object with more epochs than slots is an error. SAVE_PSF_DATA = True N_EPOCH_SLOTS = 12 @@ -91,7 +91,7 @@ N_EPOCH_SLOTS = 12 # The UNIONS sky-fixed masks, queried once per object and written as one # MASK_