Exposure maps: npoint (pointing squares) and ntaken beside nexp

[cailmdaley]

Claude (Opus) on behalf of Cail.

Each campaign now publishes four exposure-count maps instead of two, all at the mask ladder's nside 131072:

map	counts, per sky pixel, the exposures…
npoint_<run>.hsp (new)	whose pointing square contains it: the convex hull of CCDs 0–35, so chip gaps are filled and the four ear CCDs 36–39 are left out
ntaken_<run>.hsp (new)	with any CCD's imaging area (DATASEC, all 40 CCDs) on it, whether or not the PSF validated
nexp_<run>.hsp	with a PSF-validated CCD on it (unchanged)
nflagged_<run>.hsp	flagged CCD pixels of those validated CCDs (unchanged)

Why. nexp follows the real CCD tiling, so with chip gaps, ears and dithers it comes out as a busy pattern, and the counts pile up where two exposures' ears overlap. npoint is a deliberately smooth, somewhat arbitrary pointing count: every exposure is one filled square. It is the v2 counterpart of v1.X's npoint3 >= 3 cut. sp_validation's mask_v2.0.yaml says v2 has no source for that cut yet, and this map provides one. ntaken alongside nexp gives taken vs successful: ntaken − nexp is the coverage lost to CCDs whose PSF failed validation.

On 44 smk-b05 exposures, npoint is 3 almost everywhere, with seams where neighbouring squares overlap. ntaken and nexp carry the CCD pattern. ntaken − nexp picks out exactly the five CCD footprints that failed PSF validation in that region.

What changed

• exp_maps.py. The per-exposure fragment is now a packed uint16. Bits 0–2 are coverage (1 pointing square, 2 CCD imaging area, 4 validated CCD) and bits 3–10 are the flagged-pixel count, clipped at 255. A HealSparse record-array map was the obvious alternative, but HealSparse stores record arrays uncompressed (33 MB per exposure). Packed fragments are 2.55 MB median, against 2.13 MB for the uint8 fragments they replace.
• Two input sources, one rasteriser. The headers and flag arrays can come from the split scratch store (--exp-dir, what the rule uses) or from the delivered MEFs (--image-mef/--flag-mef, HDU = CCD + 1). split_exp keeps the delivered header's WCS, so the two agree. On exposure 2086785 they gave identical fragments over all 5,373,752 pixels.
• merge_exposure_maps.py sums the fragments into the four maps. It fails, naming the exposure and the backfill command, if it meets an old-format fragment, and it fails if a count would overflow its dtype.
• Wiring. Rule outputs, exposure_maps_targets, README, astra.yaml (new decision masking.pointing_square, and nexp_map_valid_psf_ccds extended to cover ntaken), universes/committed.yaml, and the params pin (only the exposure_maps shell digest moved).

Backfilling existing campaigns

Fragments that existing campaigns already wrote are in the old format, and their scratch stores have been reclaimed. Rerunning exp_maps.py in raw-MEF mode on the staged exposures rewrites the fragment and its exp_maps.json, and snakemake then reads the fragment through durable_edge as before. This takes ~14 s per exposure. On Nibi that is about 5k fragments, so one small job.

Verification

• Not-slow suite: 967 passed, 4 skipped. tests/unit/test_exp_maps.py, test_decisions.py and tests/workflow: all pass. astra-tools validate: pass.
• Unit fixture: a CCD grid with gaps, an ear and one unvalidated CCD. The tests check that the gaps count in npoint and not ntaken, the ear in ntaken and not npoint, the unvalidated CCD in ntaken and not nexp, that split and MEF sources give identical fragments, and that the merge rejects an old fragment.
• Real data: all 44 smk-b05 exposures were rebuilt from raw MEFs. nexp and nflagged are bit-identical to production over the whole campaign footprint.
• Independent review (GPT-6.1 Sol): nothing blocking. The pointing polygon matches analytic tangent-plane coverage across RA = 0/360, at Dec 80° and 89.8°, at both poles and in both WCS handednesses. A reclaimed-exposure DAG probe schedules no exp_maps rebuilds and targets all four maps.

#960 (comment sweep) touches the same two scripts and will need a small merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FD35GJ3MDN7ceDhFwxzMbU