Exposure stores node-local under sp run --in-allocation (#953)

[cailmdaley]

Closes #953. Stacked on feat/sp-in-allocation; review that branch first.

What this does

A production batch on Nibi runs ~25 tiles inside one SLURM allocation (sp run --in-allocation). Along the way it processes ~43 exposures. For each one it writes intermediate files: split CCD images, SExtractor catalogues and PSF models. That comes to ~7 GB and ~1,000 files per exposure, ~300 GB per batch, all on the shared /scratch filesystem. Once the tiles that use an exposure are done, these files are deleted. Only two small products are kept: a PSF tarball and an exposure-map fragment.

This PR writes those intermediate files to the compute node's local disk ($SLURM_TMPDIR, ~3 TB per node) instead. Tile intermediates already live there.

Why

• Fewer files and bytes on /scratch. A batch's peak /scratch use drops by ~300 GB and ~40K files, which leaves more room under the per-user quota for running several batches at once.
• Faster I/O. Exposure processing reads and writes many files. Local disk avoids the shared network filesystem: the same move cut tile_detect from 8.7 to 1.0 min (tile_detect: build the catalogue in memory, run SExtractor on node-local disk #947).

Nothing downstream needs these files after the batch ends. The persisted products are unchanged.

How

The bookkeeping stays put and only the bulk data moves. Each exposure directory on /scratch holds:

• manifests/, the "this step is done" records the workflow reads;
• logs/;
• output/, the bulk.

output/ becomes a symlink to a directory on local disk; everything else stays on /scratch. The workflow's logic is therefore untouched: it sees the same paths and the same manifests.

Where	Change
workflow/bin/sp	Under --in-allocation, picks a local directory ($SLURM_TMPDIR/exp-store/<run>) and passes it to the jobs as SP_EXP_STORE.
Snakefile (unit_pre)	The first step of each exposure creates the symlink. Without SP_EXP_STORE, nothing changes.
clean_exposure.py	When cleaning an exposure, also deletes the local directory the symlink points to. It only deletes directories inside SP_EXP_STORE.
sp (alloc_resume)	On a restart in a new allocation, the old local disk is gone. Symlinks pointing at nothing are removed along with their manifests, so those exposures rerun.

What to check when reviewing

• Resume is safe. An exposure whose data was lost with its node gets recomputed, never read as done.
• Nothing reads output/ after the exposure is cleaned. The PSF tarball and exposure maps are packed before cleaning, as before.

Not yet done

• No production batch has run on this yet. The first batch of the next wave will measure peak local-disk use (expected ~300 GB plus tile stores, against 3 TB).
• The exposure jobs' setup text changed. tests/workflow/params_pin.json needs regenerating (--update-params-pin). A campaign already in flight will rerun its unfinished exposures once.

Tested by hand in a scratch directory: the symlink is created, and cleaning removes both the link and the local data.

Claude (Opus, nibi surveyor) on behalf of Cail.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PuzG3FLs6mDQuXqzSwRAnz