Keep exposure stores node-local in single-allocation runs; persist only per-exposure products #953

On Nibi a production batch is now one 144-core allocation running about 25 tiles (sp run --in-allocation). Its tile work is already node-local (#947 and the node-local tile stores). Its exposure stores are not: each batch builds about 43 of them on /scratch. Each node has 3 TB of local disk, so the exposure stores should live there too, and only the per-exposure products the catalogue needs should be persisted.

What the first production wave measured (smk-b01–b05, 10 Oct)

• Scratch bytes are what binds. With wave 1's stores live, /scratch reached ~1.9 TiB against Nibi's 1 TiB per-user soft quota, and it dropped back to 783 GiB as batches finished. The scale is ~6.6 GiB per exposure store, ~300 GiB per batch. Above the soft quota a 60-day grace counter runs, and it resets only when usage drops back below 1 TiB. So on /scratch we can run only 2–3 batches at a time, or keep relying on the grace period.
• File count does not bind at this scale. It is ~1,000 files per exposure store, ~50K per batch.
• Boundary recompute is real but cheap. The five batches build 214 exposure stores for 156 distinct exposures, so 27% are recomputed at batch edges. The exposure chain is ~1 of the ~12 billed core-h per tile (g12: 0.99), so the recompute costs about 2% of the total (inferred from g12's split).

Proposal

In --in-allocation mode, put the exposure stores on node-local disk ($SLURM_TMPDIR), next to the tile stores, and persist to the products directory only what downstream reads: the per-exposure PSF tars and the exposure-map fragments.

• A batch's /scratch footprint goes to roughly its products, so the soft quota stops limiting how many batches run at once. It is then limited by the scheduler and fairshare.
• It makes Pack per-exposure intermediates so a Nibi campaign fits the 1M-file quota #951 (packing per-exposure intermediates for the 1M-file quota) unnecessary for single-allocation runs. Pack per-exposure intermediates so a Nibi campaign fits the 1M-file quota #951 still matters for the multi-job Slurm-executor launch.
• I/O moves off shared NFS. This is the same effect tile_detect: build the catalogue in memory, run SExtractor on node-local disk #947 had for tile_detect: 8.7 → 1.0 min.

Not proposed: a cross-batch shared exposure store. It would save the ~2% recompute, but it would bring back a persistent shared store, with locking between concurrent batches and a bigger /scratch footprint. Ordering batches as compact 2D blocks gets most of that 2% for free.

What to check while implementing:

• Peak node-local use per batch. It should be ~300 GiB stores plus tile stores, well under 3 TB, but measure it on a dense batch.
• That a failed batch can be resumed. With node-local stores it re-runs from the start: ~2 h, which is acceptable.
• That nothing downstream (the star catalogue merge, the coverage and defect maps, sp_check_products.py) reads an exposure store directly rather than a persisted product.

Claude (Opus, nibi chair) on behalf of Cail. Measurements by the nibi surveyor session from the smk-b01–b05 ledger.