Repository navigation
sp run --in-allocation: run a campaign inside one SLURM allocation - #964
Open
cailmdaley wants to merge 3 commits into
Open
cailmdaley wants to merge 3 commits into
cailmdaley wants to merge 3 commits into
Conversation
Runs both invocations under snakemake's local executor inside the allocation sp is called from, so a small campaign pays one queue wait instead of one per job. sp derives --cores from the cpuset and --resources mem_mb from the tightest cgroup memory limit (less a reserve), refuses to run outside an allocation, and defaults the node-local tile store to $SLURM_TMPDIR/tile-store through the existing tile_store_root rebind. README documents sizing and the resume caveat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz
… per allocation Data campaigns name node-local stores by bare tile id, so two campaigns over the same tiles sharing one allocation would share stores. Default to $SLURM_TMPDIR/tile-store/<run>, and document one srun step per campaign. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz
A job cut short by its time limit leaves two things a plain resubmission trips over. Its node-local tile store is gone, so a tile whose tile_vignets.json survived without a final_cat stops at tile.smk's "vignette store is missing" guard; and a killed snakemake leaves its lock. sp now deletes those manifests before the run, and records the job that holds the state dir so that a lock whose job has left the queue is released. Any other lock stops the run before anything is swept. With -n both only report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sp run --in-allocationruns a whole campaign inside the SLURM allocation it is called from, under snakemake's local executor, so a campaign waits in the queue once instead of once per job.--in-allocation--executor local --cores N --resources mem_mb=M, with N from the cpuset and M from the tightest cgroup memory limit less a reserve; later explicit flags win. Refuses to run outside an allocation.$SLURM_TMPDIR/tile-store/<run>, so two campaigns can share one allocation (onesrunstep each);tile_store_root:still winstile_vignets.jsonfor tiles with nofinal_cat(their node-local store died with the job) and releases a snakemake lock whose job has left the queue; any other lock stops the run.-nonly reports.workflow/README.mdgains an "Execution mode: one allocation" section covering sizing (one node, ≤4 GB/core, the widest rule's last retry must fit) and the resume path.Why now
Every v2 production batch on Nibi already runs through this branch: the integration branches g15 through g18 carry it, and g18 alone ran 33 one-node batches (~800 tiles, P3) with no run failures. It is the base of #954 (exposure stores node-local under
--in-allocation, #953). This PR lands the mechanism production depends on, so that it stops being campaign-local.Scope
workflow/bin/sp,workflow/README.md.--in-allocationis opt-in.Verification
sp run --in-allocationon one 144-core node; g18's 33 batches (~800 tiles) had no run failures (surveyor's report, 11 Oct).-nreports the sweep and lock release without acting.Stacked after this: #954 → base moves to
developonce this merges.Claude (Opus 5.5) on behalf of Cail, from the Nibi scale-up.
🤖 Generated with Claude Code
https://claude.ai/code/session_01SYQ7H2aF2jXdXv9FSKyagY