Repository navigation
smoke-test recipes: one-shot RESCHEDULE_BEGIN, sync-style HF token lookup - #228
Merged
Merged
Conversation
…okup Port two fixes from the deployed Delta/Perlmutter copies (patched 2026-09-11) into the generic SLURM recipe, plus the token fix into the PBS recipe. 1. RESCHEDULE_BEGIN propagation (SLURM only). The in-job resubmit used bare `sbatch --begin=...`, which defaults to --export=ALL, so the documented test kickoff `RESCHEDULE_BEGIN=now+3minutes sbatch ...` rode down the chain and made every generation reschedule 3 minutes ahead forever. Resubmit with `env -u RESCHEDULE_BEGIN` so the next generation recomputes from CADENCE_DAYS/RUN_HOUR, which still persist. The PBS recipe already passes only SELF_PATH/CLUSTER via `qsub -v`, so it is unaffected. 2. HF token lookup (both recipes). Only ~/.cache/huggingface/token was read. Match the sync scripts: respect an existing HF_TOKEN, else the first readable of HF_TOKEN_FILE, ~/.hf_token, ~/.cache/huggingface/token (whitespace stripped); export HUGGING_FACE_HUB_TOKEN too; warn if none. Docs: note the override is one-shot and document the token knobs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports two fixes from the deployed Delta/Perlmutter
.sbatchcopies (patched 2026-09-11) into the generic recipes inscripts/.1.
RESCHEDULE_BEGINrode down the chain (SLURM)The in-job resubmit was a bare
sbatch --begin="$RESCHEDULE_BEGIN" --parsable "$SELF_PATH".sbatchdefaults to--export=ALL, so the test kickoff the header and docs recommend (RESCHEDULE_BEGIN=now+3minutes sbatch ...) propagated to generation 2, which then rescheduled 3 minutes ahead, and so on forever.Fix:
env -u RESCHEDULE_BEGIN sbatch ...so the next generation recomputes fromCADENCE_DAYS/RUN_HOUR. Those two knobs still persist via--export=ALL, as intended.The PBS recipe already passes only
SELF_PATH/CLUSTERviaqsub -v, soRESCHEDULE_ATnever propagated there. Left alone.2. HF token lookup (both recipes)
Both recipes read only
~/.cache/huggingface/token. They now match the sync scripts' order: an existingHF_TOKENenv var wins; otherwise the first readable of$HF_TOKEN_FILE,~/.hf_token,~/.cache/huggingface/token(whitespace stripped withtr -d '[:space:]').HUGGING_FACE_HUB_TOKENis exported alongside; a warning is printed if nothing is found.Docs
docs/nightly-smoke-testing.md: the testing section now says the override is one-shot and why (SLURMenv -uvs PBSqsub -vsemantics); the knobs table gainsHF_TOKEN/HF_TOKEN_FILEwith a short paragraph on the lookup order.Verification
bash -npasses on both recipes. Behavioural change already exercised by the patched Delta/Perlmutter copies.🤖 Generated with Claude Code