[OwenPriceSkelly]

Problem

rootstock smoke-test accumulated results in memory and wrote them to manifest.json only once, after the last checkpoint and the custom-weights legs. Two 12 h Delta nightly runs on 2026-08-13 (SLURM 21119813, 21119858) hit their wall-time limit mid-run and left nothing: no stamps on disk, no push seen by the dashboard.

Change

• Each checkpoint's outcome (its VerificationRecord plus its weight-file capture) is written to {root}/manifest.json as soon as it is known, in a short locked read-modify-write (_persist_outcome). A run killed by the scheduler keeps every verified/failed stamp it reached, and they ride along with the next successful push from that install.
• The custom-weights legs persist the same way. They still take passed_keys, baselines and base_captures from the in-memory main loop; that logic is untouched.
• The batched end-of-run apply is removed. The end-of-run update_and_push_manifest remains the single push.
• New optional --push-every N: also push after every N recorded results, to bound how much a killed run leaves unpushed. Default 0 keeps today's one push. --no-push suppresses every push, periodic ones included. The nightly recipes are unchanged.
• JSON/text output, --no-push, checkpoint-first selection and the shared-install clusters projection (schema v6) are unchanged.

Tests

New tests in tests/cli/test_smoke_test.py simulate a wall-time kill by raising a BaseException from the verify stub mid-run and assert the manifest on disk already holds the earlier stamps (canonical loop, weight files, and custom legs), that a concurrent manifest write made during a verification survives, that the default run pushes exactly once, and the --push-every / --no-push interaction.

uv run pytest      # 985 passed, 3 skipped
uv run ty check    # clean
uv run ruff check / format --check   # clean

Docs

docs/nightly-smoke-testing.md now describes when the manifest is written versus pushed, what a wall-time kill leaves behind, and --push-every. CLAUDE.md usage line updated.

🤖 Generated with Claude Code