ComfyUI Base 3.22.0: the cuBLAS probe beside the venv build; the install's wall-clock timeline

[Soul-Craft]

Why

Paul's priority 3, install speed. The last Video Replacer pod (yswh3tkypxhxev, 1x B200 EU-RO-1, base 3.20.0, every row: 42 files, 295.5 GB) took about 5:17 from create to the App on its fresh attempt. Its 16:55 figure includes a failed first attempt and about 11 minutes with the pod stopped. Two findings fed this release:

• VENV (116 s) held an apt install of cuda-libraries-dev-13-0 (33 packages, 2,251 MB fetched one file at a time, 4,146 MiB unpacked). That install ran although nothing built from source (PKG_NO_SAGE=1).
• The stage clock has seconds only, so what ran beside what (the critical path) was inferred, not read.

What

1. The cuBLAS probe runs beside the venv build. _base_cuda_toolchain_start starts in the background next to the models and tools jobs. _base_cuda_toolchain_join runs before pkg_post_venv, the first place a build can start, so every build still finds the toolkit fixed or is told why not. The venv's own install runs before the probe in either order. A run that ends early waits for the probe and never kills it: an apt stopped halfway leaves dpkg needing dpkg --configure -a. A dry run, runtime mode and BASE_CUDA_FOREGROUND=1 keep the old order. The clock gains CUDA TOOLCHAIN (in the background, beside the venv build).
2. state/logs/<pkg>_<ts>.timeline.tsv sits beside timings.tsv, which is unchanged. Columns are start TAB end TAB seconds TAB what in epoch seconds, earliest first. It covers every stage and every background stage (SYSTEM, MODELS, TOOLS VENV, CUDA TOOLCHAIN) from its start to its end, and every model file twice: download <rel> and sha256 <rel> (the re-read at the job's end). Each file's console line now ends in (sha256 Ns of it).
3. podctl PODCTL_TIMES=1 stamps each line [HH:MM:SSZ +N.Ns]. It is off by default, so parsers read podctl's lines as before.

Tests

suite/test_critical_path.py (unit tier) covers:

• the probe runs in the background, and its verdict, output and warning come back at the join;
• the foreground forms (BASE_CUDA_FOREGROUND=1, a dry run);
• a probe that died without a verdict is asked again;
• an early exit waits for the probe and never kills it;
• the pipeline's order;
• the timeline's order and format;
• the per-file spans from the serial path, the parallel path and the early pass;
• podctl's stamp.

Nothing ran on the Mac except bash -n, py_compile and package.py --base / --check (house rule: suites run in CI only).

Exercised on

No host. Not run on hardware: the next RunPod pod measures it.

Measurement plan (pod-runner, the D pass)

1. Pin base-v3.22.0 (one more offline dry install). Export PODCTL_TIMES=1 for podctl install. install_timings.sh copies state/logs/*.timeline.tsv.
2. Use one fresh pod and one clean install. A failed attempt restarts the clock at the next create.
3. D runs with BASE_STAGED=1 and PKG_LANES set but empty (the orchestrator's yes). Only the two B300-only files (87.2 GB, the 66.3 GB long pole of the last pod) arrive after the server.
4. Report:
   • create to App;
   • the critical path, read from timeline.tsv;
   • the VENV seconds and the CUDA TOOLCHAIN row;
   • each file's download and sha256 seconds;
   • the Mac's 46 s, split by the stamped lines.
5. Pass marks: VENV at least 25 s under 116 s on a pod whose image lacks cuBLAS; create to App at or under 4:15 on a cached B200.

The ranked levers (GHCR image and network volume not recommended, nothing paid) and their first-party sources are in the orchestrator's notes: install-speed-ranked.md and install-speed-sources.md.

🤖 Generated with Claude Code