Soul-Craft · Oct 17, 2026 · Oct 17, 2026 · Oct 17, 2026
@@ -154,6 +154,26 @@ start and the fake pod's boot; the fake RunPod for `--gpus 2`, the count query,
 - One shot rendered on server 1 is byte-identical in content to the same shot on server 0 (same seed).
 - Host RAM with both servers loaded (the 4/8 rows of `lowestPrice` once they have stock).
 
+## 8. The cuBLAS probe beside the venv build; the install's timeline (3.22.0)   (2026-10-17, no pod used)
+
+**Why.** Paul's priority 3, install speed. The last Video Replacer pod (yswh3tkypxhxev, 1x B200, base 3.20.0, every row:
+42 files, 295.5 GB) took about 5:17 from create to the App on its fresh attempt (its 16:55 includes a failed first
+attempt and the pod stopped for about 11 minutes). The long pole was the 66.3 GB B300-only file (147 s with its sha256),
+with VENV (116 s) beside it, and VENV held an apt install of `cuda-libraries-dev-13-0` (2,251 MB, 33 packages) on an
+install that built nothing. The stage clock had seconds only, so what ran beside what was inferred, not read.
+
+**What changed.** `lib/40-packs.sh`: `_base_cuda_toolchain_start` / `_join` / `_stop`; `lib/95-summary.sh` starts the
+probe with the venv and joins it before `pkg_post_venv`. `lib/00-env.sh`: `_base_span`, `BASE_SPANS`, `BASE_FILE_SPANS`,
+`BASE_EPOCH0`, `BASE_CUDA_FOREGROUND`; `lib/92-onepass.sh`: `_base_timeline_write` (`<pkg>_<ts>.timeline.tsv`);
+`lib/50-models.sh`: each download's `BASE_VERIFY_SECS` and spans; the background joins (system, models, tools) add
+theirs. `_build/pod/podctl.py`: `PODCTL_TIMES=1`.
+
+**Proven, and how.** `suite/test_critical_path.py` in CI (nothing ran on this Mac but `bash -n` and the packager).
+
+**Owed (the next pod).** The VENV stage without the apt, the `CUDA TOOLCHAIN (in the background…)` row's seconds, and the
+timeline's critical path; the measurement plan is in the PR. The ranked levers and the sources are in the orchestrator's
+notes (`install-speed-ranked.md`, `install-speed-sources.md`).
+
 ## Requests
 
 ### 1. To the MiniMax H3 Video Replacer: declare the app's port

@@ -1,30 +1,30 @@
 af891a1c6f3f3958b43ea1f97a079ff898394ac72b2f49f88f3db66276f94eaf  base.sh
 577a83f8e3e3c4d69e04f4cbcee7ead42c49981c76f9bb07f8f78fd076876e01  comfyui-base-script.sh
-17e09f0c32655044fdc038fb046f87a5d0548ed9601f59fcb97fb3ea379c6d30  comfyui-base-handbook.md
-6ed25779fb19fdab1aa720b59c911750e3c4c9208e0496bb72e2605302947616  VERSION
+0052e9272c027a271b54c19b6aad625e3da0b087e768a0a1acfad76103ab1ada  comfyui-base-handbook.md
+46058ff89ee6dab89b7f99a6a21b117815ecf0e1f3967e51802d462c95f332fb  VERSION
 939c10b34e782bb77792eb876d8a51d74b38192598483c8c628f5da1b877397d  pytest.ini
 8aa67b7331d052dbbd1dcf9b357f088891e0fa0f4ed56e6669c105046fe881b2  LICENSE
-75cf532f7718f7b546d033810c2aa623abd788605c42df4fa25e5f57108dddb3  lib/00-env.sh
+3dcc371917e882b798f0f326a61d0fc3e1b6d65a799adba36344371d26568405  lib/00-env.sh
 bb3428df51590d49871f14f6bf1badaa66ce3627bbb0181516b8fa14ce040e2d  lib/10-discover.sh
-4f477b9ce63fd1d7388f79acc32eb040b561a37db4c06f9bcf325245c29fe0ff  lib/15-system.sh
+d642ea2ae686b78f4596f3ebb81a2ebdc751e84d7e580fb931b101d40bdc5084  lib/15-system.sh
 2f16862a9c5f675dd1d20e8b56263832e1cf991e4cb1a05e1fd00bec2231158b  lib/20-comfyui.sh
 ba32023a2c0319c5675d37f10ff30874462f2a45ce78323e1c1f714f12933c97  lib/30-venv.sh
-60c559d7c7e89b5966154575dead1969392f1861b0b872d2143bb631d8ed3d92  lib/40-packs.sh
+72ca6f74f967828dbb5f44087a1689186bdf5c347ae58dc3fa18b19a94a4f97b  lib/40-packs.sh
 5a410ec6f4d7790ec57c07296c1d911cac729c01947985cdc8816e63873d3f7a  lib/45-mcp.sh
 19aca634d77a27185efa3aaf4b2bfdab1a21af82334acb18c94a9aab5be241c4  lib/47-latest.sh
 db941b718ddf05903b0dc867e2a55378f96fcda776b2aa359f21c2c929a8d390  lib/48-runtime.sh
-f2eeaade67ddc4118e0950cb09a9067f730158f38fc148a1359f6ff51f9074f6  lib/50-models.sh
-5eb9452a949de071a61a9c81c047a7560b86cfdd4ced26ecb1eca24111bb9b51  lib/52-models-early.sh
+6f4ebe61c63d143a7e24228d7ec9ef4313b76c87a278a575ccffd6fb63bb4ebd  lib/50-models.sh
+4bae452bbac2cf9d2a5bbd31819942e565b8eab367636ab91795911fee10c82b  lib/52-models-early.sh
 89fd610570781905c0793838ce76e460b1c6b7c8ef25e18c478115fd345251c7  lib/60-sync.sh
 432bebc4ebe0675db14c6d8875327bae1cc0b5f9837b0c181d32bdb8f5c76171  lib/62-own-packs.sh
 2feed9211c92aab85499c9f30ab5de5ecebc86af3956e48d0bf4bc57540c51ab  lib/70-hygiene.sh
-e42674d02fa4b857bbd25119efeac27b0496e83e30522685604aeea7e41018f6  lib/75-boot.sh
+e9e64c96ab5df20fee132a50b84eac257bc8270dd429d86df9a16b2aed39e377  lib/75-boot.sh
 141ba2100e2f921e80053742ce3dae4b6d86cdcd60109cc095242453812d988f  lib/80-server.sh
 d5aee6a3e6d71ad9fabc5683bd9d1d592f252d63046118e53b29716f0ef66965  lib/84-run-local.sh
 139ad7df5650f6bd237416eaae0a2cdf28709b11f1c9dc08d47610e07f9b3557  lib/85-launch.sh
 9eda7369f29fc5e4bcc7f56ad64c3b5879fdd2844306dbdf463615fed49af46a  lib/90-ledger.sh
-917f38ec2418c9c4923ca59b815b013939601c6cf46a8d016b36889e4f0089ff  lib/92-onepass.sh
-25f40ce1f0acde64e1ad024c93e3ebed092480be88ae65ca75f72f1f30f023ee  lib/95-summary.sh
+9bb5ad4f2e909110c7030eb338fff8630f54e62b95ada99b47507f0395f95db7  lib/92-onepass.sh
+1a46cb09b8d0bd334a9c2e92f86ba45e6ab91ebdb27fc7517947624e41a8134d  lib/95-summary.sh
 82943bbc7c31835bf997afb70e41ba9c56ddaf82712a3348537e60f911c2fb6a  lib/boot.sh
 3eee2b5d5bafb0c1501aa2183dcc33367e4662a6f9717348a0d30f11378939b6  py/app_proxy.py
 58a84a14ec739aeadef47b83e8b30827713b2c21c0f690d3d3ce981c64894a9c  py/arch_fit.py
@@ -59,6 +59,7 @@ fcd96d1acae7a2761983039fb6a326cbc31711a62ac4f5d0509f2fe124b12246  py/venv_probe.
 ff2a64b4475aca687b6a98625c1d7eb328ee54052e02db70b8a88607250a5769  suite/test_bare.py
 2c72647bfba5cd37297b8a253aba512e95db5a672bf30611acc3c6f181026165  suite/test_canvas.py
 3d90e87daca9797667acb72644e98cfcfa39bcc37a80fbbac19e696b18de6717  suite/test_comfyui.py
+78584c505587b78a44d71dbc3662a60bd57cdde8be91f068261c07f626fa0b42  suite/test_critical_path.py
 1dccf801f852fd24988da239155828025382cc5aeb191e5f3446c362647bc470  suite/test_crusoe.py
 6907c8283d0332361b63c525dcd664a82fa2f0640fcd6f8fcfa7d8d7c98d614c  suite/test_derive.py
 2a1cd2075a4b9197bc4655bb6849cce36c7c56014245fcdaa1ebe19f0ca728e6  suite/test_hashes.py

@@ -1 +1 @@
-3.21.0
+3.22.0
@@ -729,7 +729,9 @@ def item_for_zip(zip_path, logs=None):
 class PodIO:
     """The real pod: ssh (no forwards), scp, the packager. `say` prints progress lines."""
     def __init__(self, api, pod, env=None, packager=None):
+        import time
         self.api, self.pod, self.env = api, pod, env or os.environ
+        self._t0 = time.monotonic()        # 3.22.0: PODCTL_TIMES=1 stamps each line from here
         self.provider = api if isinstance(api, Provider) else RunPodProvider(api)      # an Api is RunPod's; a provider is any host's
         self.packager = packager or (pathlib.Path(__file__).resolve().parents[1] / "package.py")
     def check_zip(self, d):
@@ -869,7 +871,17 @@ def sleep(self, s):
         import time
         time.sleep(s)
     def say(self, *a):
-        print(*a, flush=True)
+        print(*(_stamp(self.env, getattr(self, "_t0", None)) + list(a)), flush=True)
+
+
+def _stamp(env, t0):
+    """3.22.0: PODCTL_TIMES=1 → ["[15:00:06Z +46.2s]"] (UTC wall time and seconds since the PodIO was made), so the Mac's
+    half of an install (lease, uploads, preflight, the poll that sees STEP RC) lines up with the pod's own timeline.tsv;
+    otherwise []. Off by default: every caller that parses podctl's lines reads them unchanged."""
+    import datetime, time
+    if (env or {}).get("PODCTL_TIMES") != "1" or t0 is None:
+        return []
+    return ["[%sZ +%.1fs]" % (datetime.datetime.now(datetime.timezone.utc).strftime("%H:%M:%S"), time.monotonic() - t0)]
 
 
 def _tail(text, n=25):

@@ -1,6 +1,6 @@
 # ComfyUI Base — Handbook
 
-**Version 3.21.0.** The shared toolchain every workflow package on a machine sources, on **any host with an NVIDIA
+**Version 3.22.0.** The shared toolchain every workflow package on a machine sources, on **any host with an NVIDIA
 GPU** (RunPod, Verda, Crusoe, an owned box) and on any image or OS that gives it a driver and Python 3. One command,
 run once per machine as **step one**; then each workflow package is **step two**, still one command. From a Mac,
 `podctl` reaches the machine and hands its boot to the base (§1); after that every boot is the base's.
@@ -628,7 +628,8 @@ that does not need another's output runs beside it, and a join stands only where
            ◄────────────────────────────────────────────────────┘
            ├─ model downloads (BASE_FETCH_JOBS at once, largest first) ... background ─┐ joined before the model stage
            ├─ JupyterLab's tools venv .................................... background ─┼─┐ joined at BOOT
-           └─ venv snapshot → venv → toolkit → pkg_post_venv (Sage) → own packs        │ │
+           ├─ cuBLAS probe and its apt install (3.22.0) ................... background │ │ joined before pkg_post_venv
+           └─ venv snapshot → venv → pkg_post_venv (Sage) → own packs                  │ │
                 → packs pip (skipped when the venv build installed this exact set)     │ │
                 → Comfy MCP → newest → [local-runtime copy, when lib/84-run-local.sh is in] │
                 → pkg_pre_models ◄─────────────────────────────────────────────────────┘ │
@@ -645,7 +646,12 @@ tree. A store that already holds this exact install green (same package files, s
 place) runs none of it: `ALREADY INSTALLED`, ComfyUI started if nothing serves, done in seconds (`BASE_FULL=1` runs it all).
 
 **One pass, and its clock** (3.2.0). Every stage header closes the stage before it with its seconds; the summary prints
-them slowest first and `state/logs/<pkg>_<ts>.timings.tsv` keeps them, which podctl fetches with the console. When the
+them slowest first and `state/logs/<pkg>_<ts>.timings.tsv` keeps them, which podctl fetches with the console. 3.22.0:
+beside it, `<pkg>_<ts>.timeline.tsv` keeps WHEN: `start TAB end TAB seconds TAB what` in epoch seconds, earliest first,
+for every stage, every background stage (from its start to its end, not just its seconds) and every model file twice
+(`download <rel>`, the whole job, and `sha256 <rel>`, its re-read at the end), so the critical path of an install is read
+from the run; each file's line in the console also says how many of its seconds were the sha256. `PODCTL_TIMES=1` stamps
+podctl's own lines with the UTC time and the seconds since the install began, so the Mac's half lines up with it. When the
 only change a run makes is the package's own packs (ComfyUI, every other pack and the venv's package list unchanged), the
 import check loads ComfyUI with just those packs (`--disable-all-custom-nodes --whitelist-custom-nodes`); anything short
 of a pass there is judged again by the full check. `<script> preflight` is what podctl runs before an install: seconds,
@@ -831,6 +837,19 @@ for any invocation that somehow arrives without the environment. The suite asser
 
 ## 10. Record
 
+- 3.22.0: install speed on the critical path, and a clock that shows it (Paul's priority 3, 2026-10-17). Read from the
+  last Video Replacer pod (1x B200, EU-RO-1, base 3.20.0, every row: 42 files, 295.5 GB): about 5:17 from create to the
+  App for a fresh install, with the 66.3 GB B300-only file the long pole (147 s with its sha256) and VENV 116 s beside
+  it, `cuda-libraries-dev-13-0` installed inside VENV although nothing built (33 packages, 2,251 MB fetched one file at
+  a time, 4,146 MiB unpacked). (1) `base_cuda_toolchain` starts with the venv build in the background
+  (`_base_cuda_toolchain_start`, lib/40-packs.sh) and is joined before `pkg_post_venv`, the first place a build can start,
+  so every build still finds the toolkit fixed or is told why not; the clock shows `CUDA TOOLCHAIN (in the background,
+  beside the venv build)`. A dry run, runtime mode and `BASE_CUDA_FOREGROUND=1` keep the old order. (2)
+  `state/logs/<pkg>_<ts>.timeline.tsv` beside the timings (§5, "One pass, and its clock"): wall-clock spans for every
+  stage, background stage and model file, with each file's sha256 seconds in its console line too. (3) `PODCTL_TIMES=1`
+  stamps podctl's lines. The unit tier covers the background probe (its verdict, its warning, the foreground forms, a
+  probe that died, the pipeline's order), the timeline's order and format, the per-file spans from the serial, the
+  parallel and the early pass, and podctl's stamp. Not run on hardware: the next pod measures it.
 - 3.21.0: which test lane a smoke takes (2026-10-17). `py/gpu_fit.py` gains `LANES`, `fast_dtypes()` and
   `pick_lane(peak_gib, minutes, needs)`, which builds on `fits()` for GPU classes instead of a second rule, and a
   `--peak-gib/--minutes/--needs` mode on the command line (§1, "Which test lane"). The unit tier covers the order, the
@@ -1077,7 +1096,7 @@ for any invocation that somehow arrives without the environment. The suite asser
   one extension `core.py` imports unconditionally, and the Triton mode calls no qattn kernel, so that build takes about
   a minute; if its proof then finds Triton does not run, every kernel is built and proven again. The proof's mode is now
   also kept on the volume (`state/sage/<sm>.mode`), so it outlives a venv rebuild. The cuBLAS probe runs only before a
-  build. `SAGE_WHEELS=0` skips the download, `SAGE_FUSED_ONLY=0` the `_fused`-only build, `SAGE_WHEELS_URL` names
+  build (3.22.0: the install's own probe now runs beside the venv build and is joined before `pkg_post_venv`). `SAGE_WHEELS=0` skips the download, `SAGE_FUSED_ONLY=0` the `_fused`-only build, `SAGE_WHEELS_URL` names
   another releases root. The Sage step needs no new join with the early downloads: it runs in `pkg_post_venv`, inside
   their window. Exercised on this Mac against a file:// release mirror and upstream's real setup.py and core.py; the
   wheels are built by CI; the pod measurement is in the PR.

@@ -71,6 +71,7 @@
 #                        largest first; 1 is the one-at-a-time behaviour of 3.0 to 3.10.
 # BASE_MODELS_FOREGROUND=1  3.11.0: the model downloads wait for the venv build, as before (lib/52-models-early.sh)
 # BASE_TOOLS_FOREGROUND=1   3.11.0: JupyterLab's tools venv builds in the BOOT stage, as before (lib/75-boot.sh)
+# BASE_CUDA_FOREGROUND=1    3.22.0: the cuBLAS probe (and its apt install) runs after the venv, as before (lib/40-packs.sh)
 # BASE_PACK_JOBS=<n>     3.11.0: how many packs clone or fetch at once (default 8)
 # BASE_FULL=1            3.11.0: the whole pipeline even when this store already holds this exact install green
 #                        (lib/92-onepass.sh: same package files, same base, same GPU arch, every file in place)
@@ -133,9 +134,16 @@ hdr(){   _base_stage_mark "$*"; echo -e "\n${CYA}══ $* ══${NC}"; }
 BASE_STAGE_NAME=""; BASE_STAGE_T0=0; BASE_STAGE_TIMES=()
 _base_stage_mark(){
   local now="$SECONDS"
-  if [ -n "$BASE_STAGE_NAME" ]; then BASE_STAGE_TIMES+=("$(( now - BASE_STAGE_T0 ))|$BASE_STAGE_NAME"); fi
+  if [ -n "$BASE_STAGE_NAME" ]; then BASE_STAGE_TIMES+=("$(( now - BASE_STAGE_T0 ))|$BASE_STAGE_NAME"); _base_span "$BASE_STAGE_T0" "$now" "$BASE_STAGE_NAME"; fi
   BASE_STAGE_NAME="$1"; BASE_STAGE_T0="$now"
 }
+# 3.22.0: and WHEN each stage ran. Seconds alone cannot say what ran beside what, so the critical path of an install was
+# inferred. Every span (a stage, a background stage at its join, each model file's download and its sha256 re-read) keeps
+# its two ends as this shell's SECONDS (a background job's subshell counts on the same clock), and the summary writes them
+# as wall-clock times: state/logs/<pkg>_<ts>.timeline.tsv (lib/92-onepass.sh). BASE_EPOCH0 is the epoch SECONDS counts from.
+BASE_SPANS=(); BASE_FILE_SPANS=()   # the stages; each model file (download, then its sha256 re-read)
+BASE_EPOCH0="$(( ${EPOCHSECONDS:-$(date +%s)} - SECONDS ))"
+_base_span(){ BASE_SPANS+=("$1|$2|$3"); }   # <t0> <t1> <label>, both ends in SECONDS
 would(){ echo -e "${CYA}  → would $*${NC}"; }   # --check wording
 todo(){  echo -e "${CYA}  → $*${NC}"; }          # queued work, not a fault: this run is about to do it
 warn(){  echo -e "${YEL}  ! $*${NC}"; BASE_WARN+=("$*"); }   # reported in the summary, never blocks
@@ -265,6 +273,7 @@ _base_on_exit(){
   _base_install_unlock                               # 2.5.0: never leave a shared volume locked by a dead run
   ! declare -F _base_models_early_stop >/dev/null || _base_models_early_stop || true   # 3.11.0: early downloads end with the run that started them (they resume)
   ! declare -F _base_tools_stop >/dev/null || _base_tools_stop || true                 # 3.11.0: and so does the background tools venv
+  ! declare -F _base_cuda_toolchain_stop >/dev/null || _base_cuda_toolchain_stop || true   # 3.22.0: the background cuBLAS probe (lib/40-packs.sh)
   for d in ${BASE_TMPDIRS[@]+"${BASE_TMPDIRS[@]}"}; do if [ -n "$d" ] && [ -d "$d" ]; then rm -rf "$d"; fi; done
   if [ "$rc" -ne 0 ] && [ "$BASE_SUMMARY_DONE" = "0" ] && [ -n "$BASE_LOG" ]; then echo "  aborted (exit $rc) — log: $BASE_LOG"; fi
   [ "$BASE_LOGGING" = "1" ] && sleep 0.3   # let tee flush the last lines

@@ -208,7 +208,7 @@ _base_system_start(){ # → 0 when the OS stage is now running in the background
       printf 'BASE_SYSTEM_ROWS=('; for x in ${BASE_SYSTEM_ROWS[@]+"${BASE_SYSTEM_ROWS[@]}"}; do printf ' %q' "$x"; done; printf ' )\n'
       printf '_base_sys_failed=('; for x in ${BASE_FAILED[@]+"${BASE_FAILED[@]:$nf}"}; do printf ' %q' "$x"; done; printf ' )\n'
       printf '_base_sys_warn=('; for x in ${BASE_WARN[@]+"${BASE_WARN[@]:$nw}"}; do printf ' %q' "$x"; done; printf ' )\n'
-      printf '_base_sys_secs=%q\n_base_sys_rc=%q\n' "$(( SECONDS - t0 ))" "$rc"
+      printf '_base_sys_t0=%q\n_base_sys_secs=%q\n_base_sys_rc=%q\n' "$t0" "$(( SECONDS - t0 ))" "$rc"
     } > "$dir/vars.sh"
   ) > "$dir/out" 2>&1 &
   _BASE_SYSTEM_PID=$!
@@ -217,7 +217,7 @@ _base_system_start(){ # → 0 when the OS stage is now running in the background
 }
 _base_system_join(){ # → the OS stage's status (10: the machine must restart first); its output shown, its results in this run
   [ -n "$_BASE_SYSTEM_PID" ] || return 0
-  local _base_sys_failed=() _base_sys_warn=() _base_sys_secs=0 _base_sys_rc=0 x
+  local _base_sys_failed=() _base_sys_warn=() _base_sys_t0=0 _base_sys_secs=0 _base_sys_rc=0 x
   wait "$_BASE_SYSTEM_PID" 2>/dev/null || true
   _BASE_SYSTEM_PID=""
   cat "$_BASE_SYSTEM_DIR/out" 2>/dev/null || true
@@ -226,5 +226,6 @@ _base_system_join(){ # → the OS stage's status (10: the machine must restart f
   for x in ${_base_sys_failed[@]+"${_base_sys_failed[@]}"}; do BASE_FAILED+=("$x"); done
   for x in ${_base_sys_warn[@]+"${_base_sys_warn[@]}"}; do BASE_WARN+=("$x"); done
   BASE_STAGE_TIMES+=("$_base_sys_secs|SYSTEM (in the background)")
+  _base_span "$_base_sys_t0" "$(( _base_sys_t0 + _base_sys_secs ))" "SYSTEM (in the background)"
   return "$_base_sys_rc"
 }