From 66bed7928e4bd82ab49daac001f8b9668047c16e Mon Sep 17 00:00:00 2001 From: Paul Foley <8870031+Soul-Craft@users.noreply.github.com> Date: Sat, 17 Oct 2026 15:52:14 -0600 Subject: [PATCH 1/2] ComfyUI Base 3.22.0: the cuBLAS probe beside the venv build; the install's wall-clock timeline The last Video Replacer pod (1x B200, base 3.20.0, every row) spent 116 s in VENV, an apt install of cuda-libraries-dev-13-0 (2,251 MB, 33 packages) inside it although nothing built from source, and its stage clock could not say what ran beside what. - base_cuda_toolchain starts with the venv build in the background and is joined before pkg_post_venv, the first place a build can start; a run that ends early waits for it (never kills apt). A dry run, runtime mode and BASE_CUDA_FOREGROUND=1 keep the old order. - state/logs/_.timeline.tsv beside timings.tsv: wall-clock spans for every stage, background stage and model file (download, then its sha256 re-read); each file's console line names its sha256 seconds. - podctl: PODCTL_TIMES=1 stamps each line with UTC time and seconds since the install began. suite/test_critical_path.py (unit). Not run on hardware. Co-Authored-By: Claude Opus 5.5 --- HANDOFF.md | 20 +++++ MANIFEST.sha256 | 21 ++--- VERSION | 2 +- _build/pod/podctl.py | 14 ++- comfyui-base-handbook.md | 25 +++++- comfyui-base.zip | Bin 731327 -> 738813 bytes lib/00-env.sh | 11 ++- lib/15-system.sh | 5 +- lib/40-packs.sh | 50 +++++++++++ lib/50-models.sh | 42 ++++++--- lib/52-models-early.sh | 9 +- lib/75-boot.sh | 5 +- lib/92-onepass.sh | 21 ++++- lib/95-summary.sh | 3 +- suite/test_critical_path.py | 171 ++++++++++++++++++++++++++++++++++++ 15 files changed, 359 insertions(+), 40 deletions(-) create mode 100644 suite/test_critical_path.py diff --git a/HANDOFF.md b/HANDOFF.md index 2e3cfb3..bcb336c 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -154,6 +154,26 @@ start and the fake pod's boot; the fake RunPod for `--gpus 2`, the count query, - One shot rendered on server 1 is byte-identical in content to the same shot on server 0 (same seed). - Host RAM with both servers loaded (the 4/8 rows of `lowestPrice` once they have stock). +## 8. The cuBLAS probe beside the venv build; the install's timeline (3.22.0) (2026-10-17, no pod used) + +**Why.** Paul's priority 3, install speed. The last Video Replacer pod (yswh3tkypxhxev, 1x B200, base 3.20.0, every row: +42 files, 295.5 GB) took about 5:17 from create to the App on its fresh attempt (its 16:55 includes a failed first +attempt and the pod stopped for about 11 minutes). The long pole was the 66.3 GB B300-only file (147 s with its sha256), +with VENV (116 s) beside it, and VENV held an apt install of `cuda-libraries-dev-13-0` (2,251 MB, 33 packages) on an +install that built nothing. The stage clock had seconds only, so what ran beside what was inferred, not read. + +**What changed.** `lib/40-packs.sh`: `_base_cuda_toolchain_start` / `_join` / `_stop`; `lib/95-summary.sh` starts the +probe with the venv and joins it before `pkg_post_venv`. `lib/00-env.sh`: `_base_span`, `BASE_SPANS`, `BASE_FILE_SPANS`, +`BASE_EPOCH0`, `BASE_CUDA_FOREGROUND`; `lib/92-onepass.sh`: `_base_timeline_write` (`_.timeline.tsv`); +`lib/50-models.sh`: each download's `BASE_VERIFY_SECS` and spans; the background joins (system, models, tools) add +theirs. `_build/pod/podctl.py`: `PODCTL_TIMES=1`. + +**Proven, and how.** `suite/test_critical_path.py` in CI (nothing ran on this Mac but `bash -n` and the packager). + +**Owed (the next pod).** The VENV stage without the apt, the `CUDA TOOLCHAIN (in the background…)` row's seconds, and the +timeline's critical path; the measurement plan is in the PR. The ranked levers and the sources are in the orchestrator's +notes (`install-speed-ranked.md`, `install-speed-sources.md`). + ## Requests ### 1. To the MiniMax H3 Video Replacer: declare the app's port diff --git a/MANIFEST.sha256 b/MANIFEST.sha256 index 545f524..9cd7d2b 100644 --- a/MANIFEST.sha256 +++ b/MANIFEST.sha256 @@ -1,30 +1,30 @@ af891a1c6f3f3958b43ea1f97a079ff898394ac72b2f49f88f3db66276f94eaf base.sh 577a83f8e3e3c4d69e04f4cbcee7ead42c49981c76f9bb07f8f78fd076876e01 comfyui-base-script.sh -17e09f0c32655044fdc038fb046f87a5d0548ed9601f59fcb97fb3ea379c6d30 comfyui-base-handbook.md -6ed25779fb19fdab1aa720b59c911750e3c4c9208e0496bb72e2605302947616 VERSION +845a3a412366beb8bbd1a98a59b5196120d19f43275fb5109259af1cfadfceac comfyui-base-handbook.md +46058ff89ee6dab89b7f99a6a21b117815ecf0e1f3967e51802d462c95f332fb VERSION 939c10b34e782bb77792eb876d8a51d74b38192598483c8c628f5da1b877397d pytest.ini 8aa67b7331d052dbbd1dcf9b357f088891e0fa0f4ed56e6669c105046fe881b2 LICENSE -75cf532f7718f7b546d033810c2aa623abd788605c42df4fa25e5f57108dddb3 lib/00-env.sh +3dcc371917e882b798f0f326a61d0fc3e1b6d65a799adba36344371d26568405 lib/00-env.sh bb3428df51590d49871f14f6bf1badaa66ce3627bbb0181516b8fa14ce040e2d lib/10-discover.sh -4f477b9ce63fd1d7388f79acc32eb040b561a37db4c06f9bcf325245c29fe0ff lib/15-system.sh +d642ea2ae686b78f4596f3ebb81a2ebdc751e84d7e580fb931b101d40bdc5084 lib/15-system.sh 2f16862a9c5f675dd1d20e8b56263832e1cf991e4cb1a05e1fd00bec2231158b lib/20-comfyui.sh ba32023a2c0319c5675d37f10ff30874462f2a45ce78323e1c1f714f12933c97 lib/30-venv.sh -60c559d7c7e89b5966154575dead1969392f1861b0b872d2143bb631d8ed3d92 lib/40-packs.sh +72ca6f74f967828dbb5f44087a1689186bdf5c347ae58dc3fa18b19a94a4f97b lib/40-packs.sh 5a410ec6f4d7790ec57c07296c1d911cac729c01947985cdc8816e63873d3f7a lib/45-mcp.sh 19aca634d77a27185efa3aaf4b2bfdab1a21af82334acb18c94a9aab5be241c4 lib/47-latest.sh db941b718ddf05903b0dc867e2a55378f96fcda776b2aa359f21c2c929a8d390 lib/48-runtime.sh -f2eeaade67ddc4118e0950cb09a9067f730158f38fc148a1359f6ff51f9074f6 lib/50-models.sh -5eb9452a949de071a61a9c81c047a7560b86cfdd4ced26ecb1eca24111bb9b51 lib/52-models-early.sh +6f4ebe61c63d143a7e24228d7ec9ef4313b76c87a278a575ccffd6fb63bb4ebd lib/50-models.sh +4bae452bbac2cf9d2a5bbd31819942e565b8eab367636ab91795911fee10c82b lib/52-models-early.sh 89fd610570781905c0793838ce76e460b1c6b7c8ef25e18c478115fd345251c7 lib/60-sync.sh 432bebc4ebe0675db14c6d8875327bae1cc0b5f9837b0c181d32bdb8f5c76171 lib/62-own-packs.sh 2feed9211c92aab85499c9f30ab5de5ecebc86af3956e48d0bf4bc57540c51ab lib/70-hygiene.sh -e42674d02fa4b857bbd25119efeac27b0496e83e30522685604aeea7e41018f6 lib/75-boot.sh +e9e64c96ab5df20fee132a50b84eac257bc8270dd429d86df9a16b2aed39e377 lib/75-boot.sh 141ba2100e2f921e80053742ce3dae4b6d86cdcd60109cc095242453812d988f lib/80-server.sh d5aee6a3e6d71ad9fabc5683bd9d1d592f252d63046118e53b29716f0ef66965 lib/84-run-local.sh 139ad7df5650f6bd237416eaae0a2cdf28709b11f1c9dc08d47610e07f9b3557 lib/85-launch.sh 9eda7369f29fc5e4bcc7f56ad64c3b5879fdd2844306dbdf463615fed49af46a lib/90-ledger.sh -917f38ec2418c9c4923ca59b815b013939601c6cf46a8d016b36889e4f0089ff lib/92-onepass.sh -25f40ce1f0acde64e1ad024c93e3ebed092480be88ae65ca75f72f1f30f023ee lib/95-summary.sh +9bb5ad4f2e909110c7030eb338fff8630f54e62b95ada99b47507f0395f95db7 lib/92-onepass.sh +1a46cb09b8d0bd334a9c2e92f86ba45e6ab91ebdb27fc7517947624e41a8134d lib/95-summary.sh 82943bbc7c31835bf997afb70e41ba9c56ddaf82712a3348537e60f911c2fb6a lib/boot.sh 3eee2b5d5bafb0c1501aa2183dcc33367e4662a6f9717348a0d30f11378939b6 py/app_proxy.py 58a84a14ec739aeadef47b83e8b30827713b2c21c0f690d3d3ce981c64894a9c py/arch_fit.py @@ -59,6 +59,7 @@ fcd96d1acae7a2761983039fb6a326cbc31711a62ac4f5d0509f2fe124b12246 py/venv_probe. ff2a64b4475aca687b6a98625c1d7eb328ee54052e02db70b8a88607250a5769 suite/test_bare.py 2c72647bfba5cd37297b8a253aba512e95db5a672bf30611acc3c6f181026165 suite/test_canvas.py 3d90e87daca9797667acb72644e98cfcfa39bcc37a80fbbac19e696b18de6717 suite/test_comfyui.py +78584c505587b78a44d71dbc3662a60bd57cdde8be91f068261c07f626fa0b42 suite/test_critical_path.py 1dccf801f852fd24988da239155828025382cc5aeb191e5f3446c362647bc470 suite/test_crusoe.py 6907c8283d0332361b63c525dcd664a82fa2f0640fcd6f8fcfa7d8d7c98d614c suite/test_derive.py 2a1cd2075a4b9197bc4655bb6849cce36c7c56014245fcdaa1ebe19f0ca728e6 suite/test_hashes.py diff --git a/VERSION b/VERSION index 6075c9a..a7e7070 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -3.21.0 +3.22.0 diff --git a/_build/pod/podctl.py b/_build/pod/podctl.py index 40d7012..c3277e0 100644 --- a/_build/pod/podctl.py +++ b/_build/pod/podctl.py @@ -729,7 +729,9 @@ def item_for_zip(zip_path, logs=None): class PodIO: """The real pod: ssh (no forwards), scp, the packager. `say` prints progress lines.""" def __init__(self, api, pod, env=None, packager=None): + import time self.api, self.pod, self.env = api, pod, env or os.environ + self._t0 = time.monotonic() # 3.22.0: PODCTL_TIMES=1 stamps each line from here self.provider = api if isinstance(api, Provider) else RunPodProvider(api) # an Api is RunPod's; a provider is any host's self.packager = packager or (pathlib.Path(__file__).resolve().parents[1] / "package.py") def check_zip(self, d): @@ -869,7 +871,17 @@ def sleep(self, s): import time time.sleep(s) def say(self, *a): - print(*a, flush=True) + print(*(_stamp(self.env, getattr(self, "_t0", None)) + list(a)), flush=True) + + +def _stamp(env, t0): + """3.22.0: PODCTL_TIMES=1 → ["[15:00:06Z +46.2s]"] (UTC wall time and seconds since the PodIO was made), so the Mac's + half of an install (lease, uploads, preflight, the poll that sees STEP RC) lines up with the pod's own timeline.tsv; + otherwise []. Off by default: every caller that parses podctl's lines reads them unchanged.""" + import datetime, time + if (env or {}).get("PODCTL_TIMES") != "1" or t0 is None: + return [] + return ["[%sZ +%.1fs]" % (datetime.datetime.now(datetime.timezone.utc).strftime("%H:%M:%S"), time.monotonic() - t0)] def _tail(text, n=25): diff --git a/comfyui-base-handbook.md b/comfyui-base-handbook.md index 67cdbd8..ea8804c 100644 --- a/comfyui-base-handbook.md +++ b/comfyui-base-handbook.md @@ -628,7 +628,8 @@ that does not need another's output runs beside it, and a join stands only where ◄────────────────────────────────────────────────────┘ ├─ model downloads (BASE_FETCH_JOBS at once, largest first) ... background ─┐ joined before the model stage ├─ JupyterLab's tools venv .................................... background ─┼─┐ joined at BOOT - └─ venv snapshot → venv → toolkit → pkg_post_venv (Sage) → own packs │ │ + ├─ cuBLAS probe and its apt install (3.22.0) ................... background │ │ joined before pkg_post_venv + └─ venv snapshot → venv → pkg_post_venv (Sage) → own packs │ │ → packs pip (skipped when the venv build installed this exact set) │ │ → Comfy MCP → newest → [local-runtime copy, when lib/84-run-local.sh is in] │ → pkg_pre_models ◄─────────────────────────────────────────────────────┘ │ @@ -645,7 +646,12 @@ tree. A store that already holds this exact install green (same package files, s place) runs none of it: `ALREADY INSTALLED`, ComfyUI started if nothing serves, done in seconds (`BASE_FULL=1` runs it all). **One pass, and its clock** (3.2.0). Every stage header closes the stage before it with its seconds; the summary prints -them slowest first and `state/logs/_.timings.tsv` keeps them, which podctl fetches with the console. When the +them slowest first and `state/logs/_.timings.tsv` keeps them, which podctl fetches with the console. 3.22.0: +beside it, `_.timeline.tsv` keeps WHEN: `start TAB end TAB seconds TAB what` in epoch seconds, earliest first, +for every stage, every background stage (from its start to its end, not just its seconds) and every model file twice +(`download `, the whole job, and `sha256 `, its re-read at the end), so the critical path of an install is read +from the run; each file's line in the console also says how many of its seconds were the sha256. `PODCTL_TIMES=1` stamps +podctl's own lines with the UTC time and the seconds since the install began, so the Mac's half lines up with it. When the only change a run makes is the package's own packs (ComfyUI, every other pack and the venv's package list unchanged), the import check loads ComfyUI with just those packs (`--disable-all-custom-nodes --whitelist-custom-nodes`); anything short of a pass there is judged again by the full check. `