Repository navigation
fix(gpu): raise nvidia-cdi-refresh TimeoutStartSec for multi-GPU nodes (GB300) - #9899
Merged
Cameron Meissner (cameronmeissner) merged 1 commit intoOct 26, 2026
Merged
Conversation
Contributor
Windows Unit Test Results 3 files 18 suites 1m 6s ⏱️ Results for commit 15ca828. ♻️ This comment has been updated with latest results. |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The narrow timeout change preserves existing failure handling, includes a focused regression assertion, and has no identified blocking issues.
No open issues 🎉
What changed in this PR
Raises the NVIDIA CDI refresh startup timeout to prevent slow GPU initialization on GB200/GB300 nodes from failing Linux node provisioning.
Changes:
- Adds a 300-second default timeout, configurable through
NVIDIA_CDI_REFRESH_TIMEOUT_START_SEC, while preserving existing exit-code handling. - Adds a ShellSpec assertion for the timeout written to the systemd drop-in.
| File | Description |
|---|---|
spec/parts/linux/cloud-init/artifacts/cse_config_gpu_spec.sh |
Verifies the drop-in includes TimeoutStartSec=300. |
parts/linux/cloud-init/artifacts/cse_config_gpu.sh |
Writes the configurable timeout and documents its purpose. |
Review effort: Balanced
Xu Xue (xuexu6666)
force-pushed
the
xuxue/gpu-cdi-refresh-timeout
branch
from
October 26, 2026 03:13
fbcd60b to
1aee110
Compare
…well (GB300) nvidia-container-toolkit 1.20.1 (aks-gpu #177) regresses GB200/GB300 managed-driver provisioning. 1.19.1 left this Type=oneshot unit at systemd's oneshot default of TimeoutStartSec=infinity; 1.20.1 ships /usr/lib/systemd/system/nvidia-cdi-refresh.service.d/10-container-engines.conf which orders the container engines after it AND pins TimeoutStartSec=90s. On Grace-Blackwell the start is slow -- nvidia-ctk cdi generate ~81s over the arm64 Grace NVLink/nvml path plus a cold ~28s nvidia-smi -L precheck (>90s) -- so systemd SIGTERMs it. A timeout is result=timeout (SIGTERM), not a SuccessExitStatus code, so install.sh's `systemctl restart` still fails -> CSE exit 84 -> kubelet never starts -> node never joins. Write the TimeoutStartSec=300 override to a SEPARATE 90-aks-cdi-refresh-timeout drop-in (arm64 only). The 90- prefix is load-bearing: systemd merges drop-ins in lexical filename order (last wins), so the override must sort AFTER the toolkit's 10-container-engines.conf -- a 10-aks-* name sorts before it and loses (verified: TimeoutStartUSec stays 90s). Verified live on GB300 that the 90- drop-in raises TimeoutStartUSec 90s -> 5min. x86 generates in <1s (measured: 8x H200), so no override is written there. Overridable via NVIDIA_CDI_REFRESH_TIMEOUT_START_SEC.
Xu Xue (xuexu6666)
force-pushed
the
xuxue/gpu-cdi-refresh-timeout
branch
from
October 26, 2026 03:36
1aee110 to
15ca828
Compare
Contributor
|
Contributor
|
Contributor
|
Nishchay (awesomenix)
approved these changes
Oct 26, 2026
Cameron Meissner (cameronmeissner)
approved these changes
Oct 26, 2026
Contributor
Author
✅ Validated live on real GB300 (abe2e, toolkit 1.20.1 + this fix)Ran the GB300 managed-DRA + compute-domain scenario in AgentBaker abe2e on a real GB300 rack (eastus2euap), with the exact toolkit that causes the regression (1.20.1) and this fix applied. Result: 5/5 GB300 nodes healthy and
Measured cold cdi-refresh start on GB300: ~137s (service journal: Negative control: the same path on 1.20.1 without this fix is So on real GB300, the regressing toolkit + this fix → managed driver + DRA + compute-domain all come up clean. |
Cameron Meissner (cameronmeissner)
merged commit Oct 26, 2026
3ffe873
into
main
40 of 42 checks passed
Cameron Meissner (cameronmeissner)
deleted the
xuxue/gpu-cdi-refresh-timeout
branch
October 26, 2026 21:53
Abigail Liang (abigailliang-aks-sig-node)
pushed a commit
that referenced
this pull request
Oct 30, 2026
Dallas Delaney (dallasd1)
pushed a commit
that referenced
this pull request
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GB200/GB300 managed-driver (
--gpu-driver Install) nodes fail to join (CSE exit 84, kubelet never starts) after theaks-gpu-cuda-ltsimage was built with nvidia-container-toolkit 1.20.1 (aks-gpu #177, first in image tag580.159.04-20261020182457).Root cause
nvidia-cdi-refresh.serviceisType=oneshot. Toolkit 1.19.1 left it at systemd's oneshot default ofTimeoutStartSec=infinity. 1.20.1 ships a new drop-in/usr/lib/systemd/system/nvidia-cdi-refresh.service.d/10-container-engines.confthat orders the container engines after the unit and pinsTimeoutStartSec=90s.On Grace-Blackwell the unit's start is slow:
nvidia-smi -Lprecheck: ~28s (cold, before persistence is warm)nvidia-ctk cdi generate: ~81s (enumerating the GPUs + driver libs/firmware over the arm64 Grace NVLink/nvml path)Total ~109s > 90s → systemd SIGTERMs it. A timeout is
result=timeout(SIGTERM), not an exit code, so the existingSuccessExitStatus=1 2 127drop-in can't cover it.install.sh's blockingsystemctl restart nvidia-cdi-refresh.servicethen returns non-zero → the aks-gpu install container exits non-zero → retried, but every retry now aborts atnvidia-installer(module already loaded, can't unload on a live GPU node) →retrycmdexhausts → CSE exit 84.Blast radius
Grace-Blackwell arm64 only (GB200/GB300). x86 is unaffected — measured on a live 8× H200:
nvidia-smi -L0.11s,cdi generate<1s (and on 1.19.1 the unit wasTimeoutStartSec=infinity).Fix
Write
TimeoutStartSec=300to a separate, higher-sorted drop-in90-aks-cdi-refresh-timeout.conf, arm64-gated (getCPUArch).The
90-prefix is load-bearing: systemd merges drop-ins in lexical filename order, last wins. The toolkit's10-container-engines.confsets 90s, and a10-aks-*name sorts before it (a<c) and loses — so the override must sort after it. (The existing10-aks-tolerate-generate-failure.confis left untouched forSuccessExitStatus; it doesn't conflict with ordering.)Verified live on a GB300 node: a
10-aks-*timeout drop-in leavesTimeoutStartUSec=1min 30s; the90-drop-in raises it to5min. x86 writes no override (stays at the 90s default). Overridable viaNVIDIA_CDI_REFRESH_TIMEOUT_START_SEC.Scope is intentionally limited to the timeout. The
components.jsonbump to the 1.20.1 image is the fleet-exposure event and must land with/after this fix.Testing
cse_config_gpu_spec.sh: 79/79 pass — arm64 asserts the90-drop-in exists withTimeoutStartSec=300(and the10-drop-in has no timeout); x86 asserts the90-drop-in is not written.shellcheckclean (no new findings).make generate/testdata churn — drop-ins are written at runtime, not rendered into customData.TimeoutStartUSec90s → 5min on a GB300 node.