Skip GPU resource validation when device plugin is disabled on the node

[framsouza]

• Fixes the CrashLoopBackOff of nvidia-operator-validator on nodes where the user has intentionally disabled the device plugin via nvidia.com/gpu.deploy.device-plugin=false, by short-circuiting Plugin.validateGPUResource with a sentinel error when the label is set.
• Also skips validation on nodes labeled nvidia.com/gpu-operator.resource-allocation.mode=dra, so the classic validator no-ops cleanly on nodes migrated to the DRA stack (belt-and-suspenders during reconciliation windows).
• No status file is written on the skip path, so the metrics exporter and any downstream reader correctly see "not validated" rather than a false-positive "plugin-ready".

Test plan

• go vet ./cmd/nvidia-validator/ clean
• go test ./cmd/nvidia-validator/ all pass, including 10 new subtests
• End-to-end verified on kind , see session below

Local tests with kind

Kind local tests output

# Build the validator from this branch for the kind node arch
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 \
go build -o /tmp/gpu-op-2550-test/nvidia-validator ./cmd/nvidia-validator

DOCKER_BUILDKIT=0 docker build -t nvidia-validator-2550:test /tmp/gpu-op-2550-test

# Cluster + image load
kind create cluster --name gpu-op-2550
kind load docker-image nvidia-validator-2550:test --name gpu-op-2550

# Namespace + SA + ClusterRole granting nodes:get
kubectl apply -f /tmp/gpu-op-2550-test/rbac.yaml

The reused Pod spec runs only the plugin component with
WITH_WAIT=false, WITH_WORKLOAD=false:

apiVersion: v1
kind: Pod
metadata:
name: plugin-validation
namespace: gpu-op-test
spec:
serviceAccountName: validator
restartPolicy: Never
containers:
- name: plugin-validation
    image: nvidia-validator-2550:test
    imagePullPolicy: IfNotPresent
    command: ["sh", "-c"]
    args: ["nvidia-validator 2>&1; echo EXIT=$?"]
    env:
    - {name: COMPONENT, value: plugin}
    - {name: WITH_WAIT, value: "false"}
    - {name: WITH_WORKLOAD, value: "false"}
    - {name: NODE_NAME, valueFrom: {fieldRef: {fieldPath: spec.nodeName}}}
    - {name: OPERATOR_NAMESPACE, value: gpu-op-test}
    - {name: OUTPUT_DIR, value: /tmp}

- Bug validation:

$ kubectl get node gpu-op-2550-control-plane --show-labels | grep -oE 'nvidia\.com/[^,]*'
(no nvidia.com/* labels)

$ kubectl apply -f plugin-validation-pod.yaml
pod/plugin-validation created

$ kubectl -n gpu-op-test logs plugin-validation
time="2026-07-28T12:49:00Z" level=info msg="version: unknown"
time="2026-07-28T12:49:00Z" level=info msg="GPU resources are not yet discovered by the node, retry: 1"
time="2026-07-28T12:49:05Z" level=info msg="GPU resources are not yet discovered by the node, retry: 2"
time="2026-07-28T12:49:10Z" level=info msg="GPU resources are not yet discovered by the node, retry: 3"
time="2026-07-28T12:49:15Z" level=info msg="GPU resources are not yet discovered by the node, retry: 4"
time="2026-07-28T12:49:20Z" level=info msg="GPU resources are not yet discovered by the node, retry: 5"

Matches the symptom in the issue: 30 retries × 5 s → non-zero exit → CrashLoopBackOff.

- Fix path with deploy.device-plugin=false 

$ kubectl label node gpu-op-2550-control-plane \
    nvidia.com/gpu.deploy.device-plugin=false --overwrite node/gpu-op-2550-control-plane labeled

$ kubectl apply -f plugin-validation-pod.yaml
pod/plugin-validation created

$ kubectl -n gpu-op-test get pod plugin-validation
NAME                READY   STATUS      RESTARTS   AGE
plugin-validation   0/1     Completed   0          5s

$ kubectl -n gpu-op-test logs plugin-validation
time="2026-07-28T12:50:42Z" level=info msg="version: unknown"
time="2026-07-28T12:50:42Z" level=info msg="Device plugin is disabled on this node, skipping GPU resource validation"
EXIT=0

Extra check that no plugin-ready status file is created (probe pod runs the validator, then ls /tmp/):

$ kubectl -n gpu-op-test logs probe-b
time="2026-07-28T12:51:08Z" level=info msg="version: unknown"
time="2026-07-28T12:51:08Z" level=info msg="Device plugin is disabled on this node, skipping GPU resource validation"
---
total 8
drwxrwxrwt    2 root     root          4096 Oct  8  2025 .
drwxr-xr-x    1 root     root          4096 Jul 28 12:51 ..
EXIT=0

/tmp is empty, downstream consumers correctly see "not validated."

- Fix path with resource-allocation.mode=dra 

$ kubectl label node gpu-op-2550-control-plane nvidia.com/gpu.deploy.device-plugin-node/gpu-op-2550-control-plane unlabeled

$ kubectl label node gpu-op-2550-control-plane \
    nvidia.com/gpu-operator.resource-allocation.mode=dra --overwrite
node/gpu-op-2550-control-plane labeled

$ kubectl apply -f plugin-validation-pod.yaml
pod/plugin-validation created

$ kubectl -n gpu-op-test get pod plugin-validation
NAME                READY   STATUS      RESTARTS   AGE
plugin-validation   0/1     Completed   0          5s

$ kubectl -n gpu-op-test logs plugin-validation
time="2026-07-28T12:51:43Z" level=info msg="version: unknown"
time="2026-07-28T12:51:43Z" level=info msg="Device plugin is disabled on this node, skipping GPU resource validation"
EXIT=0

- Negative control: deploy.device-plugin=true, old retry behavior preserved

$ kubectl label node gpu-op-2550-control-plane \
    nvidia.com/gpu-operator.resource-allocation.mode-
node/gpu-op-2550-control-plane unlabeled

$ kubectl label node gpu-op-2550-control-plane \
    nvidia.com/gpu.deploy.device-plugin=true --overwrite
node/gpu-op-2550-control-plane labeled

$ kubectl apply -f plugin-validation-pod.yaml
pod/plugin-validation created

$ kubectl -n gpu-op-test get pod plugin-validation
NAME                READY   STATUS    RESTARTS   AGE
plugin-validation   1/1     Running   0          18s

$ kubectl -n gpu-op-test logs plugin-validation | tail -5
time="2026-07-28T12:52:11Z" level=info msg="version: unknown"
time="2026-07-28T12:52:11Z" level=info msg="GPU resources are not yet discovered by the node, retry: 1"
time="2026-07-28T12:52:16Z" level=info msg="GPU resources are not yet discovered by the node, retry: 2"
time="2026-07-28T12:52:21Z" level=info msg="GPU resources are not yet discovered by the node, retry: 3"
time="2026-07-28T12:52:26Z" level=info msg="GPU resources are not yet discovered by the node, retry: 4"

Retry loop unchanged proves the fix is strictly additive.

- Edge case: deploy.device-plugin=paused-for-driver-upgrade, not treated as disabled

$ kubectl label node gpu-op-2550-control-plane \
    nvidia.com/gpu.deploy.device-plugin=paused-for-driver-upgrade --overwrite
node/gpu-op-2550-control-plane labeled

$ kubectl apply -f plugin-validation-pod.yaml
pod/plugin-validation created

$ kubectl -n gpu-op-test get pod plugin-validation
NAME                READY   STATUS    RESTARTS   AGE
plugin-validation   1/1     Running   0          12s

$ kubectl -n gpu-op-test logs plugin-validation | tail -4
time="2026-07-28T12:53:20Z" level=info msg="version: unknown"
time="2026-07-28T12:53:20Z" level=info msg="GPU resources are not yet discovered by the node, retry: 1"
time="2026-07-28T12:53:25Z" level=info msg="GPU resources are not yet discovered by the node, retry: 2"
time="2026-07-28T12:53:30Z" level=info msg="GPU resources are not yet discovered by the node, retry: 3"

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.