fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard - #632
Merged
spraveenio merged 1 commit intoAug 21, 2026
Merged
Conversation
Deploying via OLM on OpenShift with Radeon GPUs detected zero GPUs. On an 8x Radeon AI PRO R9700 (0x7551) cluster this failed every test with "No nodes with AMD/GPU found in the cluster". On OpenShift the NodeFeatureRule is created by hand from docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc (a NodeFeatureDiscovery CR and a NodeFeatureRule CR) listed Instinct device IDs only, so no node received feature.node.kubernetes.io/amd-gpu and nothing the operator deploys was ever scheduled. Root cause is duplication with nothing enforcing agreement: the commit adding the Radeon AI PRO R9700 updated the two gpu-nfd-default-rule.yaml copies and left the documentation untouched. Fix, and a guard so it cannot recur silently: - docs/installation/openshift-olm.md: add the 16 missing amd-gpu and 2 missing amd-vgpu device IDs to both YAML blocks. - docs/troubleshooting.md: new section for the amd-gpu label being absent, covering both causes -- no rule applied at all, or a rule that omits the device ID. - hack/check-nfd-device-ids.py: compare every copy of the list against the source of truth, hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml, and report the exact missing/extra IDs per rule. The helm copy is a build output -- make helm-k8s does rm -rf on helm-charts-k8s and repopulates it from template-patch -- so drift there means the wrong file was edited. - make check-nfd-device-ids: thin wrapper so contributors get the identical result locally. - .github/workflows/nfd-device-id-check.yml: calls the make target. No branch or path filter, so it can safely be made a required status check.
bhatnitish
force-pushed
the
user/bhatnitish/gpuop-1062-nfd-device-ids-rocm
branch
from
August 21, 2026 20:38
f185035 to
154d243
Compare
yansun1996
approved these changes
Aug 21, 2026
spraveenio
pushed a commit
that referenced
this pull request
Aug 24, 2026
…632) (#635) Deploying via OLM on OpenShift with Radeon GPUs detected zero GPUs. On an 8x Radeon AI PRO R9700 (0x7551) cluster this failed every test with "No nodes with AMD/GPU found in the cluster". On OpenShift the NodeFeatureRule is created by hand from docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc (a NodeFeatureDiscovery CR and a NodeFeatureRule CR) listed Instinct device IDs only, so no node received feature.node.kubernetes.io/amd-gpu and nothing the operator deploys was ever scheduled. Root cause is duplication with nothing enforcing agreement: the commit adding the Radeon AI PRO R9700 updated the two gpu-nfd-default-rule.yaml copies and left the documentation untouched. Fix, and a guard so it cannot recur silently: - docs/installation/openshift-olm.md: add the 16 missing amd-gpu and 2 missing amd-vgpu device IDs to both YAML blocks. - docs/troubleshooting.md: new section for the amd-gpu label being absent, covering both causes -- no rule applied at all, or a rule that omits the device ID. - hack/check-nfd-device-ids.py: compare every copy of the list against the source of truth, hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml, and report the exact missing/extra IDs per rule. The helm copy is a build output -- make helm-k8s does rm -rf on helm-charts-k8s and repopulates it from template-patch -- so drift there means the wrong file was edited. - make check-nfd-device-ids: thin wrapper so contributors get the identical result locally. - .github/workflows/nfd-device-id-check.yml: calls the make target. No branch or path filter, so it can safely be made a required status check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deploying the GPU Operator via OLM on OpenShift with Radeon GPUs detects zero GPUs. On an 8x Radeon AI PRO R9700 (
0x7551) cluster, every test failed withNo nodes with AMD/GPU found in the cluster.On OpenShift the NodeFeatureRule is created by hand from
docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc — aNodeFeatureDiscoveryCR and aNodeFeatureRuleCR — listed AMD Instinct device IDs only, so no node ever receivedfeature.node.kubernetes.io/amd-gpuand nothing the operator deploys was scheduled.Root cause
The AMD GPU PCI device-ID list is duplicated across several hand-maintained files with nothing enforcing agreement. The commit that added the Radeon AI PRO R9700 (
0x7551) updated the twogpu-nfd-default-rule.yamlcopies and left the documentation untouched.Drift measured against the source of truth: 16 missing
amd-gpuIDs and 2 missingamd-vgpuIDs.Changes
docs/installation/openshift-olm.mdamd-gpu/ +2amd-vgpudevice IDs, both YAML blocksdocs/troubleshooting.mdamd-gpulabelhack/check-nfd-device-ids.pyMakefilemake check-nfd-device-ids.github/workflows/nfd-device-id-check.ymlSource of truth is
hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml.make helm-k8srunsrm -rf helm-charts-k8sand then repopulates it fromtemplate-patch, sohelm-charts-k8s/templates/gpu-nfd-default-rule.yamlis a build output — drift there means the wrong file was edited, ormake helm-k8swas not re-run before committing. The checker reports both directions and names the exact IDs.The workflow carries no branch or path filter: a path-filtered workflow reports no status on unrelated PRs, which stalls merges if it is configured as a required status check. The check runs in about a second.
Validation
pyyaml, yielding 30amd-gpuand 11amd-vgpuentries — the added standalone comments inside the flow sequences do not break copy-paste.make check-nfd-device-idsis green, and is unaffected by a missing docker socket.Docs lint: passes in CI — see the
Documentation / MarkdownandDocumentation / Spellingchecks on this PR.