Skip to content

fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard - #632

Merged
spraveenio merged 1 commit into
ROCm:mainfrom
bhatnitish:user/bhatnitish/gpuop-1062-nfd-device-ids-rocm
Aug 21, 2026
Merged

fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard#632
spraveenio merged 1 commit into
ROCm:mainfrom
bhatnitish:user/bhatnitish/gpuop-1062-nfd-device-ids-rocm

Conversation

@bhatnitish

@bhatnitish bhatnitish commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Deploying the GPU Operator via OLM on OpenShift with Radeon GPUs detects zero GPUs. On an 8x Radeon AI PRO R9700 (0x7551) cluster, every test failed with No nodes with AMD/GPU found in the cluster.

On OpenShift the NodeFeatureRule is created by hand from docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc — a NodeFeatureDiscovery CR and a NodeFeatureRule CR — listed AMD Instinct device IDs only, so no node ever received feature.node.kubernetes.io/amd-gpu and nothing the operator deploys was scheduled.

Root cause

The AMD GPU PCI device-ID list is duplicated across several hand-maintained files with nothing enforcing agreement. The commit that added the Radeon AI PRO R9700 (0x7551) updated the two gpu-nfd-default-rule.yaml copies and left the documentation untouched.

Drift measured against the source of truth: 16 missing amd-gpu IDs and 2 missing amd-vgpu IDs.

Changes

File Change
docs/installation/openshift-olm.md +16 amd-gpu / +2 amd-vgpu device IDs, both YAML blocks
docs/troubleshooting.md New section for a missing amd-gpu label
hack/check-nfd-device-ids.py Drift checker (Python 3 stdlib only)
Makefile make check-nfd-device-ids
.github/workflows/nfd-device-id-check.yml CI, a thin caller of the make target

Source of truth is hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml. make helm-k8s runs rm -rf helm-charts-k8s and then repopulates it from template-patch, so helm-charts-k8s/templates/gpu-nfd-default-rule.yaml is a build output — drift there means the wrong file was edited, or make helm-k8s was not re-run before committing. The checker reports both directions and names the exact IDs.

The workflow carries no branch or path filter: a path-filtered workflow reports no status on unrelated PRs, which stalls merges if it is configured as a required status check. The check runs in about a second.

Validation

  • Run before the doc fix, the checker failed with exactly the 16+2 IDs. It drove the doc edit, so the fix cannot be partial. It passes after.
  • Both doc YAML blocks parse via pyyaml, yielding 30 amd-gpu and 11 amd-vgpu entries — the added standalone comments inside the flow sequences do not break copy-paste.
  • Checker verified in both directions (missing and extra IDs), plus its parse-failure guard and its guard against a documentation code block being removed wholesale.
  • make check-nfd-device-ids is green, and is unaffected by a missing docker socket.

Docs lint: passes in CI — see the Documentation / Markdown and Documentation / Spelling checks on this PR.

Deploying via OLM on OpenShift with Radeon GPUs detected zero GPUs. On an
8x Radeon AI PRO R9700 (0x7551) cluster this failed every test with
"No nodes with AMD/GPU found in the cluster".

On OpenShift the NodeFeatureRule is created by hand from
docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc
(a NodeFeatureDiscovery CR and a NodeFeatureRule CR) listed Instinct device
IDs only, so no node received feature.node.kubernetes.io/amd-gpu and nothing
the operator deploys was ever scheduled.

Root cause is duplication with nothing enforcing agreement: the commit adding
the Radeon AI PRO R9700 updated the two gpu-nfd-default-rule.yaml copies and
left the documentation untouched.

Fix, and a guard so it cannot recur silently:

- docs/installation/openshift-olm.md: add the 16 missing amd-gpu and 2 missing
  amd-vgpu device IDs to both YAML blocks.
- docs/troubleshooting.md: new section for the amd-gpu label being absent,
  covering both causes -- no rule applied at all, or a rule that omits the
  device ID.
- hack/check-nfd-device-ids.py: compare every copy of the list against the
  source of truth, hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml,
  and report the exact missing/extra IDs per rule. The helm copy is a build
  output -- make helm-k8s does rm -rf on helm-charts-k8s and repopulates it
  from template-patch -- so drift there means the wrong file was edited.
- make check-nfd-device-ids: thin wrapper so contributors get the identical
  result locally.
- .github/workflows/nfd-device-id-check.yml: calls the make target. No branch
  or path filter, so it can safely be made a required status check.
@bhatnitish
bhatnitish force-pushed the user/bhatnitish/gpuop-1062-nfd-device-ids-rocm branch from f185035 to 154d243 Compare August 21, 2026 20:38

@spraveenio spraveenio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@spraveenio
spraveenio merged commit 668bec0 into ROCm:main Aug 21, 2026
5 checks passed
spraveenio pushed a commit that referenced this pull request Aug 24, 2026
…632) (#635)

Deploying via OLM on OpenShift with Radeon GPUs detected zero GPUs. On an
8x Radeon AI PRO R9700 (0x7551) cluster this failed every test with
"No nodes with AMD/GPU found in the cluster".

On OpenShift the NodeFeatureRule is created by hand from
docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc
(a NodeFeatureDiscovery CR and a NodeFeatureRule CR) listed Instinct device
IDs only, so no node received feature.node.kubernetes.io/amd-gpu and nothing
the operator deploys was ever scheduled.

Root cause is duplication with nothing enforcing agreement: the commit adding
the Radeon AI PRO R9700 updated the two gpu-nfd-default-rule.yaml copies and
left the documentation untouched.

Fix, and a guard so it cannot recur silently:

- docs/installation/openshift-olm.md: add the 16 missing amd-gpu and 2 missing
  amd-vgpu device IDs to both YAML blocks.
- docs/troubleshooting.md: new section for the amd-gpu label being absent,
  covering both causes -- no rule applied at all, or a rule that omits the
  device ID.
- hack/check-nfd-device-ids.py: compare every copy of the list against the
  source of truth, hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml,
  and report the exact missing/extra IDs per rule. The helm copy is a build
  output -- make helm-k8s does rm -rf on helm-charts-k8s and repopulates it
  from template-patch -- so drift there means the wrong file was edited.
- make check-nfd-device-ids: thin wrapper so contributors get the identical
  result locally.
- .github/workflows/nfd-device-id-check.yml: calls the make target. No branch
  or path filter, so it can safely be made a required status check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants