Skip to content

[CP 1658] fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard (GPUOP-1062) - #634

Closed
ci-penbot-01 wants to merge 1 commit into
ROCm:mainfrom
ci-penbot-01:CP.O2O.pensando.gpu-operator.1658.rocm.gpu-operator.main
Closed

[CP 1658] fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard (GPUOP-1062)#634
ci-penbot-01 wants to merge 1 commit into
ROCm:mainfrom
ci-penbot-01:CP.O2O.pensando.gpu-operator.1658.rocm.gpu-operator.main

Conversation

@ci-penbot-01

Copy link
Copy Markdown
Contributor

cp of pensando/gpu-operator#1658


Source PR Description (pensando/gpu-operator#1658):

Summary

Deploying via OLM on OpenShift with Radeon GPUs detects zero GPUs. On an 8x Radeon AI PRO R9700 (0x7551) SNO cluster this failed ~300 tests with No nodes with AMD/GPU found in the cluster (jobd 33207089).

On OpenShift the NodeFeatureRule is created by hand from docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc — a NodeFeatureDiscovery CR and a NodeFeatureRule CR — listed Instinct device IDs only, so no node received feature.node.kubernetes.io/amd-gpu and every test bailed at setup in k8_get_gpu_nodes() (tests/pytests/lib/k8_util.py:176).

Fixes GPUOP-1062.

Root cause

The device-ID list is duplicated with nothing enforcing agreement. Commit 9c5ef17 added 0x7551 to the two gpu-nfd-default-rule.yaml copies and left the docs untouched. GPUOP-990 (the 1.5.1 Radeon docs pass) updated framing and the version matrix but never the device lists.

Drift measured: 16 missing amd-gpu IDs and 2 missing amd-vgpu IDs.

Changes

File Change
docs/installation/openshift-olm.md +16 amd-gpu / +2 amd-vgpu IDs, both YAML blocks
docs/troubleshooting.md New section: AMD PCI device visible but no amd-gpu label
hack/check-nfd-device-ids.py Drift checker (stdlib only)
Makefile make check-nfd-device-ids
.claude/hooks/check-nfd-drift.sh + .claude/settings.json PostToolUse hook reporting drift in-session
.github/workflows/nfd-device-id-check.yml CI, calls the make target

Source of truth is hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml. make helm-k8s runs rm -rf helm-charts-k8s and repopulates from template-patch, so the helm copy is a build output — drift there means the wrong file was edited.

The workflow carries no branch or path filter: a path-filtered workflow reports no status on unrelated PRs, which stalls merges once it is a required check.

Validation

  • Checker run before the doc fix failed with exactly the 16+2 IDs, independently reproducing the manual analysis. It drove the doc edit, so the fix cannot be partial. Passes after.
  • Both doc YAML blocks parse via pyyaml, yielding 30 amd-gpu and 11 amd-vgpu entries — the added standalone comments inside the flow sequences do not break copy-paste.
  • Hook verified: silent on unrelated edits; reports on drift; catches the extra-ID direction; trips its parse-failure and min_blocks guards; fails open with python3 absent or a malformed payload.
  • make check-nfd-device-ids green; settings.json valid JSON; workflow valid YAML; make target unaffected by a missing docker socket.

Docs lint: not runnable locally (markdownlint-cli2 / pyspelling not installed). The identical Markdown passes both docs-lint jobs in CI on the ROCm counterpart PR (#632).

Cherrypick triggered by: ACP-Automation

…(GPUOP-1062) (#1658)

* fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard (GPUOP-1062)

Deploying via OLM on OpenShift with Radeon GPUs detected zero GPUs. On an
8x Radeon AI PRO R9700 (0x7551) cluster this failed ~300 tests with
"No nodes with AMD/GPU found in the cluster".

The OLM bundle ships no NodeFeatureRule -- verified: none in bundle/manifests,
none in config/, none in hack/openshift-patch, and the operator creates none at
runtime. On OpenShift the user must create the rule by hand, and the only
source for its contents is docs/installation/openshift-olm.md. Both copy-paste
YAML blocks in that doc (a NodeFeatureDiscovery CR and a NodeFeatureRule CR)
listed Instinct device IDs only, so no node ever got the
feature.node.kubernetes.io/amd-gpu label and every test bailed at setup.

Root cause is duplication with nothing enforcing agreement: commit 9c5ef17
added 0x7551 to the two gpu-nfd-default-rule.yaml copies and left the docs
untouched.

Fix, and a guard so it cannot recur silently:

- docs/installation/openshift-olm.md: add the 16 missing amd-gpu and 2 missing
  amd-vgpu device IDs to both YAML blocks.
- docs/troubleshooting.md: new section for pci-1002.present=true with no
  amd-gpu label.
- hack/check-nfd-device-ids.py: compare every copy of the list against the
  source of truth, hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml,
  and report the exact missing/extra IDs per rule. The helm copy is a build
  output -- make helm-k8s does rm -rf on helm-charts-k8s and repopulates it
  from template-patch -- so drift there means the wrong file was edited.
- make check-nfd-device-ids: thin wrapper, so contributors without Claude Code
  or CI get the identical result.
- .claude/hooks/check-nfd-drift.sh: PostToolUse hook reporting drift in-session
  at the moment it is introduced. Fails open.
- .github/workflows/nfd-device-id-check.yml: calls the make target. No branch
  or path filter, so it can safely be made a required status check.

Plan: docs-internal/knowledge/plans/2026-08-21-gpuop-1062-openshift-nfd-radeon-device-ids.md

* fix(docs): keep the troubleshooting entry to verified behaviour

The first draft of the entry told users to look for a literal
feature.node.kubernetes.io/pci-1002.present label and described how that name
is derived. That detail came from upstream NFD documentation rather than from
anything exercised here, and the exact label varies with the deviceLabelFields
in use, so it does not belong in our troubleshooting guide.

The entry now sticks to what is verifiable in this repository: amd-gpu is the
label the operator selects on, it comes only from a rule the user supplies, the
OLM bundle ships no such rule so one must be created on OpenShift, and the Helm
chart installs the full device list on Kubernetes when installdefaultNFDRule is
enabled.

Also drops the suggested 'grep -i amd' pipeline, which could never have matched
a pci-1002.present label in the first place.

(cherry picked from commit 2aa0eaaab0f3fd82a910efd7613aa342fec238a3)
@ci-penbot-01

Copy link
Copy Markdown
Contributor Author

AI-Assisted Cherry-Pick

Source PR: #1658
Target Branch: main

The cherry-pick operation encountered merge conflicts which were resolved automatically using AI assistance.

Files with conflicts (resolved by AI):

  • .claude/settings.json:56-61
Original conflict in .claude/settings.json
<<<<<<< HEAD (deleted)
=======
      {
        "matcher": "Write|Edit",
        "hooks": [
          { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/gofmt-on-edit.sh" },
          { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/check-nfd-drift.sh" }
        ]
      }
>>>>>>> 2aa0eaaa (incoming commit wanted to keep file with new hook)

Cherry-pick triggered by: ACP-Automation

@spraveenio spraveenio closed this Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants