[CP 1658] fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard (GPUOP-1062) - #634
Closed
ci-penbot-01 wants to merge 1 commit into
Conversation
…(GPUOP-1062) (#1658) * fix(docs,ci): Radeon device IDs in OpenShift NFD rules + drift guard (GPUOP-1062) Deploying via OLM on OpenShift with Radeon GPUs detected zero GPUs. On an 8x Radeon AI PRO R9700 (0x7551) cluster this failed ~300 tests with "No nodes with AMD/GPU found in the cluster". The OLM bundle ships no NodeFeatureRule -- verified: none in bundle/manifests, none in config/, none in hack/openshift-patch, and the operator creates none at runtime. On OpenShift the user must create the rule by hand, and the only source for its contents is docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc (a NodeFeatureDiscovery CR and a NodeFeatureRule CR) listed Instinct device IDs only, so no node ever got the feature.node.kubernetes.io/amd-gpu label and every test bailed at setup. Root cause is duplication with nothing enforcing agreement: commit 9c5ef17 added 0x7551 to the two gpu-nfd-default-rule.yaml copies and left the docs untouched. Fix, and a guard so it cannot recur silently: - docs/installation/openshift-olm.md: add the 16 missing amd-gpu and 2 missing amd-vgpu device IDs to both YAML blocks. - docs/troubleshooting.md: new section for pci-1002.present=true with no amd-gpu label. - hack/check-nfd-device-ids.py: compare every copy of the list against the source of truth, hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml, and report the exact missing/extra IDs per rule. The helm copy is a build output -- make helm-k8s does rm -rf on helm-charts-k8s and repopulates it from template-patch -- so drift there means the wrong file was edited. - make check-nfd-device-ids: thin wrapper, so contributors without Claude Code or CI get the identical result. - .claude/hooks/check-nfd-drift.sh: PostToolUse hook reporting drift in-session at the moment it is introduced. Fails open. - .github/workflows/nfd-device-id-check.yml: calls the make target. No branch or path filter, so it can safely be made a required status check. Plan: docs-internal/knowledge/plans/2026-08-21-gpuop-1062-openshift-nfd-radeon-device-ids.md * fix(docs): keep the troubleshooting entry to verified behaviour The first draft of the entry told users to look for a literal feature.node.kubernetes.io/pci-1002.present label and described how that name is derived. That detail came from upstream NFD documentation rather than from anything exercised here, and the exact label varies with the deviceLabelFields in use, so it does not belong in our troubleshooting guide. The entry now sticks to what is verifiable in this repository: amd-gpu is the label the operator selects on, it comes only from a rule the user supplies, the OLM bundle ships no such rule so one must be created on OpenShift, and the Helm chart installs the full device list on Kubernetes when installdefaultNFDRule is enabled. Also drops the suggested 'grep -i amd' pipeline, which could never have matched a pci-1002.present label in the first place. (cherry picked from commit 2aa0eaaab0f3fd82a910efd7613aa342fec238a3)
Contributor
Author
AI-Assisted Cherry-PickSource PR: #1658 The cherry-pick operation encountered merge conflicts which were resolved automatically using AI assistance. Files with conflicts (resolved by AI):
Original conflict in .claude/settings.json<<<<<<< HEAD (deleted)
=======
{
"matcher": "Write|Edit",
"hooks": [
{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/gofmt-on-edit.sh" },
{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/check-nfd-drift.sh" }
]
}
>>>>>>> 2aa0eaaa (incoming commit wanted to keep file with new hook)Cherry-pick triggered by: ACP-Automation |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cp of pensando/gpu-operator#1658
Source PR Description (pensando/gpu-operator#1658):
Summary
Deploying via OLM on OpenShift with Radeon GPUs detects zero GPUs. On an 8x Radeon AI PRO R9700 (
0x7551) SNO cluster this failed ~300 tests withNo nodes with AMD/GPU found in the cluster(jobd 33207089).On OpenShift the NodeFeatureRule is created by hand from
docs/installation/openshift-olm.md. Both copy-paste YAML blocks in that doc — aNodeFeatureDiscoveryCR and aNodeFeatureRuleCR — listed Instinct device IDs only, so no node receivedfeature.node.kubernetes.io/amd-gpuand every test bailed at setup ink8_get_gpu_nodes()(tests/pytests/lib/k8_util.py:176).Fixes GPUOP-1062.
Root cause
The device-ID list is duplicated with nothing enforcing agreement. Commit 9c5ef17 added
0x7551to the twogpu-nfd-default-rule.yamlcopies and left the docs untouched. GPUOP-990 (the 1.5.1 Radeon docs pass) updated framing and the version matrix but never the device lists.Drift measured: 16 missing
amd-gpuIDs and 2 missingamd-vgpuIDs.Changes
docs/installation/openshift-olm.mdamd-gpu/ +2amd-vgpuIDs, both YAML blocksdocs/troubleshooting.mdamd-gpulabelhack/check-nfd-device-ids.pyMakefilemake check-nfd-device-ids.claude/hooks/check-nfd-drift.sh+.claude/settings.jsonPostToolUsehook reporting drift in-session.github/workflows/nfd-device-id-check.ymlSource of truth is
hack/k8s-patch/template-patch/gpu-nfd-default-rule.yaml.make helm-k8srunsrm -rf helm-charts-k8sand repopulates from template-patch, so the helm copy is a build output — drift there means the wrong file was edited.The workflow carries no branch or path filter: a path-filtered workflow reports no status on unrelated PRs, which stalls merges once it is a required check.
Validation
pyyaml, yielding 30amd-gpuand 11amd-vgpuentries — the added standalone comments inside the flow sequences do not break copy-paste.min_blocksguards; fails open withpython3absent or a malformed payload.make check-nfd-device-idsgreen;settings.jsonvalid JSON; workflow valid YAML; make target unaffected by a missing docker socket.Docs lint: not runnable locally (
markdownlint-cli2/pyspellingnot installed). The identical Markdown passes both docs-lint jobs in CI on the ROCm counterpart PR (#632).Cherrypick triggered by: ACP-Automation