Add opt-in CronJob to recover pods stuck after GPU/SRIOV device plugin restart#145
Open
Hardikrepo wants to merge 2 commits into
Open
Add opt-in CronJob to recover pods stuck after GPU/SRIOV device plugin restart#145Hardikrepo wants to merge 2 commits into
Hardikrepo wants to merge 2 commits into
Conversation
…ker CE, modern apt keyring, Calico enP* support Resolves NVIDIA#140: replaces deprecated apt_key module with get_url+gpg --dearmor, pins docker-ce to 29.4.3 with arch detection instead of state: latest, switches GPU/Network Operator helm installs from install --generate-name to upgrade --install for retry-safety, and adds enP* to Calico IP_AUTODETECTION_METHOD for GB200/Grace NIC names.
…n restart Resolves NVIDIA#132: after a node restart, pods requesting GPU or SRIOV device resources can be rejected by the kubelet with "UnexpectedAdmissionError" / "Allocate failed due to no healthy devices present" because the device plugin hasn't re-registered yet. These pods are left in Failed state and require a manual delete to get rescheduled. Adds a new `device_pod_recovery` flag (default no) that, when enabled, deploys a kube-system CronJob (with its own ServiceAccount/ ClusterRole) that runs every 2 minutes, finds pods with status.phase=Failed and status.reason=UnexpectedAdmissionError, and deletes them so their owning controller reschedules them once the device plugin is healthy again.
Hardikrepo
force-pushed
the
fix/device-plugin-pod-recovery
branch
from
f87e47f to
9ae4f99
Compare
Collaborator
|
Thank you for the contribution. Please fork the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #132.
Summary
After a node restart, pods requesting GPU or SRIOV resources can be rejected by the kubelet with
UnexpectedAdmissionError("Allocate failed due to no healthy devices present") because the device plugin hasn't re-registered its devices yet. These pods sit inFailedstate and currently require manually deleting them so the owning controller reschedules them.This adds a new opt-in flag
device_pod_recovery(defaultno) which, when enabled, deploys tokube-system:ServiceAccount+ClusterRole(get/list/delete on pods) +ClusterRoleBindingCronJobrunning every 2 minutes that finds pods withstatus.phase=Failedandstatus.reason=UnexpectedAdmissionErrorand deletes them, automating exactly the manual recovery step described in the issueDesign notes
bitnami/kubectl:latestas a minimal kubectl-only image for the CronJobTest plan
openshift.io/media_a_rx_pooladmission failure (not available in this environment)