Repository navigation
[release-1.9] Bump v1 Kubernetes dependencies to 1.36 - #3924
Merged
Merged
Conversation
Signed-off-by: Rob Bell <robell@redhat.com>
Signed-off-by: Rob Bell <robell@redhat.com>
Repair `make generate` so it runs end to end against the upgraded dependencies, and check in the regenerated assets: - defaulter-gen now emits imports for every package it traverses, which leaves an unused corev1 import in zz_generated.defaults.go. Prune the generated output with a pinned goimports pass. - gen-sdk.sh passed a relative repo root to wget, which wget2 rejects. Resolve the repo root with realpath, matching update-codegen.sh. - crd-ref-docs v0.1.0 pins x/tools v0.19.0, which does not build on Go 1.26. Bump it to v0.3.0. Signed-off-by: Rob Bell <robell@redhat.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Bump cert-controller to v0.16.0; v0.12.0 calls the removed controller.NewUnmanaged signature. - Port the webhook setup functions to the controller-runtime v0.24 generic builder: NewWebhookManagedBy now takes the object and WithCustomValidator replaces For().WithValidator(). - Use Event() rather than Eventf() where no format arguments are supplied, which go vet now rejects. Signed-off-by: Rob Bell <robell@redhat.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Bump the envtest Kubernetes version and setup-envtest branch to match controller-runtime v0.24. - client-go now negotiates protobuf by default, so the pod and service control tests captured a protobuf request body where they expected JSON. Pin the content type to JSON in the test client config. - The fake clientset now stamps a resource version onto list results, so ignore ListMeta.ResourceVersion when diffing. Signed-off-by: Rob Bell <robell@redhat.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hack/swagger is a separate module and was still pinned to k8s 0.31 and Go 1.23. Align it with the main module so the CI `go mod tidy` check stays clean. The generated swagger.json is unchanged. Signed-off-by: Rob Bell <robell@redhat.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The old 1.28-1.31 matrix predates the upgraded controller-runtime, whose setup-envtest release branch no longer publishes those binaries. Signed-off-by: Rob Bell <robell@redhat.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
golangci-lint v1.61.0 predates Go 1.26 and cannot parse it. Bump to v1.64.8 and build it from source, since the released binaries are built with an older Go than the --go target and reject it. Pin it into bin/ the way controller-gen and kustomize already are; the previous `which golangci-lint` guard silently skipped the pin whenever any version happened to be on PATH. The upgraded dependencies deprecate several APIs still in use here, so exclude SA1019 in a new .golangci.yml, which documents each one. They all still function and migrating them is a behavioural refactor rather than a dependency bump. Signed-off-by: Rob Bell <robell@redhat.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Commit 378a9e6 ("[bug] pull image from ghcr", kubeflow#2584) renamed the operator image in manifests/base/deployment.yaml from kubeflow/training-operator to ghcr.io/kubeflow/training-v1/training-operator, but left the `kustomize edit set image` calls in the deploy target and in the CI setup script pointing at the old name. Kustomize silently ignores an image override that matches nothing, so both kept deploying the published image rather than the locally built one — meaning CI has been exercising the released operator instead of the build under test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Rob Bell <robell@redhat.com>
helm/kind-action@v1.11.0 bundles a kind that has no node images newer than v1.31, so raising the tested Kubernetes versions means raising the action too. v1.14.0 defaults to kind v0.31.0, which publishes v1.31.14 through v1.35.0; the matrices move to the newest three of those. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Rob Bell <robell@redhat.com>
The Python SDK's api_client.py is swagger-codegen output and resolves model
types by swagger-codegen name. kubernetes 36.0.0 regenerated its models with
openapi-generator, which spells map types "dict[str, str]" rather than
"dict(str, str)", so every deserialisation of an object with labels or
annotations fails with
AttributeError: module 'kubeflow.training.models' has no attribute 'dict[str, str]'
The dependency was unbounded, so a fresh install picked up 36.x and the whole
e2e suite failed. Cap it below 36 until the client is regenerated.
Separately, actions/setup-python no longer provides 3.8 on current
ubuntu-latest runners, so the matrix entries pinning 3.8 and 3.9 cannot run.
Move the workflows onto 3.10 and 3.11, matching the set test-python.yaml
already exercises, and drop the corresponding setup.py classifiers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Rob Bell <robell@redhat.com>
Member
Author
|
/retest |
The pytorch-torchrun-cpu image build has been failing. Nothing in the
example changed; its dependencies were unpinned and the resolution
drifted. torch 2.4.1 (the last cp38 build) requires typing-extensions
>=4.8.0, but no current typing-extensions publishes a cp38 wheel, so pip
falls back to the sdist. Building that sdist needs flit_core, and since
--index-url replaces PyPI rather than adding to it, pip only looks at
download.pytorch.org and fails with:
No matching distribution found for flit_core<4,>=3.11
Pin to the versions the v1.9.3 image (c77ee3f) actually shipped rather
than bumping the base image, so the 1.9.4 patch release publishes the
same example it always has.
torch is pinned without the +cpu local version because the two published
architectures differ: amd64 resolves 2.4.1+cpu and arm64 resolves 2.4.1.
The bare pin matches both. Verified by building for linux/amd64 and
linux/arm64 and running demo.py.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Rob Bell <robell@redhat.com>
The 9 integration-test jobs each pass 8/9 tests and fail only test_e2e_jaxjob, which times out at the 900s wait_timeout. The failure is identical across all three Kubernetes versions and all three gang schedulers, so it is not related to the dependency bump. Nothing in the example changed; its dependencies were unpinned. The image is built from source on every run, and "pip install --upgrade jax[k8s]" now resolves jax 0.11.0 where the v1.9.3 image (c77ee3f) shipped 0.6.2. Running the workload under the same 1.2 CPU limit the test applies, 0.6.2 completes an epoch in under a second while 0.11.0 does not finish initialising. Pin to the v1.9.3 versions rather than adapting the example, so the 1.9.4 patch release runs what it has always run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Rob Bell <robell@redhat.com>
Member
Author
|
Couple of other minor fixes to highlight --
In both case I think the root cause is the deps in these images aren't pinned and they were last built >12 months ago so we're now pulling in much more recent versions. Since we're targeting a patch release, I've just pinned the dep versions to whatever was present in 1.9.3 rather than trying to update to the latest versions -- see 75535ea ( Happy to use a different approach if preferred. |
Member
Author
|
The Read the Docs build is failing because there's no RTD config file in this branch. I'm thinking that probably doesn't need fixing, but happy to address it if someone can provide guidance. |
Member
Author
|
@andreyvelich this is ready for review. PTAL. We can look to cut a 1.9.4 once we're happy with it. /assign @andreyvelich |
Member
Author
robert-bell
commented
Aug 16, 2026
Comment on lines
+15
to
+25
| k8s.io/api v0.36.3 | ||
| k8s.io/apimachinery v0.36.3 | ||
| k8s.io/client-go v0.36.3 | ||
| k8s.io/code-generator v0.36.3 | ||
| k8s.io/klog/v2 v2.140.0 | ||
| k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad | ||
| k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3 | ||
| sigs.k8s.io/controller-runtime v0.24.1 | ||
| sigs.k8s.io/scheduler-plugins v0.34.7 | ||
| sigs.k8s.io/structured-merge-diff/v6 v6.4.2 | ||
| sigs.k8s.io/yaml v1.6.0 |
Member
Author
Member
Author
|
/retest |
Kubernetes 1.33 reached end of life in June 2026, and the dependency bump moves scheduler-plugins from v0.28.9 to v0.35.4-devel. scheduler-plugins releases track a Kubernetes minor one-for-one and embed a full kube-scheduler, so the version CI deploys is built against 1.34 and is not supported against a 1.33 API server. The effect is that every gang-scheduling test hangs on that combination: all six pods stay Pending for the full 900s timeout with no FailedScheduling event, because the scheduler never acts on them. The scheduler-plugins deployment itself becomes ready, so the job runs for over 90 minutes before failing. Only 1.33 with scheduler-plugins is affected; 1.33 with none and volcano still pass, as do the unit tests. Drop 1.33 everywhere rather than keep a permanently red cell in the matrix, since it is out of support anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Rob Bell <robell@redhat.com>
Contributor
|
LGTM |
andreyvelich
left a comment
andreyvelich
left a comment
Member
There was a problem hiding this comment.
Looks great, thank you for working on this @robert-bell!
/lgtm
/approve
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andreyvelich The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Member
Author
|
Thanks @andreyvelich. Can we merge this manually? The RTD ci check isn't going to pass. |
Member
|
Sure! |
Member
Author
|
Thanks @andreyvelich, could you create a patch release for 1.9.4? I took a look at the release docs and I think I'm not able to do any of it myself as it needs write permissions. |
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Updates k8s.io/* v0.31.3 → v0.36.3, controller-runtime v0.19.1 → v0.24.1, and Go 1.23 → 1.26. Dependency upgrade only - no behavioural changes intended.
Kueue consumes this repo as a library and has asked for the bump. kubernetes-sigs/kueue#12620
Kueue is already on Go 1.26 / k8s v0.36.3 / CR v0.24.1 while pinning training-operator v1.9.3, so this removes an existing mismatch. Kueue imports exactly one package (pkg/apis/kubeflow.org/v1), and the only change to it here is regenerated openapi -- no type or signature changes.
PR is split into distinct commits for easier review.
List of changes:
Just to flag: CI has not been testing the built image
#2584 renamed the image in manifests/base/deployment.yaml to ghcr.io/kubeflow/training-v1/training-operator, but left kustomize edit set image in scripts/gha/setup-training-operator.sh and the Makefile deploy target pointing at the old kubeflow/training-operator. Kustomize silently ignores an override that matches nothing, so every integration and e2e run since Apr 2025 has tested against the published image rather than the PR build. Fixed in 3de3ce1.
This means this PR is the first run in a while where e2e actually tests the code under review -- if something red appears, it may be pre-existing rather than caused by the upgrade.
Testing
make generate (idempotent), build, vet, gofmt, make test and make golangci-lint all green. Deployed to local KinD cluster and ran the SDK e2e suite: all six job types reconcile correctly; JAX passed end-to-end; other tests were slow so I'm delegating to CI. Will fix if any issues are found.