[rhods-devops-app]

Automated Sync from main to stable

This PR automatically syncs the main branch to the stable branch by opening a pull request from main into stable.

Sync Summary

• Latest commit: 960c37e9 Use latest release trainer image in param.env

• Total commits to sync: 28

• Source: https://github.com/red-hat-data-services/trainer.git @ main

• Target: https://github.com/red-hat-data-services/trainer.git @ stable

• PR head: main

Commits to be synced

• 960c37e9 Use latest release trainer image in param.env
• 3a8cdb43 Update registry.access.redhat.com/ubi9/go-toolset Docker digest to edb9452 (Update registry.access.redhat.com/ubi9/go-toolset Docker digest to edb9452 (main) #326)
• d2182df2 Merge remote-tracking branch 'upstream/main'
• be9d7345 feat: add RHOAI Trainer recording prometheus rules (Update registry.access.redhat.com/ubi9/go-toolset Docker digest to 60fd6fb (rhoai-3.4) #307)
• 932a5dc2 Update registry.access.redhat.com/ubi9/go-toolset Docker digest to d4d25f2 (Update registry.access.redhat.com/ubi9/go-toolset Docker digest to d4d25f2 (main) #321)
• eb414082 Merge remote-tracking branch 'upstream/main'
• 95f74116 Merge pull request Update registry.access.redhat.com/ubi9/go-toolset Docker digest to 68d6b72 (rhoai-3.3) #306 from abhijeet-dhumal/cherrypick-trainer-metrics
• d921c42a fix: align Makefile.rhai deploy namespace with rhoai manifests
• 29de853d fix: render RHOAI Trainer manifests into redhat-ods-applications
• 25d7656f feat: add Trainer control-plane metrics
• d1e441c5 Merge remote-tracking branch 'upstream/main'
• 8ec7ae5f Merge pull request Update Dockerfile Digest Updates (rhoai-3.3) #298 from opendatahub-io/midstream-merge
• 51a62409 fix: generate
• 5beacbd1 Merge branch 'upstream-master' into midstream-merge
• d5a71618 Merge remote-tracking branch 'upstream/main'
• 9f0fdcf1 chore: sync security config files from security-config
• 8171102f Update registry.access.redhat.com/ubi9/go-toolset Docker digest to 68d6b72 (Update registry.access.redhat.com/ubi9/go-toolset Docker digest to 68d6b72 (main) #313)
• 18965b90 chore(test): add unit tests for the status server token authorizer (chore(test): add unit tests for the status server token authorizer kubeflow/trainer#4200)
• c53a08e2 chore: update controller tools to v0.23 and drop crdschema hack (chore: update controller tools to v0.23 and drop crdschema hack kubeflow/trainer#4249)
• d67877f8 chore(deps): bump golang.org/x/crypto from 0.57.0 to 0.58.0 in the golang group (chore(deps): bump golang.org/x/crypto from 0.57.0 to 0.58.0 in the golang group kubeflow/trainer#4223)
• 204404aa chore(test): add unit tests for TrainingRuntime and ClusterTrainingRuntime update validation (chore(test): add unit tests for TrainingRuntime and ClusterTrainingRuntime update validation kubeflow/trainer#4165)
• d7855703 fix(runtimes): install the CUDA 13 build of MLX in the MLX runtime image (fix(runtimes): install the CUDA 13 build of MLX in the MLX runtime image kubeflow/trainer#4246)
• 25538c30 fix(manifests): rename serviceMonitor value to prometheus to align with JobSet (fix(manifests): rename serviceMonitor value to prometheus to align with JobSet kubeflow/trainer#4244)
• 74f96ca2 feat(api): KEP-3015: Workload Aware Scheduling for TrainJob (feat(api): KEP-3015: Workload Aware Scheduling for TrainJob kubeflow/trainer#3219)
• 1bbebb15 chore(runtimes): add ValidateObjects test coverage for TrainingRuntime and ClusterTrainingRuntime (chore(runtimes): add ValidateObjects test coverage for TrainingRuntime and ClusterTrainingRuntime kubeflow/trainer#3892)
• 0edd89bf fix(runtimes): emit PRTE env vars so MPI plugin works with Open MPI 5 (fix(runtimes): emit PRTE env vars so MPI plugin works with Open MPI 5 kubeflow/trainer#4029)
• 0af9dc16 fix(controller): break ties when picking the best OptimizationJob trial (fix(controller): break ties when picking the best OptimizationJob trial kubeflow/trainer#4192)
• 619d75c0 feat(ci): add Prometheus monitoring overlay (feat(ci): add Prometheus monitoring overlay kubeflow/trainer#4102)

Merging

GitHub automerge is enabled for this pull request once required checks pass.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: bb7ee3d5-855e-4bd1-b791-0d37c0f76443

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon! cc @kubeflow/kubeflow-trainer-team

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[coveralls]

Coverage Report for CI Build 40062914952

Coverage increased (+2.0%) to 73.706%

Details

• Coverage increased (+2.0%) from the base build.
• Patch coverage: 81 uncovered changes across 2 files (311 of 392 lines covered, 79.34%).
• 1 coverage regression across 1 file.

Uncovered Changes

File	Changed	Covered	%
pkg/metrics/instrumentation.go	355	278	78.31%
pkg/metrics/setup.go	4	0	0.0%
Total (4 files)	392	311	79.34%

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File	Lines Losing Coverage	Coverage
pkg/metrics/setup.go	1	0.0%

---

Coverage Stats

Relevant Lines:	5853
Covered Lines:	4314
Line Coverage:	73.71%
Coverage Strength:	0.85 hits per line

---

💛 - Coveralls

[ckhordiasma]

@rhods-devops-app[bot]: The following test has Succeeded:

Test Results

47 passed, 0 failed, 4 skipped

View test details

Test	Result	Duration
[BeforeSuite]	✅ PASS	0.001001401s
[It] OptimizationJob e2e when Creating OptimizationJob with random search algorithm should provision algorithm service, run trials to completion, record best result, and perform cleanup	⏭️ SKIP	0s
[It] OptimizationJob e2e when Creating OptimizationJob with random search algorithm should fail if unsupported search algorithm is configured	⏭️ SKIP	0s
[It] TrainJob e2e when Creating TrainJob to perform the PyTorch workload should create TrainJob with PyTorch runtime reference	✅ PASS	295.179261792s
[It] TrainJob e2e when Creating TrainJob to perform OpenMPI workload should create TrainJob with DeepSpeed runtime reference	⏭️ SKIP	0.049250157s
[It] TrainJob e2e when Creating TrainJob to perform JAX workload should create TrainJob with JAX runtime reference	✅ PASS	134.650241583s
[It] TrainJob e2e when Creating TrainJob to perform XGBoost workload should create TrainJob with XGBoost runtime reference	✅ PASS	45.190424436s
[It] TrainJob e2e when Creating TrainJob to perform Flux workload should create TrainJob with Flux runtime reference	✅ PASS	28.723053508s
[It] TrainJob e2e when Creating a TrainJob with RuntimePatches should preserve user-provided manager fields	✅ PASS	0.181247842s
[It] TrainJob e2e when Creating a TrainJob with RuntimePatches should update Time only for the changed patch on update	✅ PASS	1.189104889s
[It] TrainJob e2e when Creating a TrainJob managed by an external controller should not be reconciled by the built-in controller	✅ PASS	15.091893947s
[It] TrainJob e2e when Creating a TrainJob with Resource Timeouts should fail the TrainJob with DeadlineExceeded when active timeout expires	✅ PASS	9.93787461s
[It] TrainJob e2e when Creating TrainJob with runtime status server instrumentation should inject runtime configuration which allows the runtime status endpoint to be called	⏭️ SKIP	0s
[It] TrainJob e2e when Updating a runtime a suspended TrainJob should use the original runtime configuration and not pick up the new configuration TrainingRuntime	✅ PASS	1.221410193s
[It] TrainJob e2e when Updating a runtime a suspended TrainJob should use the original runtime configuration and not pick up the new configuration ClusterTrainingRuntime	✅ PASS	1.248210462s
[BeforeSuite]	✅ PASS	0.10837682s
[It] RHAI Progression Tracking E2E Tests When progression tracking is enabled should poll metrics and update trainerStatus annotation during training	✅ PASS	80.765965495s
[It] RHAI Progression Tracking E2E Tests When progression tracking is NOT enabled should NOT create trainerStatus annotation	✅ PASS	35.100326574s
[It] RHAI Progression Tracking E2E Tests When progression tracking annotation has invalid value should NOT enable progression tracking for non-'true' values	✅ PASS	15.102670423s
[It] RHAI Progression Tracking E2E Tests When metrics polling configuration is customized should honor custom metrics port and poll interval	✅ PASS	4.146331544s
[It] RHAI Progression Tracking E2E Tests When metrics polling configuration is customized should handle minimum SDK-recommended poll interval (5s)	✅ PASS	0.112406797s
[It] RHAI Progression Tracking E2E Tests When metrics polling configuration is customized should handle maximum SDK-recommended poll interval (300s)	✅ PASS	0.11182116s
[It] RHAI Progression Tracking E2E Tests When metrics polling configuration is customized should use default interval when annotation is missing	✅ PASS	0.107984034s
[It] RHAI Progression Tracking E2E Tests When TrainJob fails during training should capture final status even when job fails	✅ PASS	72.671204144s
[It] RHAI Progression Tracking E2E Tests When metrics endpoint is unreachable should handle connection errors gracefully without crashing	✅ PASS	10.452931969s
[AfterSuite]	✅ PASS	0.016162316s
TestOpenMPICudaClusterTrainingRuntime	✅ PASS	0.030000s
TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes	✅ PASS	0.030000s
TestDraFieldsOnTrainJobReachPods	✅ PASS	1.270000s
TestRunTrainJobWithDefaultClusterTrainingRuntimes	✅ PASS	407.970000s
TestJobSetWorkflow	✅ PASS	97.680000s
TestKubeflowSdk	✅ PASS	122.250000s
TestKubeflowSdkKueueIntegration	✅ PASS	47.890000s
TestKueueWorkloadPreemptionSuspendsTrainJob	✅ PASS	3.490000s
TestKueueSchedulingReachesTrainJobPods	✅ PASS	1.390000s
TestMultiNodeOpenMPICpuTrainJob	✅ PASS	32.830000s
TestFailedJobSetWorkflow	✅ PASS	12.480000s
TestRhaiTrainingProgressionCPU	✅ PASS	443.900000s
TestRhaiJitCheckpointingCPU	✅ PASS	542.560000s
TestRhaiFeaturesCPU	✅ PASS	522.480000s
TestRhaiS3CheckpointingCPU	✅ PASS	533.110000s
TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28	✅ PASS	73.580000s
TestKueueWorkloadInadmissibleWithNonExistentLocalQueue	✅ PASS	1.200000s
TestTrainerSecureServing	✅ PASS	0.680000s
TestTrainerPrometheusScrape	✅ PASS	0.350000s
TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics	✅ PASS	8.340000s
TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth	✅ PASS	8.340000s
TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer	✅ PASS	8.370000s
TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort	✅ PASS	6.240000s
TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort	✅ PASS	11.490000s
TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort	✅ PASS	11.400000s

OCI Artifact Browser URL

View in Artifact Browser

Inspecting Test Artifacts Manually

To inspect your test artifacts manually, follow these steps:

1. Install ORAS (see the ORAS installation guide).
2. Download artifacts with the following commands:

mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/opendatahub/odh-ci-artifacts:odh-pr-test-trainer-rnt9z