[Iristack]

What this PR does / why we need it:

Add an optional --job-label-selector flag to Training Operator V1 so an operator instance can reconcile a selected subset of training jobs within its watched namespaces. For example, --job-label-selector=training.example.org/controller=new selects jobs with that label and skips unlabeled jobs and jobs assigned to another controller. An empty selector preserves the existing behavior.

This targets release-1.9 because the implementation extends the V1 controllers, rather than the V2 Trainer APIs.

• Parse the Kubernetes label selector at startup and fail early on invalid syntax.
• Apply the selector to PyTorch, TensorFlow, MPI, PaddlePaddle, JAX, and XGBoost jobs, both before Create-event initialization and immediately after fetching a job in reconciliation.
• Keep dependent-resource watches and the cache unfiltered so later job label updates can trigger reconciliation.
• Add matching, exclusion, default-behavior, and invalid-startup tests. Include the operator command package in make test so CI runs its new test.

The selector is a reconciliation boundary, not a security or admission boundary. It does not change webhook scope, assign labels, or prevent another unfiltered operator from reconciling the same job. Deployments with multiple operators still need non-overlapping ownership and appropriate leader-election configuration. Changing labels on a running job is not an atomic ownership handoff.

Validation:

• Passed the Go package test set used by make test with Kubernetes 1.36.2 envtest binaries, plus the operator command tests.
• Passed go vet ./..., go mod tidy -diff, formatting/diff checks, and applicable pre-commit checks.
• Built Linux amd64 and arm64 images; checked flag availability and invalid-selector startup on both architectures.
• Ran CPU smoke tests on two-node k3s v1.35.5+k3s1 with Volcano v1.15.3 and the unmodified Arena CLI: selected single-node and cross-node PyTorchJobs completed; changing an excluded job's label triggered reconciliation and Volcano PodGroup creation; excluded jobs stayed untouched across operator restart; disabling the selector restored the default behavior in an earlier run.
• Verified that a synthetic running Pod and Service owned by an excluded job were preserved, explicit retry-enabled jobs recreated a deleted Pod, and job deletion cleaned up dependent resources. The default no-retry policy correctly reported failure instead of recreating the Pod.

The cluster smoke tests did not run a legacy operator concurrently, GPU/NCCL workloads, numerical PyTorch training, or Volcano preemption. The full upstream Kubernetes test matrix and generation/lint workflow have not been run locally.

Which issue(s) this PR fixes:

No linked issue.

Checklist:

• Docs included if any changes are user facing

The flag is user facing; an official documentation update is still outstanding. This PR does not add repository examples or Markdown documentation.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
• Our team will review your PR soon! cc @kubeflow/kubeflow-trainer-team

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏