[XploY04]

What this PR does / why we need it:

Implements the Trainer portion of KEP-2899 Phase 2 (#3824). Finished TrainJobs can be deleted after a configured TTL, together with their dependent resources, and runtimes can provide lifecycle defaults.

• Adds mutable TrainJob.spec.ttlSecondsAfterFinished and status.startTime. TTL uses the Complete=True or Failed=True condition's LastTransitionTime; deadlines use the active window, cleared on suspension and reset on resume.
• Adds spec.runPolicy to both runtime kinds. Each explicit TrainJob field overrides its matching snapshot default. Rejects conflicting JobSet/Job TTL and Job/Pod deadline settings in runtime templates.
• Uses foreground TTL deletion with UID/resource-version preconditions. Deadline cleanup checks JobSet ownership and retries deletion without moving the terminal timestamp.
• Applies explicit TTL to finished externally managed TrainJobs. Manager-side runtime defaults do not apply because those objects have no runtime snapshot.
• Defensively restores absent snapshot type metadata while preserving identity validation. Updates generated APIs, CRDs, RBAC, documentation, and tests.

For review: TTL is skipped for TrainJobs controlled by an OptimizationJob. Its controller counts existing trials and derives results from them, so deleting finished trials can cause replacements and discard results. The KEP does not explicitly cover this interaction.

Upgrade notes: remove the forbidden lifecycle fields from existing runtime templates before upgrading. Unfinished jobs without snapshots capture live runtime defaults after upgrade; a newly inherited deadline can already be overdue. Finished jobs without snapshots need an explicit TrainJob TTL.

MultiKueue users should leave TrainJob TTL unset until their Kueue version clears TTL on worker copies (kubernetes-sigs/kueue#14779), and keep worker-runtime TTL defaults unset.

Validation:

• Go build, vet, changed-area lint, API lint, all 24 unit-test packages, and controller/runtime-core race checks.
• All 205 integration specs, 98 Helm tests, and three lifecycle e2e cases: successful completion, exhausted JobSet retries, and active deadline cleanup.
• Local API/error probes covering CEL create/update for both runtime kinds, externally managed TTL, snapshot isolation, and overdue cleanup after controller restart.

An unchanged integration cache-timing test passed on the complete rerun. Full two-cluster MultiKueue behavior has not been tested.

Which issue(s) this PR fixes:

Part of #4195. SDK exposure and the Kueue worker-copy change are separate follow-ups.

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment