[abhijeet-dhumal]

What this PR does / why we need it

Adds six upstream Trainer Prometheus metrics for runtime inventory, TrainJob creation, completion, failure, lifetime, and requested training-node counts proposed in #3429:

Metrics are registered only by the elected controller leader and collected from informer-backed snapshots without Kubernetes API calls during scraping. The implementation also handles duplicate terminal events, runtime updates/deletions, and leader transitions.

TLS, authentication, and ServiceMonitor configuration are handled separately by PR #4102.

• kubeflow_trainer_runtime_info: current TrainingRuntime and ClusterTrainingRuntime inventory.
• kubeflow_trainer_trainjob_created_total: TrainJobs observed after the initial informer cache listing.
• kubeflow_trainer_trainjob_finished_total: terminal outcomes by result, reason, and runtime.
• kubeflow_trainer_trainjob_lifetime_seconds: time from TrainJob creation to terminal state.
• kubeflow_trainer_trainjob_requested_training_nodes_per_job: requested training-node size.

The controller uses its existing informer caches instead of calling the Kubernetes API during metric collection. Runtime information is kept in an in-memory snapshot and reused when processing TrainJobs.
Only the elected controller leader processes metric events and registers the Trainer metrics. When leadership changes, the old leader removes its metrics and the new leader registers them. This prevents duplicate metrics from multiple controller replicas.

The implementation also:

• avoids counting TrainJobs already present when the cache starts;
• records terminal metrics only once per TrainJob;
• avoids high-cardinality labels such as TrainJob names and UIDs;
• includes unit and race-test coverage;
• documents metric behavior, PromQL examples, and limitations.

Validation

• Unit tests for runtime inventory, TrainJob events, runtime policy resolution, terminal-state handling, and metric lifecycle.
• Race tests for the metrics package.
• go vet ./pkg/metrics/...
• Formatting and git diff --check.
• Existing repository code-quality, generation, and language test checks.
• Prometheus metrics architecture and monitoring documentation.

Follow-up

Final CI, documentation build, approver review, and cluster E2E validation remain required before merge.

Checklist

• Docs included for the user-facing metrics and monitoring changes
• Unit tests added
• Leader ownership and registration lifecycle covered

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[robert-bell]

Thanks @abhijeet-dhumal! I've left some inline feedback.

[abhijeet-dhumal]

Thanks @robert-bell .. I’ve tried to address the metric scope and test coverage updates.

[robert-bell]

Thanks @abhijeet-dhumal this is looking good! I've left a few minor suggestions but this is looking pretty close.

[robert-bell]

@abhijeet-dhumal I ran the new PR review skill over this - can you check if these any of these comments are valid?

[andreyvelich]

Thanks @abhijeet-dhumal for the work on this! Review done using AI tools; inline comments below.

[andreyvelich]

/milestone v2.4

[robert-bell]

Thanks @abhijeet-dhumal this is looking pretty close!

[robert-bell]

Can we log that the instrumentation has been configured correctly?

Suggested change

	<-ctx.Done()
	m.log.Info("trainer-metrics instrumentation ready")
	<-ctx.Done()

[robert-bell]

Same here - can we log that the instrumentation has been configured correctly?

Suggested change

	<-ctx.Done()
	c.log.Info("runtime-metrics instrumentation ready")
	<-ctx.Done()

[robert-bell]

nit: can we move this logger into the runtimeInfoCollector and use a different name?

return &runtimeInfoCollector{
		cache:    c,
        log:      ctrl.Log.WithName("runtime-metrics-collector")
		snapshot: make(map[string]runtimeInfo),
	}

[robert-bell]

nit: same as https://github.com/kubeflow/trainer/pull/4190/changes#r4338364858 - use a logger on the runtimeInfoCollector