Ship Grafana dashboards and Prometheus alerts for operators built on the framework

[sourcehawk]

Ship Grafana dashboards and Prometheus alerts for operators built on the framework

Problem

Since #182 the framework records condition gauges (<metric_namespace>_controller_condition) and resource apply counters (ocf_resource_apply_total, ocf_resource_apply_errors_total), and every operator built on it also exposes controller-runtime's reconcile, workqueue, REST client and leader election metrics. Nothing in the repository turns those series into something an operator team can put in front of an on-call engineer: there is no dashboard showing an operator's health end to end, and no alert for the failure the apply counters were built to catch, a managed resource rewritten on every reconcile.

go-crd-condition-metrics ships a conditions browser dashboard and three condition alerts, but they cover conditions only, hardcode the exported_namespace label in the dashboard, and do not guard against stale series from former-leader pods.

In scope

• A make render pipeline under observability/ for the templated condition rules and both dashboards (keyed on the metric namespace and namespace label) and the shared cluster-wide rule files, with promtool unit tests, a Go template lint and CI.
• Alert rules: the three condition alerts carried over from go-crd-condition-metrics (per operator); ManagedResourceNotConverging and ManagedResourceApplyFailing on the framework's apply counters and ControllerReconcileErrors, ControllerReconcilePanics, ControllerWorkqueueBacklog, ControllerReconcileLatencyHigh, OperatorLeaderMissing on controller-runtime metrics (shared, installed once per cluster as ocf-managed-resources and ocf-controller-runtime).
• Two Grafana dashboards with metric-namespace-templated uids: an operator overview (controller-runtime, workqueue, framework apply metrics, condition summary, API client, process) and the per-resource conditions browser carried over and improved.
• A local Prometheus + Grafana stack driven by a Go simulator that uses the real pkg/metrics recorder, for visual confirmation of every panel and alert.
• Documentation: consumer-facing docs/observability.md, links from the metrics section of docs/component.md, README, plugin sync; observability/README.md as the maintainer guide.

Out of scope

• Delivering the artifacts through Go (embedding, a CLI render command).
• Jsonnet or Grafonnet.
• Platform alerts (target down, absent(), pod restarts, quotas) and SLO recording rules.
• Changes to go-crd-condition-metrics.
• Automated screenshot capture.

Risks & mitigations

• Alerts that are wrong at scale: a bare rate(updated) > 0 fires on legitimate churn across many owners. Mitigation: ratio-of-own-applies thresholds with an absolute floor, quantile thresholds on real histogram bucket bounds, and promtool tests for the negative cases.
• Simulator metric names drifting from controller-runtime's: mitigated by a unit test that starts a real unmanaged controller-runtime controller and asserts parity of names, labels and buckets.
• Dashboards silently breaking: mitigated by lint-dashboards in CI (valid JSON, no unrendered placeholders, expected uids, every referenced metric exists) and the visual pass on the local stack.
• Stale series from former-leader pods double counting owners: every condition query joins on the freshest series per owner (topk by (kind, id), or topk by (kind, id, condition) across condition types), alerts use max().
• Two operators in one cluster or one Grafana: shared rules install once under fixed ocf-* names, per-operator artifacts carry the metric namespace in their names and uids, and the shared rules aggregate by the operator's namespace.

Design overview

The condition rules and both dashboards are templates under observability/alerts/crd_conditions.tpl.yaml and observability/dashboards/*.tpl.json with two placeholders, identical to upstream so existing consumers change nothing: {{operator_namespace}} and {{namespace_label}}. The managed-resource and controller-runtime rules are plain shared files. make dashboards|alerts METRIC_NAMESPACE=x renders into the gitignored observability/generated/; ALERT_FORMAT=prometheusrule|rules picks between PrometheusRule objects (with optional PROMETHEUSRULE_NAMESPACE and PROMETHEUSRULE_LABELS metadata) and plain rule files. make test-alerts renders against a fixed test namespace and runs the promtool unit tests; make lint-dashboards runs the Go template test.

The operator overview dashboard filters controller-runtime (controller), workqueue (controller), framework (controller) and condition (controller) series with a single controller variable, so the documented guidance is: give metrics.NewRecorder the same name controller-runtime uses for the controller.

The local stack is docker compose (Prometheus scraping the host-run simulator with static job/namespace/pod target labels, so the exported namespace label collides into exported_namespace as in a cluster; Grafana provisioned with datasource, dashboards and rules, both bound to localhost). The simulator registers the real metrics.Collectors and condition gauge and plays a scripted world: healthy owners, one resource in a hot loop, a stuck NotReady owner, an Unknown owner, a cluster-scoped owner, failing applies, erroring reconciles, a backlogged workqueue.

flowchart LR
  subgraph templates[observability/]
    A[alerts/crd_conditions.tpl.yaml + shared *.yaml] --> R[make alerts / dashboards]
    D[dashboards/*.tpl.json] --> R
    T[alerts/tests/*.yaml] --> PT[make test-alerts]
    A --> PT
    A --> L[make lint-dashboards]
    D --> L
  end
  R --> G[generated/]
  G --> C[Consumer cluster: PrometheusRule + Grafana]
  G --> S[dev stack: docker compose]
  SIM[dev/simulator uses pkg/metrics] -->|scrape| S
  S --> V[Grafana on localhost: panels populated, alerts firing]

Loading

Sub-issues

Sub-issues will be linked below as they're filed.

[sourcehawk]

Recording how the shipped feature differs from the design overview below, so the integration PR reads against the truth:

Before: every rule file was a template rendered per metric namespace, dashboards had fixed uids, condition queries joined on topk by (id), and aggregations keyed on the operator's static topology only.

Now: only the condition rules are a template (crd_conditions.tpl.yaml, rendered as <metric-namespace>-crd-conditions); the managed-resource and controller-runtime rules are shared cluster-wide files (managed_resources.yaml, controller_runtime.yaml) wrapped once as ocf-managed-resources and ocf-controller-runtime, with PROMETHEUSRULE_NAMESPACE and PROMETHEUSRULE_LABELS for ruleSelector-based installs. Those aggregations also carry the operator's namespace so two installs of one operator alert separately. Thresholds sit on real histogram bucket bounds (ControllerReconcileLatencyHigh > 30, ControllerWorkqueueBacklog > 100). Dashboard uids are templated with the metric namespace and every condition query joins on topk by (kind, id) (or topk by (kind, id, condition) across condition types). docs/observability.md is consumer-facing; observability/README.md is the maintainer guide for the dev stack and the tests.

Why: review of the sub-PRs showed the original thresholds were unreachable against controller-runtime's buckets, per-operator rendering duplicated identical cluster-wide rules, fixed uids collide when two operators' renders share one Grafana, and id alone merges same-named owners of different kinds.

Implemented across 7cba942, 4f67035, 6b265e7 and cf3287c on the feature branch.