Add the operator overview and conditions browser Grafana dashboards

[sourcehawk]

Add the operator overview and conditions browser Grafana dashboards

Problem

There is no dashboard that shows an operator built on the framework end to end: controller-runtime reconcile and workqueue health, what the framework did to each managed resource, and the condition state of the owners it manages. The upstream conditions browser covers conditions only, hardcodes exported_namespace, shows stale series from former-leader pods, and declares a resource variable it never uses.

Approach

• observability/dashboards/ocf_operator.tpl.json (uid {{operator_namespace}}ocf_operator): variables datasource, job, controller (multi). Rows: Overview stat tiles (reconciles/s, error ratio, p99 reconcile time, p99 queue wait, active/max workers, leader, owners Ready, owners not Ready for more than 2m, owners Unknown); Reconciliation; Workqueue; Managed resources (apply rate by operation, updated rate per component/resource, not-converging ratio and apply-error ratio grouped by the full metric identity); Conditions (owners by condition type and status, "Owners not Ready" table with reason, Since and coloured Status, data link into the browser); API client; Process (collapsed). Stat tiles run instant queries.
• observability/dashboards/crd_conditions_browser.tpl.json (uid {{operator_namespace}}crd_conditions_browser): carried over from upstream with {{namespace_label}} templated, a Since column and coloured Status column, the count stats filtered consistently with the table, the resource_id variable made visible and wired into the panels, wildcard All values so cluster-scoped owners stay visible, and a tag-based link to the overview.
• Dashboard uids are templated with the metric namespace so two operators' renders coexist in one Grafana; the alert dashboard_url annotations render with the same namespace and deep-link into that operator's browser.
• Every condition query joins on the freshest series per owner: topk by (kind, id) (1, ...), or topk by (kind, id, condition) when the query spans condition types.
• make lint-dashboards runs a Go test over the templates: valid JSON/YAML after rendering, uids, no leftover placeholders, every referenced metric name exists, every alert has a promtool test, exactly one severity label valued warning or critical; it runs in the CI observability job.

Verification

• make dashboards METRIC_NAMESPACE=demo renders both files and make lint-dashboards passes.
• On the local stack, every panel of both dashboards is populated by the simulator's world, the controller variable filters all rows consistently, the "Owners not Ready" table links into the browser pre-filtered to the clicked owner, the alert dashboard_url annotations resolve to the rendered browser uid, the browser's count stats track its namespace filter, and the cluster-scoped simulator owner appears under namespace All and disappears under a concrete namespace.
• Neither dashboard has auto-refresh enabled by default.

Context

Sub-issue of #183.

[sourcehawk]

Recording the decisions made during implementation (PR #190) that changed what this issue stated:

Before: the issue named fixed dashboard uids (ocf_operator, crd_conditions_browser) and the stale-series join as topk by (id) (1, ...) on every condition query.

Now: both uids are templated with the metric namespace (<namespace>_ocf_operator, <namespace>_crd_conditions_browser), and every cross-reference including the alert dashboard_url annotations renders with the same namespace. The freshness join carries the owner's kind: topk by (kind, id) for single-condition queries and topk by (kind, id, condition) for queries spanning condition types. The browser's resource_id variable is visible, wired into the panels, filtered by the preceding variables, and every multi variable answers All with the wildcard .* so cluster-scoped owners (whose series carry no namespace label) stay visible and query size stays constant. Stat panels run instant queries; the top-level cross-links resolve by the shared ocf tag.

Why: Grafana upserts by uid, so fixed uids made one operator's import overwrite another's in a shared Grafana, the same multi-operator collision the per-operator PrometheusRule names already avoid. Plain topk by (id) collapses same-named owners of different kinds (id is only <namespace>/<name>) and, with condition=All, an owner to a single condition. The upstream browser declared resource_id but never used it, so the alerts' deep links narrowed nothing.

Implemented in 23c69d2, 2ecc7ba and 915896a.