feat(observability): add the operator overview and conditions browser dashboards

[sourcehawk]

Towards #186

Summary

Adds the two Grafana dashboard templates and the Go template test.

observability/dashboards/crd_conditions_browser.tpl.json (uid crd_conditions_browser) ports the upstream go-crd-condition-metrics browser and improves it:

• {{namespace_label}} templated everywhere upstream hardcodes exported_namespace (namespace variable, table query, organize transformation keys).
• Stale-series protection on every condition query: <selector> and topk by (id, condition) (1, <selector without status/reason matchers>), so a former-leader pod still exporting old series cannot show a stale status. The topk keys on (id, condition) rather than (id) alone because the browser's queries can span multiple condition types (condition=All); keying on id alone would collapse each owner to its single most recently transitioned condition.
• The resource_id variable is made visible (label Resource) and wired as id=~"$resource_id" into the table and all four count stats, so the alerts' dashboard_url deep links (var-kind, var-condition, var-status, var-resource_id) actually narrow the view to the one owner.
• Coloured Status column and a Since column (dateTimeFromNow) in the table, column order Namespace, Name, Condition, Status, Reason, Since.
• A dashboard link to the operator overview, refresh: "", tags ["ocf"], and the ${datasource} variable on every panel, target and the ad hoc filter (upstream had one panel pinned to a private datasource uid).

observability/dashboards/ocf_operator.tpl.json (uid ocf_operator) is the operator overview: variables datasource, job, controller (multi, All), rows

• Overview: reconciles/s, reconcile error ratio, p99 reconcile time, p99 queue wait, active/max workers, leader status, owners Ready / not Ready > 2m (debounced on lastTransitionTime) / Unknown.
• Reconciliation: rate by result (stacked), error ratio (0.25 threshold line), latency p50/p90/p99, panics, max in-progress reconcile age, workers.
• Workqueue: depth, adds/s, queue wait p99 (300s threshold line), work duration p99, retries/s, unfinished work.
• Managed resources: apply rate by operation plus errors, updated-rate-per-resource table (sorted, colour-graded), not-converging ratio (the alert expression as a time series), apply error ratio per resource.
• Conditions: owners by condition and status (bar gauge, coloured by status), owners-not-Ready table with a data link per row into the conditions browser pre-filtered to that owner's kind and resource id.
• API client: request rate by method, non-2xx responses by code.
• Process (collapsed): CPU, RSS, goroutines.

observability/observability_test.go renders every dashboard and alert template the way the Makefile does and checks: valid JSON/YAML, no placeholder left behind, uid matches the file name, refresh off, every alert carries exactly one severity label with value warning or critical and no routing labels, every referenced metric name exists (framework collectors and the condition gauge are interrogated via prometheus.Collector.Describe, the controller-runtime/workqueue/client-go/process families are a fixed list), and every alert has a promtool unit test. A new make lint-dashboards target runs it and the CI observability job gained a Lint dashboards step.

Verification

make all, make test-alerts, make lint-dashboards and golangci-lint run ./... (pinned 2.12.2) all pass:

=== RUN   TestDashboards
    --- PASS: TestDashboards/crd_conditions_browser.tpl.json (0.00s)
    --- PASS: TestDashboards/ocf_operator.tpl.json (0.00s)
=== RUN   TestAlerts
    --- PASS: TestAlerts/controller_runtime.yaml (0.00s)
    --- PASS: TestAlerts/crd_conditions.tpl.yaml (0.00s)
    --- PASS: TestAlerts/managed_resources.yaml (0.00s)
ok  	github.com/sourcehawk/operator-component-framework/observability

Visual verification against the dev stack (make observability-up, simulator running for several minutes, checks through Grafana's HTTP API and Prometheus):

• GET /api/search lists both dashboards provisioned in folder OCF; GET /api/dashboards/uid/ocf_operator returns the model; POST /api/ds/query executes panel queries through the provisioned datasource.
• Every panel target and templating variable of both rendered dashboards was executed against Prometheus with $job=demo-operator, $controller=.*, $kind=Database, remaining variables .*, $__rate_interval=1m: all 37 overview targets and all 5 browser targets returned at least one series, and every query variable resolved values (job 1, controller 2, kind 2, condition 3, status 3, reason 3, namespace 2, resource_id 5).
• Overview spot checks: leader stat 1; owners Ready 52, not Ready > 2m reads 2 (shop/orders-db 8h old, analytics/reports-db), Unknown 1 (shop/users-db); the not-Ready table shows exactly those three rows with reasons Failing/Unknown and a Since in the past; the updated-rate table has webapp/server/configmap on top at 4/s; the workers query shows database saturated at 2 of 2; selecting controller=webapp empties the not-Ready table and the workqueue backlog while database shows a depth of 40.
• Browser spot checks: kind Database, condition Ready lists the five owners; id=~"shop/orders-db" narrows the table to that one owner, so the alert deep links work.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JPihvXVfS997iGmGabTGsd

[Copilot]

🟡 Changes recommended

Dashboard filters and aggregation keys currently produce incorrect results for namespaces and same-named resources.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Grafana dashboards for end-to-end operator health and CRD condition exploration.

Changes:

• Adds operator overview and conditions-browser dashboards.
• Adds template and metric-reference validation.
• Runs dashboard linting in CI.

File summaries

File	Description
.github/workflows/test.yml	Runs dashboard linting in CI.
Makefile	Adds lint-dashboards.
observability/dashboards/crd_conditions_browser.tpl.json	Adds the condition browser.
observability/dashboards/ocf_operator.tpl.json	Adds the operator overview.
observability/observability_test.go	Validates dashboard and alert templates.

Review details

Suppressed comments (3)

observability/dashboards/crd_conditions_browser.tpl.json:389

• The False count ignores the dashboard's namespace selection, so after choosing one namespace it still reports False conditions from all namespaces. Add the namespace matcher to both sides of the stale-series filter.

              "expr": "count(\n    {{operator_namespace}}controller_condition{kind=\"$kind\", condition=~\"$condition\", status=\"False\", reason=~\"$reason\", id=~\"$resource_id\"} > 0\n  and\n    topk by (id, condition) (1, {{operator_namespace}}controller_condition{kind=\"$kind\", condition=~\"$condition\"})\n)",

observability/dashboards/crd_conditions_browser.tpl.json:458

• The Unknown count ignores $namespace, so its value does not correspond to the namespace-filtered table. Apply the namespace matcher to both selectors.

              "expr": "count(\n    {{operator_namespace}}controller_condition{kind=\"$kind\", condition=~\"$condition\", status=\"Unknown\", reason=~\"$reason\", id=~\"$resource_id\"} > 0\n  and\n    topk by (id, condition) (1, {{operator_namespace}}controller_condition{kind=\"$kind\", condition=~\"$condition\"})\n)",

observability/dashboards/crd_conditions_browser.tpl.json:531

• The True count ignores $namespace, so selecting a namespace leaves this panel showing the global count. Apply the namespace matcher to both selectors.

              "expr": "count(\n    {{operator_namespace}}controller_condition{kind=\"$kind\", condition=~\"$condition\", status=\"True\", reason=~\"$reason\", id=~\"$resource_id\"} > 0\n  and\n    topk by (id, condition) (1, {{operator_namespace}}controller_condition{kind=\"$kind\", condition=~\"$condition\"})\n)",

• Files reviewed: 4/5 changed files
• Comments generated: 7
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sourcehawk]

Review round 1, disposition of the suppressed findings:

• crd_conditions_browser.tpl.json:389/458/531, the False/Unknown/True count stats ignore the namespace selection: applied in 2ecc7ba together with the posted comment on the Total stat; all four stats now filter by $namespace on both sides of the stale-series join, verified live against the dev stack.

[Copilot]

🟡 Changes recommended

The browser’s default “All” filters exclude cluster-scoped resources and create owner-count-dependent query sizes.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

observability/dashboards/crd_conditions_browser.tpl.json:713

• With no custom All value, Grafana expands “All” into every resource ID returned by this high-cardinality variable. The default dashboard therefore sends a regex whose size grows with the owner count on every panel query, which can become very large for production operators. Use the Prometheus wildcard as the All value so these queries stay constant-size.

        "includeAll": true,

• Files reviewed: 4/5 changed files
• Comments generated: 1
• Review effort level: Balanced

[sourcehawk]

Review round 2, disposition of the suppressed finding:

• crd_conditions_browser.tpl.json:713, All on the high-cardinality resource variable expands to an owner-count-sized regex: applied in 915896a. resource_id (and the other multi variables: namespace, condition, status, reason) now carry allValue: ".*", so All renders as a constant-size wildcard.

[Copilot]

🟡 Changes recommended

The lint permits critical severity despite the stated warning-only contract, and one dashboard description has a grammatical error.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

observability/dashboards/crd_conditions_browser.tpl.json:413

• Use “an” before “Unknown” in this user-visible panel description.

          "description": "Number of $kind $condition Conditions in a Unknown state",

• Files reviewed: 8/9 changed files
• Comments generated: 1
• Review effort level: Balanced

[sourcehawk]

Review round 3, disposition of the suppressed finding:

• crd_conditions_browser.tpl.json:413, "in a Unknown state": applied in 3256589. The three repeat-stat descriptions now read "Number of $kind $condition Conditions with status False / Unknown / True", sidestepping the article entirely.