feat(observability): add a local Prometheus and Grafana stack fed by a metrics simulator

[sourcehawk]

Towards #185

Summary

Adds a local Prometheus + Grafana stack so the alert rules (and, once #186 lands, the dashboards) can be watched against realistic data.

• Compose stack (observability/dev/): prom/prometheus:v3.14.0 on :9090 and grafana/grafana:13.1.4 on :3000 (anonymous admin). Prometheus loads the dev-rendered rules and scrapes the simulator on the host with static target labels namespace: operators, pod: demo-operator-0, so the condition gauge's own namespace label arrives as exported_namespace, exactly as a ServiceMonitor scrape does in a cluster. Grafana provisions the Prometheus datasource (uid prometheus) and a file provider over observability/generated/dev/dashboards, which is empty until Add the operator overview and conditions browser Grafana dashboards #186.
• Dev render (make observability-render-dev): renders dashboards and plain-format rules for metric namespace demo into observability/generated/dev/, then shortens every for: to 2m so alerts fire within minutes. make observability-up renders, starts the containers and runs the simulator in the foreground; make observability-down stops the containers. SIMULATOR_ARGS passes extra flags, e.g. SIMULATOR_ARGS="-leader=false".
• Simulator (observability/dev/simulator/): a Go binary serving /metrics with the real pkg/metrics recorder and ocm.NewOperatorConditionsGauge("demo") for the framework series, plus hand-built controller-runtime, workqueue, REST client and leader election lookalikes. Flags: -listen (:8080), -metric-namespace (demo), -leader (true).

The scripted world, scenario by scenario:

Scenario	Controller	What it plays	Alert it exercises
healthyWebApps	webapp	50 owners across team-a..team-e, all converged, Ready=True	none (control group)
hotLoop	webapp	webapp-01's configmap rewritten 4x/s (operation="updated")	ManagedResourceNotConverging
databases	database	shop/orders-db Ready=False for 8h; shop/users-db Ready=Unknown; analytics/reports-db Ready=False with the reason flipping between Failing and Creating; PVC applies failing 3 of 4; 30% reconcile errors; p99 reconcile duration above 30s	CustomResourceConditionStuck, CustomResourceNotReady, CustomResourceConditionUnknown, ManagedResourceApplyFailing, ControllerReconcileErrors, ControllerReconcileLatencyHigh
workqueueBacklog	database	queue depth 30-50, queue wait observations of 200-600s	ControllerWorkqueueBacklog
panics	database	one panic a minute in, then every 20 minutes	ControllerReconcilePanics
-leader=false	-	leader_election_master_status 0	OperatorLeaderMissing

• Parity test (runtime_test.go): starts a real unmanaged controller-runtime controller, drives one request through its queue, and asserts every lookalike family exists in controller-runtime's registry with the same type, label names and histogram buckets. rest_client_* and leader_election_* are skipped when absent from the real registry, since client-go and the leader elector create them lazily and neither runs without an API server.

Verification

• make all and go test ./observability/... pass, including the parity test (TestRuntimeMetricsMatchControllerRuntime PASS).
• make observability-up, then:

  • curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[].health' returned "up".

  • curl -s localhost:9090/api/v1/rules | jq '[.data.groups[].name]' returned ["controller-runtime","crd-conditions","managed-resources"].

  • curl -s localhost:9090/api/v1/query --data-urlencode 'query=demo_controller_condition{condition="Ready",status="False"}' returned series with exported_namespace "analytics" and "shop", proving the target-label collision renames the gauge's namespace label the way a ServiceMonitor scrape does.

  • Within ten minutes the firing set was exactly the nine expected alerts:

    ["ControllerReconcileErrors","ControllerReconcileLatencyHigh","ControllerReconcilePanics",
     "ControllerWorkqueueBacklog","CustomResourceConditionStuck","CustomResourceConditionUnknown",
     "CustomResourceNotReady","ManagedResourceApplyFailing","ManagedResourceNotConverging"]
    

  • The only firing alert with controller="webapp" was ManagedResourceNotConverging; everything else carried controller="database".

  • Rerunning the simulator with -leader=false made OperatorLeaderMissing fire ({name="demo-operator", namespace="operators"}) after its shortened for: clock.

• Grafana answered /api/health with "ok" while the provisioned dashboards directory was still empty.
• make observability-down removed both containers and the network.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JPihvXVfS997iGmGabTGsd

[Copilot]

🟡 Changes recommended

There are a few verified issues that should be fixed before approval (misleading docs reference, scenario comment mismatch, and safer localhost-only port bindings for an anonymous-admin Grafana dev stack).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a local observability development stack (Prometheus + Grafana) plus a Go-based metrics simulator to generate realistic controller-runtime and framework metrics for validating alert rules (and future dashboards) without a live cluster.

Changes:

• Added a host-run simulator (observability/dev/simulator/) that emits framework metrics (via pkg/metrics) and controller-runtime/workqueue lookalike families guarded by a parity test.
• Added a docker-compose based local Prometheus+Grafana setup under observability/dev/ wired to scrape the simulator and load dev-rendered rules/dashboards.
• Added Makefile targets to render dev assets and run/stop the local stack; updated Go module deps for the new test.

File summaries

File	Description
observability/dev/simulator/world.go	сценарий-генератор “мира” (webapp/database) для условий, apply-метрик и runtime-метрик
observability/dev/simulator/runtime.go	определение lookalike метрик controller-runtime/workqueue/rest_client/leader_election
observability/dev/simulator/runtime_test.go	parity-тест, сверяющий семейства метрик с реальным controller-runtime registry
observability/dev/simulator/main.go	HTTP /metrics сервер и запуск сценариев симулятора
observability/dev/prometheus/prometheus.yml	конфигурация Prometheus для scrape симулятора и загрузки rule_files
observability/dev/grafana/provisioning/datasources/prometheus.yaml	provisioning datasource Prometheus (uid prometheus)
observability/dev/grafana/provisioning/dashboards/ocf.yaml	provisioning file-provider для dashboards каталога
observability/dev/docker-compose.yaml	compose стек Prometheus+Grafana для локальной разработки
Makefile	цели observability-render-dev, observability-up, observability-down и dev-render логика
go.mod	добавлена прямая зависимость github.com/prometheus/client_model для тестов

Review details

Suppressed comments (1)

observability/dev/docker-compose.yaml:20

• Grafana is configured for anonymous admin access; publishing it on all interfaces can expose an admin UI beyond localhost. Binding the port to 127.0.0.1 keeps the convenience while reducing risk.

    ports:
      - "3000:3000"

• Files reviewed: 10/10 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sourcehawk]

Review round 1, disposition of the suppressed finding:

• observability/dev/docker-compose.yaml:20, anonymous-admin Grafana published on all interfaces: applied in 36f1de0, both Grafana and Prometheus now bind to 127.0.0.1 only.

[Copilot]

🔵 Needs a closer look

The active-worker metric is effectively unsampleable, and the scripted scenario behavior lacks automated coverage.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

observability/dev/simulator/world.go:74

• activeWorkers is reset to zero immediately after recording a synthetic duration; no work is actually held for duration. Since Prometheus scrapes every five seconds, this gauge will almost always report zero even while the simulator is generating continuous reconciles, so the active-workers dashboard signal is misleading. Model a stable/concurrent worker count (or keep workers active for simulated intervals) rather than emitting an effectively unsampleable pulse.

	c.rt.activeWorkers.WithLabelValues(c.name).Set(0)

observability/dev/simulator/world.go:144

• The only new test validates the copied runtime metric schemas; none of the scenario behavior started here is exercised. Regressions in the promised labels, condition ages, error/apply ratios, leader state, or cancellation behavior would therefore pass CI. Add registry-based tests for the scripted scenarios, ideally with injectable clocks/tickers so the success, failure, and shutdown paths run deterministically.

	go w.healthyWebApps(ctx)
	go w.hotLoop(ctx)
	go w.databases(ctx)
	go w.workqueueBacklog(ctx)
	go w.panics(ctx)

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[sourcehawk]

Review round 2, disposition of the suppressed findings:

• world.go:74, activeWorkers pulse unsampleable: applied in 9744296. The per-reconcile Set(1)/Set(0) pair is gone; the workers gauge is now held persistently (database saturated at its max, webapp varying 0 to 2), so scrapes observe the load the other series imply.
• world.go:144, no automated coverage of the scripted scenarios: applied, scaled to the tool's purpose, in 9744296. A registry-based smoke test (TestWorldScriptedScenarios) starts the world and asserts the promised series: the stuck orders-db Ready=False condition at least seven hours old, pvc apply errors, the hot-loop configmap updates rising, reconcile totals for both controllers, the leader gauge, and clean shutdown on context cancel. Injectable clocks were deliberately not added; the simulator is a dev tool and require.Eventually against the first scenario ticks is enough signal.

[Copilot]

🟡 Changes recommended

Rerendering can disconnect live bind mounts, startup failures are masked, and the simulator test is nondeterministic and incomplete.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 11/11 changed files
• Comments generated: 5
• Review effort level: Balanced