feat(observability): add the alert render pipeline, alert templates and promtool tests

[sourcehawk]

Towards #184

Summary

Adds the observability/ tree with a make render pipeline, three templated Prometheus alert rule files, promtool unit tests for every rule, and a CI job that runs them.

Make targets (new ##@ Observability section in the root Makefile):

• make alerts METRIC_NAMESPACE=<ns> renders every observability/alerts/*.yaml into observability/generated/alerts/<name>.yaml, as one PrometheusRule object per file by default or as plain rule files with ALERT_FORMAT=rules. Per-operator templates (<name>.tpl.yaml) are rendered and named <metric-namespace>-<name>; shared rule files (<name>.yaml, no placeholder) are copied through and named ocf-<name>, so they are installed once per cluster rather than once per operator. NAMESPACE_LABEL (default exported_namespace) selects the label that carries the custom resource namespace; PROMETHEUSRULE_NAMESPACE and PROMETHEUSRULE_LABELS (comma-separated key=value) set the PrometheusRule metadata for ruleSelectors such as kube-prometheus-stack's release label. Missing METRIC_NAMESPACE or an unknown ALERT_FORMAT fail with a usage hint.
• make dashboards METRIC_NAMESPACE=<ns> renders observability/dashboards/*.tpl.json into observability/generated/dashboards/. The loop tolerates an empty glob, so the target already works before the first dashboard template lands.
• make test-alerts renders every template with the test namespace under both namespace label variants (shared rule files are copied as is), lints them with promtool check rules --lint=all --lint-fatal, and runs promtool test rules --diff on observability/alerts/tests/*.yaml.
• observability/generated/ is gitignored; observability/README.md carries the render commands.

Rule files (the per-operator template uses {{operator_namespace}} and {{namespace_label}}; every alert is severity: warning with no routing labels; the shared rules aggregate by the operator's namespace as well so two installs of one operator are alerted on separately):

• crd_conditions.tpl.yaml, group crd-conditions: CustomResourceNotReady, CustomResourceConditionUnknown, CustomResourceConditionStuck. Ported unchanged from go-crd-condition-metrics so the per-owner condition alerts ship from this repository; each carries a dashboard_url into the crd_conditions_browser dashboard.
• managed_resources.yaml (shared, ocf-managed-resources), group managed-resources: ManagedResourceNotConverging, ManagedResourceApplyFailing. Both are ratios of the resource's own applies with an absolute floor, keyed on (namespace, controller, owner_kind, component, resource, kind), so legitimate churn at scale and sporadic conflicts stay silent while a hot loop or a systematically failing resource type fires.
• controller_runtime.yaml (shared, ocf-controller-runtime), group controller-runtime: ControllerReconcileErrors, ControllerReconcilePanics, ControllerWorkqueueBacklog, ControllerReconcileLatencyHigh, OperatorLeaderMissing. Thresholds are ratios or p99 quantiles per controller and sit on real histogram bucket bounds (30 s reconcile time, 100 s queue wait), so they hold whether the operator manages three owners or three thousand.

CI: a new observability job in .github/workflows/test.yml installs a pinned promtool (3.14.0) and runs make test-alerts. promtool stays out of make all so contributors without it can still run everything else.

Verification

• make all: passes (fmt, lint, test, test-scaffold, test-examples, build-examples).
• make test-alerts: lint SUCCESS for all three rule files under both exported_namespace and namespace, and SUCCESS for all three unit test files (15 test cases, including a per-install case for two installs of one operator and the negative cases: churn at scale and a single edit do not fire ManagedResourceNotConverging; sporadic conflicts do not fire ManagedResourceApplyFailing; a resource that only ever failed still fires).
• make alerts METRIC_NAMESPACE=demo: writes crd_conditions.yaml, controller_runtime.yaml, managed_resources.yaml holding PrometheusRule objects demo-crd-conditions, ocf-controller-runtime, ocf-managed-resources that parse as YAML with the groups under spec; with PROMETHEUSRULE_NAMESPACE=monitoring PROMETHEUSRULE_LABELS=release=kps,team=platform the metadata carries the namespace and both labels.
• make alerts METRIC_NAMESPACE=demo ALERT_FORMAT=rules && promtool check rules observability/generated/alerts/*.yaml: SUCCESS: 5 rules found, 3 rules found, 2 rules found.
• make alerts: exits non-zero with Error: METRIC_NAMESPACE is required. and the usage hint.
• make alerts METRIC_NAMESPACE=demo ALERT_FORMAT=nope: exits non-zero with the ALERT_FORMAT error.
• make dashboards METRIC_NAMESPACE=demo: exits 0 with no dashboard templates present.
• actionlint .github/workflows/test.yml: clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JPihvXVfS997iGmGabTGsd

[Copilot]

🟡 Changes recommended

The PR as submitted has correctness and completeness gaps (broken/misleading observability docs/targets, and a likely incorrect label assumption in the workqueue alert that would prevent it from firing).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an observability/ subtree intended to let framework consumers render Prometheus alert rules (and eventually dashboards) parameterized by metric namespace, and validates alert rules via promtool tests and CI.

Changes:

• Introduces templated Prometheus alert rule files under observability/alerts/*.tpl.yaml plus promtool unit tests under observability/alerts/tests/.
• Adds root Makefile targets to render alerts/dashboards and to run promtool check rules + promtool test rules.
• Adds a new GitHub Actions job that installs promtool and runs make test-alerts; ignores rendered artifacts via .gitignore.

File summaries

File	Description
observability/README.md	Entry-point README describing how to render and test observability artifacts.
observability/alerts/crd_conditions.tpl.yaml	Alert templates for CRD condition metrics.
observability/alerts/ocf_resources.tpl.yaml	Alert templates for OCF managed-resource apply metrics.
observability/alerts/controller_runtime.tpl.yaml	Alert templates for controller-runtime and leader-election metrics.
observability/alerts/tests/crd_conditions_test.yaml	promtool tests for CRD condition alerts.
observability/alerts/tests/ocf_resources_test.yaml	promtool tests for managed-resource apply alerts.
observability/alerts/tests/controller_runtime_test.yaml	promtool tests for controller-runtime alerts.
Makefile	Adds dashboards, alerts, and test-alerts render/test targets.
.gitignore	Ignores observability/generated/ render output.
.github/workflows/test.yml	Adds an observability CI job that installs promtool and runs make test-alerts.

Review details

Suppressed comments (1)

observability/README.md:15

• make observability-up is referenced here, but there is no such target in the Makefile, so this command will fail for readers. Either add the target or remove the reference.

Run the alert unit tests with `make test-alerts` (needs `promtool`), and bring up Prometheus and Grafana with the
simulator behind them with `make observability-up`.

• Files reviewed: 9/10 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[sourcehawk]

Review round 1, disposition of the suppressed finding:

• observability/README.md:15, make observability-up does not exist yet: intentional for this sub-PR. The target, the compose stack and the simulator ship in Add a local Prometheus and Grafana stack fed by a metrics simulator #185 on the same feature branch; the README describes the final shape of the feature and is not merged to main before that.

[Copilot]

🟡 Changes recommended

The new CI job’s promtool install step pipes into tar without specifying -f -, which can cause extraction to fail and break the observability workflow run.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 9/10 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The Makefile currently can emit invalid PrometheusRule label YAML (unquoted non-string values) and does not run promtool rule unit tests against the NAMESPACE_LABEL=namespace rendered variant.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 9/10 changed files
• Comments generated: 2
• Review effort level: Lite