Add the observability render pipeline with alert rule templates, promtool tests and CI

[sourcehawk]

Add the observability render pipeline with alert rule templates, promtool tests and CI

Problem

Operators built on the framework have no alert rules for the metrics the framework and controller-runtime emit, and no way to render templated rules for their metric namespace from this repository. The condition alerts in go-crd-condition-metrics exist, but they are not shipped here and do not cover the framework's apply counters or controller-runtime.

Approach

• Create observability/ with alerts/crd_conditions.tpl.yaml (the three upstream condition alerts carried over, rendered per metric namespace), plus two shared cluster-wide rule files installed once per cluster: alerts/managed_resources.yaml (ManagedResourceNotConverging, ManagedResourceApplyFailing) and alerts/controller_runtime.yaml (ControllerReconcileErrors, ControllerReconcilePanics, ControllerWorkqueueBacklog, ControllerReconcileLatencyHigh, OperatorLeaderMissing), each with a promtool unit test file under alerts/tests/.
• Add an ##@ Observability section to the root Makefile: alerts, dashboards, test-alerts, with METRIC_NAMESPACE, NAMESPACE_LABEL (default exported_namespace), ALERT_FORMAT (prometheusrule default, rules), and optional PROMETHEUSRULE_NAMESPACE / PROMETHEUSRULE_LABELS metadata for ruleSelector-based installs. Placeholders are {{operator_namespace}} and {{namespace_label}}; output goes to the gitignored observability/generated/. Templates render as <metric-namespace>-<name>; shared files wrap as ocf-<name>.
• The not-converging and apply-failing rules use ratios of the resource's own applies with an absolute floor, keyed on (namespace, controller, owner_kind, component, resource, kind), so they are safe at scale; controller-runtime thresholds sit on real histogram bucket bounds; every rule ships as severity: warning with no routing labels.
• Add a CI job that installs a pinned promtool and runs make test-alerts.
• observability/README.md with the render commands.

Verification

• make test-alerts passes: rules lint clean with both namespace label variants and every unit test passes, including the negative cases (legitimate churn at scale does not fire ManagedResourceNotConverging; sporadic conflicts do not fire ManagedResourceApplyFailing; a second healthy install of the same operator does not mask an erroring one).
• make alerts METRIC_NAMESPACE=demo writes demo-crd-conditions, ocf-managed-resources and ocf-controller-runtime PrometheusRule objects; ALERT_FORMAT=rules writes plain rule files that promtool check rules accepts.
• make alerts without METRIC_NAMESPACE fails with a usage hint.
• The CI job runs on pull requests and passes.

Context

Sub-issue of #183.

[sourcehawk]

Recording two decisions made during implementation (PR #188) that changed what this issue stated:

Before: the issue listed a lint-dashboards make target and a CI job running both make test-alerts and make lint-dashboards, and described three alert rule files all as templates (alerts/*.tpl.yaml) rendered per metric namespace.

Now: lint-dashboards and its CI step move to #186, which introduces the dashboard templates it lints; this issue ships alerts, dashboards, test-alerts and a CI job running make test-alerts. Only the condition rules remain a template (crd_conditions.tpl.yaml); the controller-runtime and managed-resource rules are shared, cluster-wide files (controller_runtime.yaml, managed_resources.yaml) wrapped once as ocf-controller-runtime / ocf-managed-resources.

Why: a lint target with nothing to lint would be dead code until #186. The shared rules contain no metric namespace placeholder, so rendering them per operator installed a duplicate copy of identical cluster-wide rules for every operator built on the framework; naming them ocf-* makes the install idempotent. Review of PR #188 also made the thresholds reachable against controller-runtime's real histogram buckets (ControllerReconcileLatencyHigh > 30, ControllerWorkqueueBacklog > 100) and added the operator's namespace to every aggregation so two installs of one operator alert separately.

Implemented in 66b33a6 and aa7f4d5.