diff --git a/.ai/base.md b/.ai/base.md index da14ff78..e95a442b 100644 --- a/.ai/base.md +++ b/.ai/base.md @@ -27,6 +27,7 @@ Understand the intended design first: - `docs/cli.md` — the `ocf` scaffolding CLI, its flags, and what the generated code contains - `docs/guidelines.md` — best practices for structuring operators (desired state, one component per condition, etc.) - `docs/compatibility.md` — supported version combinations and compatibility policy +- `docs/observability.md` — Grafana dashboards, Prometheus alert rules, the render pipeline and the local stack ### Source to read @@ -47,6 +48,7 @@ Verify the real API before using or documenting it. Key packages: - `pkg/metrics/` — Prometheus implementation of `component.MetricsRecorder`: condition metrics plus the per-resource apply counters - `pkg/testing/` — testing utilities (`golden/` for snapshot tests, `integration/` for integration helpers) +- `observability/` — dashboard and alert templates, the dev stack and simulator, and the template lint test When changing a public API, also check `examples/` for real usage patterns and to identify what else needs updating. @@ -96,17 +98,18 @@ semantics. GoDoc is part of the public API surface. Update documentation in the **same response** as the code change — never leave them out of sync. -| Code area changed | Documentation to update | -| ------------------------------------------------- | ------------------------------------------ | -| Component builder, reconciliation, status model | `docs/component.md` | -| Primitives, field application, editors, selectors | `docs/primitives.md` | -| Primitive implementations | `docs/primitives/*.md` | -| Generic building blocks, custom resource wrappers | `docs/custom-resource.md` | -| Wrapper templates, CLI flags | `docs/cli.md` | -| Operator structuring patterns, best practices | `docs/guidelines.md` | -| Any `pkg/` export visible in the quick start | `README.md` | -| Examples | `examples/*/README.md` | -| Any file under `docs/` synced into the plugin | Run `make sync-plugin` (CI fails on drift) | +| Code area changed | Documentation to update | +| --------------------------------------------------- | ------------------------------------------ | +| Component builder, reconciliation, status model | `docs/component.md` | +| Primitives, field application, editors, selectors | `docs/primitives.md` | +| Primitive implementations | `docs/primitives/*.md` | +| Generic building blocks, custom resource wrappers | `docs/custom-resource.md` | +| Wrapper templates, CLI flags | `docs/cli.md` | +| Operator structuring patterns, best practices | `docs/guidelines.md` | +| Dashboards, alert rules, render pipeline, dev stack | `docs/observability.md` | +| Any `pkg/` export visible in the quick start | `README.md` | +| Examples | `examples/*/README.md` | +| Any file under `docs/` synced into the plugin | Run `make sync-plugin` (CI fails on drift) | When updating documentation in markdown files, make sure to run `make fmt-md` for consistent formatting. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 94b2e0e2..2e4da366 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -27,6 +27,7 @@ Understand the intended design first: - `docs/cli.md` — the `ocf` scaffolding CLI, its flags, and what the generated code contains - `docs/guidelines.md` — best practices for structuring operators (desired state, one component per condition, etc.) - `docs/compatibility.md` — supported version combinations and compatibility policy +- `docs/observability.md` — Grafana dashboards, Prometheus alert rules, the render pipeline and the local stack ### Source to read @@ -47,6 +48,7 @@ Verify the real API before using or documenting it. Key packages: - `pkg/metrics/` — Prometheus implementation of `component.MetricsRecorder`: condition metrics plus the per-resource apply counters - `pkg/testing/` — testing utilities (`golden/` for snapshot tests, `integration/` for integration helpers) +- `observability/` — dashboard and alert templates, the dev stack and simulator, and the template lint test When changing a public API, also check `examples/` for real usage patterns and to identify what else needs updating. @@ -96,17 +98,18 @@ semantics. GoDoc is part of the public API surface. Update documentation in the **same response** as the code change — never leave them out of sync. -| Code area changed | Documentation to update | -| ------------------------------------------------- | ------------------------------------------ | -| Component builder, reconciliation, status model | `docs/component.md` | -| Primitives, field application, editors, selectors | `docs/primitives.md` | -| Primitive implementations | `docs/primitives/*.md` | -| Generic building blocks, custom resource wrappers | `docs/custom-resource.md` | -| Wrapper templates, CLI flags | `docs/cli.md` | -| Operator structuring patterns, best practices | `docs/guidelines.md` | -| Any `pkg/` export visible in the quick start | `README.md` | -| Examples | `examples/*/README.md` | -| Any file under `docs/` synced into the plugin | Run `make sync-plugin` (CI fails on drift) | +| Code area changed | Documentation to update | +| --------------------------------------------------- | ------------------------------------------ | +| Component builder, reconciliation, status model | `docs/component.md` | +| Primitives, field application, editors, selectors | `docs/primitives.md` | +| Primitive implementations | `docs/primitives/*.md` | +| Generic building blocks, custom resource wrappers | `docs/custom-resource.md` | +| Wrapper templates, CLI flags | `docs/cli.md` | +| Operator structuring patterns, best practices | `docs/guidelines.md` | +| Dashboards, alert rules, render pipeline, dev stack | `docs/observability.md` | +| Any `pkg/` export visible in the quick start | `README.md` | +| Examples | `examples/*/README.md` | +| Any file under `docs/` synced into the plugin | Run `make sync-plugin` (CI fails on drift) | When updating documentation in markdown files, make sure to run `make fmt-md` for consistent formatting. diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 6d6b2950..85cf7c6c 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -25,4 +25,32 @@ jobs: make test - name: Running Scaffold Gate - run: make test-scaffold \ No newline at end of file + run: make test-scaffold + observability: + name: Alerts and dashboards + runs-on: ubuntu-latest + steps: + - name: Clone the code + uses: actions/checkout@v7 + + - name: Setup Go + uses: actions/setup-go@v7 + with: + go-version-file: go.mod + + # promtool is not part of `make all` so that contributors without it can + # still run everything else. Bump PROMETHEUS_VERSION by hand. + - name: Install promtool + env: + PROMETHEUS_VERSION: "3.14.0" + run: | + mkdir -p "$HOME/.local/bin" + curl -sSfL "https://github.com/prometheus/prometheus/releases/download/v$PROMETHEUS_VERSION/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz" \ + | tar -xz --strip-components=1 -C "$HOME/.local/bin" "prometheus-$PROMETHEUS_VERSION.linux-amd64/promtool" + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + + - name: Test alerts + run: make test-alerts + + - name: Lint dashboards + run: make lint-dashboards diff --git a/.gitignore b/.gitignore index f2d6818b..a1d509ba 100644 --- a/.gitignore +++ b/.gitignore @@ -49,3 +49,6 @@ docs/superpowers/ # MkDocs build output site/ + +# Rendered observability artifacts +observability/generated/ diff --git a/Makefile b/Makefile index 9e486268..8196cffa 100644 --- a/Makefile +++ b/Makefile @@ -105,6 +105,7 @@ sync-plugin: ## Sync framework docs into the Claude plugin skill references. $(PLUGIN_SKILLS)/structuring-operators/references \ $(PLUGIN_SKILLS)/testing-operators/references cp docs/component.md $(PLUGIN_SKILLS)/building-components/references/component.md + cp docs/observability.md $(PLUGIN_SKILLS)/building-components/references/observability.md cp docs/primitives.md $(PLUGIN_SKILLS)/using-primitives/references/primitives.md cp docs/primitives/*.md $(PLUGIN_SKILLS)/using-primitives/references/primitives/ cp docs/custom-resource.md $(PLUGIN_SKILLS)/custom-resource-wrappers/references/custom-resource.md @@ -205,6 +206,186 @@ e2e-component: ginkgo kind-create kind-set-context ## Run component E2E tests on .PHONY: e2e-full e2e-full: kind-create kind-set-context e2e kind-delete ## Full E2E lifecycle: create cluster, test, teardown. +##@ Observability + +OBS_DIR := observability +# Render output directory. The dev stack overrides it to keep its render apart. +OBS_OUT ?= $(OBS_DIR)/generated +# Prometheus metric namespace the condition gauge was created with +# (ocm.NewOperatorConditionsGauge("")). Required for rendering. +METRIC_NAMESPACE ?= +# Label carrying the namespace of the custom resource. The pod scraping the +# operator usually owns `namespace`, so the exported label arrives as +# `exported_namespace`; override with NAMESPACE_LABEL=namespace if yours does not. +NAMESPACE_LABEL ?= exported_namespace +# Shape of the rendered alert files: prometheusrule (one PrometheusRule object +# per rule file) or rules (plain files for prometheus' rule_files). +ALERT_FORMAT ?= prometheusrule +# Optional metadata for the PrometheusRule objects: the namespace to create them +# in, and comma-separated key=value labels, for example the release label a +# kube-prometheus-stack ruleSelector matches on (PROMETHEUSRULE_LABELS=release=kps). +PROMETHEUSRULE_NAMESPACE ?= +PROMETHEUSRULE_LABELS ?= +# Metric namespace the alert unit tests are written against. +ALERT_TEST_NAMESPACE := test_operator + +# Fail unless METRIC_NAMESPACE was given and both variables render to valid +# output. $(1) is the target name for the hint. The namespace prefixes a metric +# name, so it is restricted to metric name characters, and it prefixes the +# dashboard uids, which Grafana limits to 40 characters of [A-Za-z0-9_-]: the +# longest suffix, `_crd_conditions_browser`, is 23 characters, leaving 17 for +# the namespace (observability_test.go pins that arithmetic to the dashboard +# file names), and it names the PrometheusRule objects with `_` mapped to `-`, +# so it must start with a letter. NAMESPACE_LABEL is substituted into PromQL as +# a label name, so it must match the Prometheus label name grammar. +define require_metric_namespace +@[ -n "$(METRIC_NAMESPACE)" ] || { \ + echo "Error: METRIC_NAMESPACE is required."; \ + echo "Usage: make $(1) METRIC_NAMESPACE=my_operator"; \ + exit 1; \ +} +@echo "$(METRIC_NAMESPACE)" | grep -Eq '^[A-Za-z][A-Za-z0-9_]{0,16}$$' || { \ + echo "Error: METRIC_NAMESPACE '$(METRIC_NAMESPACE)' is not renderable."; \ + echo "It must start with a letter, match ^[A-Za-z][A-Za-z0-9_]*$$ and be at most 17 characters:"; \ + echo "it names the PrometheusRule -crd-conditions, which must start with a letter or digit,"; \ + echo "and the dashboard uid _crd_conditions_browser must fit Grafana's 40 character limit."; \ + exit 1; \ +} +@echo "$(NAMESPACE_LABEL)" | grep -Eq '^[A-Za-z_][A-Za-z0-9_]*$$' || { \ + echo "Error: NAMESPACE_LABEL '$(NAMESPACE_LABEL)' is not a Prometheus label name."; \ + echo "It must match ^[A-Za-z_][A-Za-z0-9_]*$$, for example exported_namespace or namespace."; \ + exit 1; \ +} +endef + +# Render a template to stdout. $(1) template path, $(2) metric namespace, +# $(3) namespace label. +define render_template +sed -e 's/{{operator_namespace}}/$(2)_/g' -e 's/{{namespace_label}}/$(3)/g' $(1) +endef + +.PHONY: dashboards +dashboards: ## Render the Grafana dashboards for METRIC_NAMESPACE into observability/generated/dashboards. + $(call require_metric_namespace,dashboards) + @echo "Rendering dashboards for $(METRIC_NAMESPACE) (namespace label: $(NAMESPACE_LABEL))..." + @mkdir -p $(OBS_OUT)/dashboards + @rm -f $(OBS_OUT)/dashboards/*.json + @for file in $(OBS_DIR)/dashboards/*.tpl.json; do \ + [ -e "$$file" ] || continue; \ + name=$$(basename "$$file" .tpl.json); \ + $(call render_template,"$$file",$(METRIC_NAMESPACE),$(NAMESPACE_LABEL)) > "$(OBS_OUT)/dashboards/$$name.json"; \ + done + +.PHONY: alerts +alerts: ## Render the Prometheus alert rules for METRIC_NAMESPACE into observability/generated/alerts. + $(call require_metric_namespace,alerts) + @echo "Rendering alerts for $(METRIC_NAMESPACE) (namespace label: $(NAMESPACE_LABEL), format: $(ALERT_FORMAT))..." + @mkdir -p $(OBS_OUT)/alerts + @rm -f $(OBS_OUT)/alerts/*.yaml + @for file in $(OBS_DIR)/alerts/*.yaml; do \ + [ -e "$$file" ] || continue; \ + case "$$file" in \ + *.tpl.yaml) \ + name=$$(basename "$$file" .tpl.yaml); \ + rule_name="$(METRIC_NAMESPACE)-$$name" ;; \ + *) \ + name=$$(basename "$$file" .yaml); \ + rule_name="ocf-$$name" ;; \ + esac; \ + rule_name=$$(echo "$$rule_name" | tr '[:upper:]' '[:lower:]' | tr '_:' '--'); \ + out="$(OBS_OUT)/alerts/$$name.yaml"; \ + case "$(ALERT_FORMAT)" in \ + rules) \ + $(call render_template,"$$file",$(METRIC_NAMESPACE),$(NAMESPACE_LABEL)) > "$$out" ;; \ + prometheusrule) \ + { \ + echo "apiVersion: monitoring.coreos.com/v1"; \ + echo "kind: PrometheusRule"; \ + echo "metadata:"; \ + echo " name: $$rule_name"; \ + [ -z "$(PROMETHEUSRULE_NAMESPACE)" ] || echo " namespace: $(PROMETHEUSRULE_NAMESPACE)"; \ + if [ -n "$(PROMETHEUSRULE_LABELS)" ]; then \ + echo " labels:"; \ + for kv in $$(echo "$(PROMETHEUSRULE_LABELS)" | tr ',' ' '); do \ + case "$$kv" in \ + ?*=*) echo " $${kv%%=*}: \"$${kv#*=}\"" ;; \ + *) echo "Error: PROMETHEUSRULE_LABELS entry '$$kv' is not key=value." >&2; exit 1 ;; \ + esac; \ + done; \ + fi; \ + echo "spec:"; \ + $(call render_template,"$$file",$(METRIC_NAMESPACE),$(NAMESPACE_LABEL)) \ + | sed -e 's/^/ /' -e 's/[[:space:]]*$$//'; \ + } > "$$out" ;; \ + *) \ + echo "Error: ALERT_FORMAT must be prometheusrule or rules, got '$(ALERT_FORMAT)'."; exit 1 ;; \ + esac; \ + done + +.PHONY: test-alerts +test-alerts: ## Lint and unit test the alert rules with promtool. + @command -v promtool >/dev/null 2>&1 || { \ + echo "Error: promtool is required to test the alert rules."; \ + echo "It ships with prometheus: https://prometheus.io/download/"; \ + exit 1; \ + } + @set -e; \ + tmpdir=$$(mktemp -d "$${TMPDIR:-/tmp}/ocf-alerts.XXXXXX"); \ + trap 'rm -rf "$$tmpdir"' EXIT; \ + mkdir -p "$$tmpdir/tests" "$$tmpdir/namespace-label"; \ + for file in $(OBS_DIR)/alerts/*.yaml; do \ + [ -e "$$file" ] || continue; \ + case "$$file" in \ + *.tpl.yaml) \ + name=$$(basename "$$file" .tpl.yaml); \ + $(call render_template,"$$file",$(ALERT_TEST_NAMESPACE),exported_namespace) > "$$tmpdir/$$name.yaml"; \ + $(call render_template,"$$file",$(ALERT_TEST_NAMESPACE),namespace) > "$$tmpdir/namespace-label/$$name.yaml" ;; \ + *) \ + cp "$$file" "$$tmpdir/"; \ + cp "$$file" "$$tmpdir/namespace-label/" ;; \ + esac; \ + done; \ + cp $(OBS_DIR)/alerts/tests/*.yaml "$$tmpdir/tests/"; \ + echo "Linting rules..."; \ + promtool check rules --lint=all --lint-fatal "$$tmpdir"/*.yaml; \ + echo "Linting rules with NAMESPACE_LABEL=namespace..."; \ + promtool check rules --lint=all --lint-fatal "$$tmpdir"/namespace-label/*.yaml; \ + echo "Running unit tests..."; \ + promtool test rules --diff "$$tmpdir"/tests/*.yaml + +.PHONY: lint-dashboards +lint-dashboards: ## Check the dashboard and alert templates render to valid files that reference real metrics. + go test ./$(OBS_DIR)/ + +OBS_DEV_NAMESPACE := demo +OBS_DEV_OUT := $(OBS_DIR)/generated/dev +# Extra flags for the simulator, e.g. SIMULATOR_ARGS="-leader=false". +SIMULATOR_ARGS ?= + +.PHONY: observability-render-dev +observability-render-dev: ## Render dashboards and plain rules for the dev stack, with every `for:` shortened to 2m. + @$(MAKE) --no-print-directory dashboards METRIC_NAMESPACE=$(OBS_DEV_NAMESPACE) OBS_OUT=$(OBS_DEV_OUT) + @$(MAKE) --no-print-directory alerts METRIC_NAMESPACE=$(OBS_DEV_NAMESPACE) OBS_OUT=$(OBS_DEV_OUT) ALERT_FORMAT=rules + @for file in $(OBS_DEV_OUT)/alerts/*.yaml; do \ + sed -E 's/^( *for: ).*/\12m/' "$$file" > "$$file.tmp" && mv "$$file.tmp" "$$file"; \ + done + +.PHONY: observability-up +observability-up: observability-render-dev ## Start Prometheus (:9090) and Grafana (:3000) and run the simulator on the host. + docker compose -f $(OBS_DIR)/dev/docker-compose.yaml up -d + @ready=0; for i in $$(seq 1 30); do \ + curl -fsS 127.0.0.1:9090/-/ready >/dev/null 2>&1 && { ready=1; break; }; \ + sleep 1; \ + done; \ + [ "$$ready" = "1" ] || { echo "Error: prometheus did not become ready"; exit 1; }; \ + curl -fsS -X POST 127.0.0.1:9090/-/reload + @echo "Grafana: http://localhost:3000 Prometheus: http://localhost:9090/alerts" + go run ./$(OBS_DIR)/dev/simulator -metric-namespace=$(OBS_DEV_NAMESPACE) $(SIMULATOR_ARGS) + +.PHONY: observability-down +observability-down: ## Stop the local Prometheus and Grafana. + docker compose -f $(OBS_DIR)/dev/docker-compose.yaml down + # go-install-tool will 'go install' any package with custom target and name of binary, if it doesn't exist # $1 - target path with name of binary diff --git a/README.md b/README.md index 45156186..3e016edb 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,7 @@ Full documentation, including a step-by-step tutorial, is at | [CLI](https://sourcehawk.github.io/operator-component-framework/cli/) | Scaffold wrapper packages with `ocf scaffold wrapper` | | [Guidelines](https://sourcehawk.github.io/operator-component-framework/guidelines/) | Patterns for structuring operators well | | [Testing](https://sourcehawk.github.io/operator-component-framework/testing/) | Golden snapshots and version-matrix coverage | +| [Observability](https://sourcehawk.github.io/operator-component-framework/observability/) | Grafana dashboards and Prometheus alerts for your operator | | [Compatibility](https://sourcehawk.github.io/operator-component-framework/compatibility/) | Supported Kubernetes and controller-runtime versions | The full Go API reference is on [pkg.go.dev](https://pkg.go.dev/github.com/sourcehawk/operator-component-framework). diff --git a/docs/component.md b/docs/component.md index 9d21e7ae..a0f674b2 100644 --- a/docs/component.md +++ b/docs/component.md @@ -842,12 +842,15 @@ func init() { recCtx := component.ReconcileContext{ // ... - Metrics: metrics.NewRecorder("webapp-controller", conditions, collectors), + Metrics: metrics.NewRecorder("webapp", conditions, collectors), } ``` -The controller name becomes the `controller` label on every series the recorder emits. Passing `nil` for either -collector disables that family; passing `nil` for `Metrics` itself disables both. +The controller name becomes the `controller` label on every series the recorder emits. It must match the +controller-runtime controller name (the lower-cased kind passed to `For`, unless `Named` overrides it) so that the +shipped [dashboards and alerts](observability.md) correlate the framework's series with controller-runtime's reconcile +and workqueue series. Passing `nil` for either collector disables that family; passing `nil` for `Metrics` itself +disables both. ### Resource metrics diff --git a/docs/index.md b/docs/index.md index 23b9cf04..9a92998b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -33,6 +33,10 @@ New to the framework? Start with **Getting Started**. Already building and looki Golden snapshots and version-matrix golden generation. +- :material-chart-line: **[Observability](observability.md)** + + Grafana dashboards and Prometheus alerts for your operator. + ## Why this exists diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 00000000..7da5d072 --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,357 @@ +# Observability + +The framework ships Grafana dashboards and Prometheus alert rules for the metrics an operator built on it exposes: the +[condition and resource apply metrics](component.md#metrics) recorded through `pkg/metrics`, and the reconcile, +workqueue, REST client, leader election and process series every controller-runtime operator exports. They live under +`observability/` in the repository as templates, keyed on the metric namespace of your operator, and render with `make`. +A local Prometheus and Grafana stack fed by a simulator lets you look at every panel and every alert without a cluster. + +## What ships + +| Artifact | File | Scope | +| ------------------------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| OCF Operator dashboard | `dashboards/ocf_operator.tpl.json` | Per operator. Rendered uid `_ocf_operator`. Operator health end to end: reconciliation, workqueue, managed resource applies, condition summary, API client, process. | +| CRD Conditions Browser | `dashboards/crd_conditions_browser.tpl.json` | Per operator. Rendered uid `_crd_conditions_browser`. Per-owner condition drill-down; the target of the condition alerts' `dashboard_url` links. | +| Condition alerts | `alerts/crd_conditions.tpl.yaml` | Per operator, rendered per metric namespace as the `PrometheusRule` `-crd-conditions`. `CustomResourceNotReady`, `CustomResourceConditionUnknown`, `CustomResourceConditionStuck`. | +| Managed resource alerts | `alerts/managed_resources.yaml` | Shared, cluster-wide. Installed once as the `PrometheusRule` `ocf-managed-resources`, whichever operator rendered it. `ManagedResourceNotConverging`, `ManagedResourceApplyFailing`. | +| Controller-runtime alerts | `alerts/controller_runtime.yaml` | Shared, cluster-wide. Installed once as the `PrometheusRule` `ocf-controller-runtime`. `ControllerReconcileErrors`, `ControllerReconcilePanics`, `ControllerWorkqueueBacklog`, `ControllerReconcileLatencyHigh`, `OperatorLeaderMissing`. | + +The split follows the metrics. The condition gauge is named after the metric namespace +(`_controller_condition`), so its rules and both dashboards carry a placeholder and render per +operator. The apply counters (`ocf_resource_apply_total`, `ocf_resource_apply_errors_total`) and the controller-runtime +families have fixed names shared by every operator in the cluster, so their rules contain no placeholder, tell operators +apart by label, and are installed once. `make alerts` writes the shared files alongside the per-operator one on every +render; apply them from whichever operator's render you like, the content is identical. + +## Rendering + +The render pipeline is `make` and `sed`; it needs no Go toolchain. Three different things are called a namespace on this +page, so to be precise: the **metric namespace** is the string your operator passed to `ocm.NewOperatorConditionsGauge` +(the prefix of the condition gauge's name), the **owner namespace** is the Kubernetes namespace of a custom resource, +carried as a label on its condition series, and the **operator namespace** is the Kubernetes namespace the operator pod +runs in, stamped on every series by the scrape job. + +Clone the repository and, from its root, render with your metric namespace: + +```bash +make dashboards METRIC_NAMESPACE=myoperator +make alerts METRIC_NAMESPACE=myoperator +``` + +Output lands in `observability/generated/`, which is gitignored. Each target removes the files it previously rendered +before writing, so a dashboard or rule file that a framework upgrade renamed or dropped does not linger and get +installed again: + +``` +observability/generated/ +├── alerts/ +│ ├── controller_runtime.yaml PrometheusRule ocf-controller-runtime (shared) +│ ├── crd_conditions.yaml PrometheusRule myoperator-crd-conditions +│ └── managed_resources.yaml PrometheusRule ocf-managed-resources (shared) +└── dashboards/ + ├── crd_conditions_browser.json + └── ocf_operator.json +``` + +Two placeholders are substituted: `{{operator_namespace}}` becomes `_`, so every reference to the +condition gauge reads `myoperator_controller_condition` (the placeholder is named for the metric namespace, not a +Kubernetes one), and `{{namespace_label}}` becomes the value of `NAMESPACE_LABEL`. Pass the same variables to both +`make dashboards` and `make alerts`. Both placeholders and every variable below are the same as in +[go-crd-condition-metrics](https://github.com/sourcehawk/go-crd-condition-metrics), so a build that already renders that +repository's artifacts needs no change. + +| Variable | Default | Effect | +| -------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `METRIC_NAMESPACE` | required | The argument to `ocm.NewOperatorConditionsGauge`. Names the condition gauge, the dashboard uids and the condition `PrometheusRule`. A letter followed by metric name characters (`[A-Za-z0-9_]`), at most 17 characters: it names the `PrometheusRule` objects with `_` mapped to `-`, so it must start with a letter, and the longest rendered uid must fit Grafana's 40 character limit; rendering fails otherwise. | +| `NAMESPACE_LABEL` | `exported_namespace` | The label carrying the owner's namespace on condition series, see below. Must be a Prometheus label name (`[A-Za-z0-9_]`, not starting with a digit); rendering fails otherwise. | +| `ALERT_FORMAT` | `prometheusrule` | `prometheusrule` wraps each rule file in a `monitoring.coreos.com/v1` `PrometheusRule` for the Prometheus Operator; `rules` writes the plain `groups:` file that Prometheus loads through `rule_files`. | +| `PROMETHEUSRULE_NAMESPACE` | unset | `metadata.namespace` of the `PrometheusRule` objects. Unset leaves it to `kubectl apply -n`. | +| `PROMETHEUSRULE_LABELS` | unset | Comma-separated `key=value` pairs written to `metadata.labels`. An entry without `=` fails the render. A kube-prometheus-stack install selects rules by its release label, so pass `PROMETHEUSRULE_LABELS=release=`. | +| `OBS_OUT` | `observability/generated` | Render output directory. | + +`PrometheusRule` names are the metric namespace or `ocf` prefix joined to the file name, lower-cased, with `_` and `:` +folded to `-`, so a namespace of `My_Operator` renders as `my-operator-crd-conditions`. + +### The namespace label + +The condition gauge exports the owner's namespace as a `namespace` label. When a ServiceMonitor or PodMonitor scrapes +the operator, Prometheus stamps the scrape target's own labels on every series, and the target's `namespace` label (the +operator pod's namespace) collides with the exported one. Prometheus resolves the collision by renaming the exported +label to `exported_namespace`, which is why that is the default. Pass `NAMESPACE_LABEL=namespace` when your scrape sets +`honorLabels: true`, or does not stamp a `namespace` target label at all, so the exported label arrives unchanged. + +The setting affects the condition rules and both dashboards. Nothing else is namespace-scoped by owner: the apply +counters carry no owner namespace by design, and the `namespace` and `job` the controller-runtime and managed-resource +rules aggregate by are the operator's own namespace and scrape job, stamped by Prometheus. Together they let two +installs of one operator in a cluster alert separately, and keep two operators in one namespace that happen to share a +controller name from merging into one ratio, where a healthy operator would dilute a failing one below the threshold. +Outside a cluster both labels are simply absent, which is harmless. + +### Installing + +With the default `ALERT_FORMAT`, apply the rendered rules into the namespace your Prometheus Operator watches: + +```bash +make alerts METRIC_NAMESPACE=myoperator PROMETHEUSRULE_NAMESPACE=monitoring PROMETHEUSRULE_LABELS=release=kube-prometheus-stack +kubectl apply -f observability/generated/alerts/ +``` + +With `ALERT_FORMAT=rules`, add the three files to your Prometheus `rule_files`. + +For the dashboards, either import the two JSON files through Grafana's UI or API, or, with the Grafana sidecar that +kube-prometheus-stack deploys, ship them as a ConfigMap carrying the sidecar's label (`grafana_dashboard` by default): + +```bash +kubectl create configmap myoperator-dashboards -n monitoring --from-file=observability/generated/dashboards/ +kubectl label configmap myoperator-dashboards -n monitoring grafana_dashboard=1 +``` + +Check the rendered files into the repository that deploys your operator, and re-render when you upgrade the framework. +Tune thresholds and `for:` durations in the rendered files, or with a kustomize patch over them; the sections below say +what each threshold means so the change is deliberate. Give the two shared `PrometheusRule` objects one owner in the +cluster: two teams applying differently tuned copies under the same name overwrite each other, and applying them into +two namespaces installs every shared alert twice. + +## Naming the controller + +The OCF Operator dashboard filters four metric families with one `controller` variable: controller-runtime's reconcile +series, the workqueue series, the framework's apply counters and the condition gauge. controller-runtime labels the +first two with the name of the controller, which is the lower-cased kind passed to `For` unless `Named` overrides it. +The framework labels the last two with the name you pass to `metrics.NewRecorder`. For the dashboard to correlate them, +the two names must be the same: + +```go +ctrl.NewControllerManagedBy(mgr). + For(&v1.WebApp{}). + Named("webapp"). // optional, "webapp" is the default for kind WebApp + Complete(r) + +recCtx := component.ReconcileContext{ + // ... + Metrics: metrics.NewRecorder("webapp", conditions, collectors), +} +``` + +The alerts key on the same label, so the `controller` in a `ManagedResourceNotConverging` notification and the +`controller` in a `ControllerReconcileErrors` notification then name the same thing. `OperatorLeaderMissing` is the one +exception: leader election is per operator, not per controller, and its `name` label is the lease name. + +## Alerts + +Every rule ships with `severity: warning` and no routing labels; severity, thresholds and routing are yours to tune. No +rule on the apply counters or the controller-runtime metrics creates a series per owner: they aggregate by the +operator's static topology, so the same rules hold whether the operator manages three owners or three thousand. The +per-owner signal comes from the condition rules. + +### Managed resources + +Shared, installed once as `ocf-managed-resources`. Both rules key on +`(namespace, job, controller, owner_kind, component, resource, kind)`: the labels of `ocf_resource_apply_total` plus the +scrape namespace and job. They fire per resource type, not per owner, because the counters carry no owner identity. + +| Alert | Fires when | Threshold | `for` | +| ------------------------------ | ---------------------------------------------------- | ----------------------------------------------------------------------------- | ----- | +| `ManagedResourceNotConverging` | most applies of one resource type rewrite the object | `updated / all applies` over 15m `> 0.5`, and more than 15 updates in 15m | 15m | +| `ManagedResourceApplyFailing` | most apply attempts of one resource type fail | `errors / (errors + applies)` over 15m `> 0.5`, and more than 5 errors in 15m | 15m | + +`ManagedResourceNotConverging` is the alert the apply counters were built for: a managed resource rewritten on most +reconcile. [Resource metrics](component.md#resource-metrics) explains what an `updated` apply on every reconcile means +and why events do not catch it. + +The rule measures the share of a resource's own applies that rewrote it, not the absolute `updated` rate. A bare +`rate(updated) > 0` is wrong at scale: legitimate spec changes across many owners keep the aggregate `updated` rate +above zero indefinitely. Legitimate churn is followed by a `none` apply on the next reconcile, which keeps its ratio at +or below one half; a hot loop pushes the ratio to one. The floor of 15 updates in 15 minutes keeps a single edit on an +otherwise idle resource from producing a ratio of one over a handful of samples. Raise the floor if your operator's +resources are edited in bursts; lower the ratio only if you are sure your reconcile cadence never produces a `none` +between two legitimate updates. + +`ManagedResourceApplyFailing` counts every failure of an attempt: mutating the desired object, the server-side apply +patch, and the classification after it. Transient conflicts among successful applies stay under the ratio, and a +resource failing for one owner among many stays under the floor; that owner's `Ready` condition goes `False` and the +condition rules catch it. The `or` in the denominator keeps the ratio defined for a resource that has never applied +successfully, where no success series exists to add. The framework records no event for a failed apply, so the `kubectl` +command in the notification's description lists the owners' `Ready` condition reason and message, which is where the +failure lives. + +### Controller-runtime + +Shared, installed once as `ocf-controller-runtime`. All but the last rule aggregate by `(namespace, job, controller)`; +`OperatorLeaderMissing` keys on the lease name, which is unique within a namespace. + +| Alert | Fires when | Threshold | `for` | +| -------------------------------- | ---------------------------------------------- | -------------------------------------------------------------------------------- | ----- | +| `ControllerReconcileErrors` | a controller's reconciles mostly return errors | `result="error"` share of `controller_runtime_reconcile_total` over 10m `> 0.25` | 15m | +| `ControllerReconcilePanics` | a reconcile panicked | `increase(controller_runtime_reconcile_panics_total[10m]) > 0` | none | +| `ControllerWorkqueueBacklog` | items wait too long for a worker | p99 of `workqueue_queue_duration_seconds` over 10m `> 100` seconds | 15m | +| `ControllerReconcileLatencyHigh` | reconciles are slow | p99 of `controller_runtime_reconcile_time_seconds` over 10m `> 30` seconds | 15m | +| `OperatorLeaderMissing` | no replica holds the leader lease | `max by (namespace, name) (leader_election_master_status) == 0` | 5m | + +The thresholds are ratios and quantiles rather than absolute rates for the same reason as above: they hold at any scale. +The backlog rule uses queue wait time rather than queue depth, because no depth is right for every operator, whereas +items waiting minutes for a worker is wrong at any scale. + +Both quantile thresholds sit on histogram bucket bounds so that they mean what they say. controller-runtime's reconcile +time histogram has 60 seconds as its largest finite bucket, and `histogram_quantile` never returns more than the last +finite bound, so a threshold of 60 or above could never fire; 30 is the highest bound that leaves room above it. The +workqueue histogram has one bucket per decade, so `> 100` means more than one percent of items waited longer than 100 +seconds, and the p99 value reported in the notification is interpolated within that bucket. Move these thresholds only +to another bucket bound: the reconcile histogram's bounds from ten seconds up are 10, 15, 20, 25, 30, 40, 50 and 60, and +the workqueue histogram's are 1, 10, 100 and 1000. A threshold between two bounds, such as 45, fires exactly like the +bound below it and only reads as if it were stricter. + +`OperatorLeaderMissing` is silent in two cases: when leader election is off, because the gauge is not exported at all, +and when no replica is alive to export it, for example a crash loop before the elector starts. Pair it with your +platform's target-down alert for the operator's scrape job. + +### Conditions + +Per operator, rendered as `-crd-conditions`. The metric value of +`_controller_condition` is the condition's `lastTransitionTime`, which the rules rely on. + +| Alert | Fires when | Threshold | `for` | +| -------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----- | +| `CustomResourceNotReady` | an owner's `Ready` condition is `False` | `max by (controller, kind, name, )` of `condition="Ready", status="False"` | 30m | +| `CustomResourceConditionUnknown` | any condition of an owner is `Unknown` | `max by (controller, kind, name, condition, )` of `status="Unknown"` | 30m | +| `CustomResourceConditionStuck` | an owner's `Ready` condition has not been `True` for six hours | `time() - max(...)` of `condition="Ready", status!="True"` `> 21600` | none | + +Every rule aggregates with `max()` instead of matching series directly, and that is load bearing twice over. The +aggregation drops the `reason`, `status` and `id` labels, so a controller that keeps changing the reason while an owner +stays unhealthy does not restart the `for:` clock every time. And `max()` keeps the freshest `lastTransitionTime`, so +two series for one owner, such as a reason change still inside the lookback window, collapse to the freshest rather than +adding up the way `sum()` would. + +The status matcher on its own would still fire on a former leader pod's stale series (see [Stale series](#stale-series) +below), so before applying it every rule joins on the freshest series per owner across every status, with the same +`and topk by (...) (1, ...)` join the dashboards use. A stale `False` or `Unknown` series loses that join as soon as the +current leader exports a later `lastTransitionTime` for the owner, whatever its status. + +`CustomResourceNotReady` and `CustomResourceConditionStuck` are scoped to `Ready` on purpose. Matching `status="False"` +across every condition type would fire forever on negative-polarity conditions such as `Degraded`, where `False` is the +healthy state. To cover your own positive-polarity conditions, widen the `condition` matcher in the rendered file, for +example `condition=~"Ready|CertificateReady"`. The stuck rule keeps `condition` in its `by` clause, so that edit alone +gives one alert per owner and condition. `CustomResourceNotReady` aggregates without `condition`, so add it to the `by` +clause as well if you want a separate alert per condition rather than one per owner. `CustomResourceConditionUnknown` is +not scoped, because `Unknown` is bad whatever the polarity. + +`CustomResourceConditionStuck` has no `for:` clause because its expression is itself a duration comparison. A `for:` +clause measures how long the alert has been true, which is bounded by how long the series has been continuously present; +a scrape gap, an operator restart or a ruler restart silently restarts that clock. The stuck rule measures how long the +owner has been in its state according to its own status, so it survives all three and reports the real age. It also +covers `Unknown`, which `CustomResourceNotReady` does not. Tune the `21600` (six hours) to the longest time an owner of +yours can legitimately take to become ready. + +Each condition alert carries a `dashboard_url` annotation that deep-links into the CRD Conditions Browser rendered for +the same metric namespace, narrowed to the one owner. The `id` label is `/`, or `/` for a +cluster-scoped owner, so the link works without a conditional. + +## Dashboards + +Both dashboards are Grafana JSON at schema version 41 with a `datasource` variable, auto-refresh off, and the `ocf` tag. +A dashboard link at the top of each lists every dashboard carrying that tag, which is how they cross-reference each +other regardless of the folder or sub-path Grafana serves them from. + +The uids are templated with the metric namespace because Grafana upserts dashboards by uid: with fixed uids, importing a +second operator's render into a shared Grafana would overwrite the first. Two operators in one Grafana therefore get +`alpha_ocf_operator` and `beta_ocf_operator`, and each operator's condition alerts link to its own browser. + +### OCF Operator + +Variables, in cascade: `namespace` (Operator namespace), `job` (Operator, the scrape job) and `controller` (multi, All). +Every controller-runtime operator on a cluster exports the same metric names, so nothing in `controller_runtime_*`, +`workqueue_*`, `rest_client_*`, `process_*` or the apply counters says which operator a series belongs to; the scrape +labels do. `namespace` is the namespace the operator pod runs in and `job` its scrape job, usually the name of the +metrics Service or ServiceMonitor. Together they select one install of one operator, the same pair the shared alert +rules aggregate by, so two installs of the same operator in different namespaces stay apart. `namespace` defaults to +All, which also matches series that carry no namespace label at all, so an operator scraped outside a cluster still +renders. `namespace` scopes the controller-runtime, workqueue, apply and process panels, whose `namespace` label is +always the pod's; the condition panels are scoped by `job` alone, because on condition series that label is the CR's +namespace when rendered with `NAMESPACE_LABEL=namespace`. `controller` narrows within that operator, see +[Naming the controller](#naming-the-controller). Panel titles say CR for what the framework calls the owner: one custom +resource instance the controller reconciles, identified by its kind, namespace and name. Rows are ordered by what an +on-call reader asks first: is the operator healthy right now, are the owners healthy, is anything being rewritten or +failing, then the controller's own reconciliation and workqueue internals, the API client, and the process. + +| Row | What it answers | +| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Overview | Is the operator healthy right now? Four stat tiles with sparklines over the dashboard range: reconciles per second, reconcile error ratio, p99 reconcile time and p99 queue wait. Below them, owner counts on one panel, Ready, not Ready for more than two minutes (debounced on `lastTransitionTime`, so a rollout in progress does not count) and Unknown, then leader status and active against max workers. All three owner counts are scoped to the `Ready` condition; unlike `CustomResourceConditionUnknown`, the Unknown count does not cover other conditions. | +| Conditions | Which CRs are unhealthy? CRs by Ready status over time, stacked True, False and Unknown so a rollout or an outage shows as a band; a bar gauge of unhealthy conditions counting owners per kind, condition and status for every condition that is not True (empty when everything is healthy, and independent of how many kinds or condition types the operator has); and a full width CRs not Ready table (kind, namespace, name, reason, since, status) whose rows link into the CRD Conditions Browser filtered to that owner. | +| Managed resources | Is anything being rewritten or failing? Apply rate by operation with the error rate on the same panel, the `updated` rate per resource over time with a legend table sorted by last value so the worst offender is on top, the not-converging ratio per resource, and the apply error ratio per resource. The not-converging panel plots the `ManagedResourceNotConverging` expression without its floor, so a resource heading for the alert is visible before it fires; the error ratio's denominator falls back to the error rate alone when no success series exists, as the alert does. | +| Reconciliation | Where does reconcile time go? Reconcile rate by result, error ratio over time, latency at p50, p90 and p99, panics, the age of the longest in-progress reconcile (`workqueue_longest_running_processor_seconds`), and workers. | +| Workqueue | Is the operator keeping up? Depth, adds per second, queue wait p99, work duration p99, retries per second, unfinished work. | +| API client | Is the API server pushing back? `rest_client_requests_total` rate by method and non-2xx responses by code. | +| Process | Collapsed. CPU, resident memory and goroutines for the `job`. The CPU and memory panels draw the container request and limit as dashed lines when kube-state-metrics is scraped, joining `kube_pod_container_resource_requests` and `_limits` to the operator's scrape target on its `namespace` and `pod` labels; a pod with several containers shows its largest. | + +Reason strings are operator-specific, so colour is driven by `status` (`True` green, `False` red, `Unknown` yellow) and +reasons appear as text. + +### CRD Conditions Browser + +The browser answers "which owners of this kind are in this state, and since when". Its variables narrow each other from +left to right: `kind` (single choice), then `condition`, `status`, `reason`, `namespace` and `resource_id`, all +multi-select with All, plus ad hoc filters. The Operator Conditions row shows the count of matching conditions and a +table of them by name, namespace, condition, status and reason with a since column; the collapsed Status Counts row +breaks the count down by `False`, `Unknown` and `True`. + +`resource_id` is what the alerts drill into: a `CustomResourceNotReady` notification opens the browser with `kind`, +`condition`, `status` and `resource_id` preset, and the CRs not Ready table on the operator dashboard does the same for +the row you click. + +Every multi variable answers All with the regular expression `.*` rather than a list of every value. That keeps the +query size constant however many owners exist, and it keeps cluster-scoped owners visible: their condition series carry +no namespace label, and a `=~".*"` matcher on an absent label matches, whereas a list of observed namespaces would not. + +### Stale series + +The condition gauge is exported by whichever pod recorded it. After a leader change the former leader keeps exporting +its last values until it restarts, so for a while two series describe one owner, and a plain `count()` double counts. +Every condition query in both dashboards joins on the freshest series per owner: + +```promql + and topk by (kind, id) (1, ) +``` + +The metric value is the `lastTransitionTime`, so `topk` keeps the most recently transitioned series for each owner and +drops the stale duplicate. The join carries `kind` because `id` is only `/`, and two owners of +different kinds can share it. Queries that span more than one condition type, such as the Unhealthy conditions panel, +join on `topk by (kind, id, condition)` instead, so each owner keeps one freshest series per condition rather than one +overall. The browser pins `kind` through its variable, so its queries join on `topk by (id, condition)`. The condition +alerts apply the same join before their status matcher, keyed on `(job, controller, kind, name, )` and, +for the rules that keep `condition` in their `by` clause, `condition`. `job` is in every grouping so two installs +exporting the same metric namespace never dedupe or merge across each other, and the browser link in each notification +carries it. `job` is the whole install identity these rules need: two installs only collide on a series when both export +the same CR (same controller, kind, namespace and name), which means both are reconciling the same object, a deployment +the framework does not support rather than one to alert on. + +#### Known limitation: equal timestamps + +A reason-only update keeps `lastTransitionTime` (`meta.SetStatusCondition` preserves it while the status is unchanged), +so after a leader change a former leader that is still being scraped can export the previous reason with exactly the +same value as the current leader's series. The alerts are unaffected, because every rule drops `reason` before it +aggregates and a tie implies the same status. The dashboards' reason-filtered panels can pick either series for as long +as both are scraped. The window is short: controller-runtime stops the manager when it loses the lease, so the old pod +restarts and its series disappear within a scrape interval or two. Dedupe on a leadership signal would close it, at the +cost of a fallback for operators that run without leader election, and is not done. + +## Previewing locally + +A clone of the repository can bring up Prometheus and Grafana with simulated operator data behind them, so you can look +at every panel and every alert before installing anything in a cluster. You need docker with the compose plugin and Go. + +```bash +make observability-up +``` + +Grafana serves on `http://localhost:3000` with anonymous admin access and both dashboards provisioned; Prometheus serves +on `http://localhost:9090`, with the alerts on `http://localhost:9090/alerts`. The simulator plays a scripted world in +which every panel is populated and every alert fires within a few minutes (`OperatorLeaderMissing` only with +`make observability-up SIMULATOR_ARGS="-leader=false"`). The simulator also serves kube-state-metrics lookalikes for its +own pod's requests and limits on `/ksm/metrics`, scraped as a separate `kube-state-metrics` job, so the Process panels +show their dashed lines. Stop the simulator with Ctrl-C, then remove the containers: + +```bash +make observability-down +``` + +Maintainers find the stack's internals, the scripted world and the tests that guard the templates in +`observability/README.md` in the repository. diff --git a/e2e/component/suite_test.go b/e2e/component/suite_test.go index 69f89e9b..70ea5664 100644 --- a/e2e/component/suite_test.go +++ b/e2e/component/suite_test.go @@ -57,7 +57,7 @@ var _ = BeforeSuite(func() { By("creating E2E reconciler") recorder := events.NewFakeRecorder(1000) metricsRecorder := metrics.NewRecorder( - "e2e-component", ocm.NewOperatorConditionsGauge("e2e_component"), metrics.NewCollectors(), + "clustertestapp", ocm.NewOperatorConditionsGauge("e2e_component"), metrics.NewCollectors(), ) clusterReconciler = framework.NewClusterE2EReconciler( mgr.GetClient(), diff --git a/e2e/primitives/suite_test.go b/e2e/primitives/suite_test.go index a2bc0a2d..e4e40b9e 100644 --- a/e2e/primitives/suite_test.go +++ b/e2e/primitives/suite_test.go @@ -82,21 +82,20 @@ var _ = BeforeSuite(func() { By("creating E2E reconcilers") recorder := events.NewFakeRecorder(1000) - metricsRecorder := metrics.NewRecorder( - "e2e-primitives", ocm.NewOperatorConditionsGauge("e2e_primitives"), metrics.NewCollectors(), - ) + conditions := ocm.NewOperatorConditionsGauge("e2e_primitives") + collectors := metrics.NewCollectors() reconciler = framework.NewE2EReconciler( mgr.GetClient(), mgr.GetScheme(), recorder, - metricsRecorder, + metrics.NewRecorder("testapp", conditions, collectors), mgr.GetAPIReader(), ) clusterReconciler = framework.NewClusterE2EReconciler( mgr.GetClient(), mgr.GetScheme(), recorder, - metricsRecorder, + metrics.NewRecorder("clustertestapp", conditions, collectors), mgr.GetAPIReader(), ) diff --git a/go.mod b/go.mod index fa7e1b64..ecfa8fb9 100644 --- a/go.mod +++ b/go.mod @@ -8,6 +8,7 @@ require ( github.com/onsi/gomega v1.42.1 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 github.com/prometheus/client_golang v1.23.2 + github.com/prometheus/client_model v0.6.2 github.com/sourcehawk/go-crd-condition-metrics v1.1.0 github.com/spf13/cobra v1.10.2 github.com/stretchr/testify v1.11.1 @@ -46,7 +47,6 @@ require ( github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect - github.com/prometheus/client_model v0.6.2 // indirect github.com/prometheus/common v0.67.5 // indirect github.com/prometheus/procfs v0.19.2 // indirect github.com/sourcehawk/go-prometheus-gaugevecset v1.1.0 // indirect diff --git a/mkdocs.yml b/mkdocs.yml index 6d751653..61c8bd5f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -112,6 +112,7 @@ nav: - CLI: cli.md - Guidelines: guidelines.md - Testing: testing.md + - Observability: observability.md - Reference: - Compatibility: compatibility.md - Go API ↗: https://pkg.go.dev/github.com/sourcehawk/operator-component-framework diff --git a/observability/README.md b/observability/README.md new file mode 100644 index 00000000..987a6e40 --- /dev/null +++ b/observability/README.md @@ -0,0 +1,90 @@ +# Observability + +Grafana dashboards and Prometheus alert rules for operators built on the framework, plus a local stack to look at them. +Full documentation: [docs/observability.md](../docs/observability.md). + +Render for your operator, where `METRIC_NAMESPACE` is the argument you gave `ocm.NewOperatorConditionsGauge`: + + make dashboards METRIC_NAMESPACE=myoperator + make alerts METRIC_NAMESPACE=myoperator + +Output lands in `generated/`. `generated/alerts/` contains the per-operator condition rules, named after the metric +namespace, plus the shared `ocf-*` rules for controller-runtime and the managed resource counters, which are installed +once per cluster. Add `NAMESPACE_LABEL=namespace` if your scrape keeps the exported `namespace` label, and +`ALERT_FORMAT=rules` for plain rule files instead of `PrometheusRule` objects. `PROMETHEUSRULE_NAMESPACE` and +`PROMETHEUSRULE_LABELS` set the metadata of the `PrometheusRule` objects; for kube-prometheus-stack pass +`PROMETHEUSRULE_LABELS=release=`. + +The rest of this file is for maintainers of the templates. + +## Local stack + +`dev/` holds a docker compose file for Prometheus and Grafana and a Go simulator that plays a scripted operator. The +simulator records the framework's own series through the real `metrics.Recorder`; the controller-runtime, workqueue, +REST client and leader election series are lookalikes with the same names, labels and buckets, guarded by the parity +test below. You need docker with the compose plugin and Go. + + make observability-up + +The target renders both dashboards and plain rule files for the metric namespace `demo` (`OBS_DEV_NAMESPACE`) into +`generated/dev/`, rewrites every `for:` in the rendered rules to `2m` so the alerts fire within minutes (this rewrite +exists only in the dev render), starts the containers, waits for Prometheus to become ready, and runs the simulator in +the foreground with `go run`. Prometheus listens on `127.0.0.1:9090` and Grafana on `127.0.0.1:3000` with anonymous +admin access and both dashboards provisioned into the OCF folder. Prometheus scrapes the simulator every five seconds +with the static target labels `job="demo-operator"`, `namespace="operators"` and `pod="demo-operator-0"`, so the +namespace the condition gauge exports collides into `exported_namespace` exactly as it does in a cluster. Stop the +simulator with Ctrl-C, then `make observability-down` removes the containers. + +`SIMULATOR_ARGS` passes extra flags to the simulator; `make observability-up SIMULATOR_ARGS="-leader=false"` reports the +replica as a standby so `OperatorLeaderMissing` fires. + +The world has two controllers, `webapp` (owner kind `WebApp`, components `server` and `ingress`) and `database` (owner +kind `Database`, components `storage` and `backup`). Each scenario targets one alert or one dashboard feature: + +| Scenario | Exercises | +| ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| Fifty healthy `WebApp` owners across five namespaces, reconciling on a jittered interval and converging with `none` | No alert fires for them; the baseline the ratio thresholds are tested against | +| `webapp-01`'s `server/configmap` applied as `updated` four times a second while its other resources converge | `ManagedResourceNotConverging`, the Updated rate and Not-converging ratio panels | +| `Database` `shop/orders-db` `Ready=False` with a `lastTransitionTime` eight hours in the past | `CustomResourceConditionStuck` at once, `CustomResourceNotReady` after the shortened `for:` | +| `Database` `shop/users-db` `Ready=Unknown` and `StorageReady=Unknown` | `CustomResourceConditionUnknown`, the Owners Unknown tile | +| `Database` `analytics/reports-db` `Ready=False` with its reason flipping every three minutes | `CustomResourceNotReady` keeps firing: a reason change does not restart the `for:` clock | +| A cluster-scoped `Database` `shared-gateway` with no namespace label | The CRD Conditions Browser's wildcard All keeps it visible; the alerts' cluster-scoped wording | +| `storage/pvc` applies failing three times out of four | `ManagedResourceApplyFailing`, the Apply error ratio panel | +| Thirty percent of `database` reconciles returning an error | `ControllerReconcileErrors` | +| `database` reconciles taking five seconds to two minutes | `ControllerReconcileLatencyHigh` | +| `database` queue wait observations of several minutes with a deep queue and both workers busy | `ControllerWorkqueueBacklog`, the Workqueue row | +| A `database` panic one minute in and every twenty minutes after | `ControllerReconcilePanics` | +| `leader_election_master_status{name="demo-operator"}` at 1, or 0 with `-leader=false` | The Leader tile, `OperatorLeaderMissing` | + +Within a few minutes of starting, every alert in the table except `OperatorLeaderMissing`, which needs +`SIMULATOR_ARGS="-leader=false"`, is visible as firing on `http://localhost:9090/alerts`. + +## Testing + +Three checks guard the artifacts. CI runs `make test-alerts` and `make lint-dashboards` in the "Alerts and dashboards" +job; the simulator parity test is an ordinary Go test and runs under `make test` in the unit-test job. + +`make test-alerts` needs `promtool`, which ships with Prometheus and is deliberately left out of `make all` so +contributors without it are unaffected. It renders `crd_conditions.tpl.yaml` with the metric namespace `test_operator` +into two directories, one with `exported_namespace` and one with `namespace` as the namespace label, copies the two +shared rule files into both, lints every file with `promtool check rules --lint=all --lint-fatal`, and runs the unit +tests with `promtool test rules --diff`. Each rule file has a test file of the same name under `alerts/tests/` +(`crd_conditions_test.yaml` for `crd_conditions.tpl.yaml`), and every alert has a firing case plus the negative cases +that justify its design, among them legitimate churn at scale and a single edit on an idle resource for +`ManagedResourceNotConverging`, sporadic conflicts among successful applies for `ManagedResourceApplyFailing`, and a +reason change mid-window and a cluster-scoped owner for the condition rules. Annotations are compared exactly, so a +wording change shows up as a diff. After editing a rule, run `make test-alerts` and `make lint-dashboards`; the latter +also asserts that every alert still has a unit test. + +`make lint-dashboards` runs `go test ./observability/`, which renders every template with a fixed namespace and checks +that each dashboard is valid JSON, keeps its uid (`_`) and its disabled auto-refresh, and that no +`{{placeholder}}` survived rendering. It then pulls every metric name out of every panel query, variable query and alert +expression and asserts each one exists: the framework's own families and the condition gauge are gathered from the real +collectors, the controller-runtime, workqueue, client-go, leader election and process families from a fixed list. For +the alerts it also asserts that every rule carries only the `severity` label and has a promtool unit test. + +`go test ./observability/dev/simulator/` includes the parity test that keeps the simulator honest: it starts a real, +unmanaged controller-runtime controller, drives one request through its workqueue so controller-runtime initialises its +reconcile and workqueue series, then asserts that every lookalike family the simulator defines exists in +controller-runtime's registry with the same type, label names and histogram buckets. A controller-runtime upgrade that +renames a label the dashboards depend on fails this test rather than emptying a panel. diff --git a/observability/alerts/controller_runtime.yaml b/observability/alerts/controller_runtime.yaml new file mode 100644 index 00000000..0b8031eb --- /dev/null +++ b/observability/alerts/controller_runtime.yaml @@ -0,0 +1,127 @@ +# Alerting rules for the controller-runtime metrics every operator exposes: +# reconcile results and latency, the workqueue, leader election. +# +# These rules carry no metric namespace placeholder: they are shared by every +# operator in the cluster and installed once, as the PrometheusRule +# `ocf-controller-runtime`. +# +# Thresholds are deliberately conservative and expressed as ratios or +# quantiles, so that they hold whether the operator manages three owners or +# three thousand. See docs/observability.md for the thresholds to tune. +# +# `namespace` is the operator's own namespace and `job` its scrape job, both +# stamped by Prometheus (absent when scraping outside a cluster, which is +# harmless). Keeping both in every aggregation alerts on two installs of one +# operator separately, and stops two operators in one namespace that share a +# controller name from diluting each other's ratios. OperatorLeaderMissing +# keys on the lease name instead, which is unique within a namespace. +groups: + - name: controller-runtime + rules: + - alert: ControllerReconcileErrors + for: 15m + expr: > + ( + sum by (namespace, job, controller) (rate(controller_runtime_reconcile_total{result="error"}[10m])) + / + sum by (namespace, job, controller) (rate(controller_runtime_reconcile_total[10m])) + ) > 0.25 + labels: + severity: warning + annotations: + summary: >- + Controller `{{ $labels.controller }}` fails {{ $value | humanizePercentage }} of its reconciles + description: | + Over the last 10 minutes, {{ $value | humanizePercentage }} of the reconciles of controller `{{ $labels.controller }}` returned an error. Errors are requeued with backoff, so a sustained ratio this high means the controller is stuck on something rather than riding out a blip. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep -i "error" | grep {{ $labels.controller }} + ``` + + - alert: ControllerReconcilePanics + expr: > + sum by (namespace, job, controller) (increase(controller_runtime_reconcile_panics_total[10m])) > 0 + labels: + severity: warning + annotations: + summary: >- + Controller `{{ $labels.controller }}` panicked during reconcile + description: | + Controller `{{ $labels.controller }}` recovered from {{ $value | printf "%.0f" }} panic(s) in the last 10 minutes. The reconcile that panicked was requeued, so the same owner will likely panic again on its next turn. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep -A 20 "Observed a panic" + ``` + + # Queue wait time rather than queue depth: depth has no threshold that is + # right for every operator, whereas items waiting minutes for a worker is + # wrong at any scale. controller-runtime labels every workqueue series + # with both `name` and `controller` (same value) in every version the + # framework supports (v0.22 and later), so aggregating by `controller` + # keeps one label across the reconcile rules, the workqueue rules and the + # dashboards. The workqueue histogram has one bucket per decade, so the + # threshold sits on a bucket bound: a p99 above 100 seconds means more + # than one percent of items waited longer than 100 seconds. The value in + # the description is interpolated within that bucket. + - alert: ControllerWorkqueueBacklog + for: 15m + expr: > + histogram_quantile( + 0.99, + sum by (namespace, job, controller, le) (rate(workqueue_queue_duration_seconds_bucket[10m])) + ) > 100 + labels: + severity: warning + annotations: + summary: >- + Controller `{{ $labels.controller }}` cannot keep up with its workqueue + description: | + More than one percent of the items in the workqueue of controller `{{ $labels.controller }}` waited longer than 100 seconds before a worker picked them up (p99 wait {{ $value | humanizeDuration }}, interpolated within the histogram bucket). The queue grows faster than the workers drain it: too few concurrent reconciles, reconciles that take too long, or a burst of events the controller cannot absorb. + + Quick check with: + ``` + kubectl top pod -l control-plane=controller-manager + ``` + + - alert: ControllerReconcileLatencyHigh + for: 15m + expr: > + histogram_quantile( + 0.99, + sum by (namespace, job, controller, le) (rate(controller_runtime_reconcile_time_seconds_bucket[10m])) + ) > 30 + labels: + severity: warning + annotations: + summary: >- + Controller `{{ $labels.controller }}` reconciles are slow, p99 above 30 seconds + description: | + The p99 reconcile time of controller `{{ $labels.controller }}` over the last 10 minutes is {{ $value | humanizeDuration }}, above the 30 second threshold. Every slow reconcile holds a worker, so sustained latency this high turns into a workqueue backlog. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep {{ $labels.controller }} | tail -50 + ``` + + # Silent when leader election is off (the gauge is not exported) and when + # no replica is alive to export it, for example a crash loop before the + # elector starts; pair it with your platform's target-down alert for the + # operator job. + - alert: OperatorLeaderMissing + for: 5m + expr: > + max by (namespace, name) (leader_election_master_status) == 0 + labels: + severity: warning + annotations: + summary: >- + Operator `{{ $labels.name }}` has no leader + description: | + No replica of operator `{{ $labels.name }}` has held the leader election lease for 5 minutes, so none of its controllers is reconciling. + + Quick check with: + ``` + kubectl get lease -A --field-selector metadata.name={{ $labels.name }} + ``` diff --git a/observability/alerts/crd_conditions.tpl.yaml b/observability/alerts/crd_conditions.tpl.yaml new file mode 100644 index 00000000..98635233 --- /dev/null +++ b/observability/alerts/crd_conditions.tpl.yaml @@ -0,0 +1,191 @@ +# Alerting rules for the status condition metrics the framework records through +# pkg/metrics (github.com/sourcehawk/go-crd-condition-metrics under the hood). +# +# See docs/observability.md for how these rules are rendered and which +# thresholds to tune. +# +# Every rule aggregates the metric instead of matching series directly. That is +# deliberate and load bearing: +# +# * it drops the reason, status and id labels, so a controller that keeps +# changing the reason while a resource stays unhealthy does not restart the +# `for:` clock every time, +# * max() keeps the value a lastTransitionTime, the freshest one, so two +# series for one owner (a reason change still inside the lookback window) +# collapse to the freshest rather than adding up the way sum() would. +# +# The status matcher alone is not enough against a former leader pod that keeps +# exporting its last values until it restarts: its stale `False` series would +# fire even though the current leader reports `True`. So every rule first joins +# on the freshest series per owner across every status, the same +# `and topk by (...) (1, ...)` join the dashboards use, and only then applies +# its status matcher. A stale series loses the join because the current leader's +# series, whatever its status, carries the later lastTransitionTime. +# +# Every grouping also keeps `job`, the scrape job Prometheus stamps on the +# series, so two installs of the operator that export the same metric namespace +# never dedupe or merge across each other, and the browser link opens on the +# install that fired. +groups: + - name: crd-conditions + rules: + # A resource whose Ready condition has been False for too long. + # + # Scoped to Ready on purpose. Matching `status="False"` across every + # condition would fire forever on negative polarity conditions such as + # Degraded or FailedToProvision, where False is the healthy state. Add + # your own positive polarity conditions to the matcher below. + - alert: CustomResourceNotReady + for: 30m + expr: > + max ( + {{operator_namespace}}controller_condition{condition="Ready", status="False"} + and + topk by (job, controller, kind, name, {{namespace_label}}) ( + 1, {{operator_namespace}}controller_condition{condition="Ready"} + ) + ) by (job, controller, kind, name, {{namespace_label}}) + labels: + severity: warning + annotations: + summary: >- + {{- if $labels.{{namespace_label}} -}} + {{ $labels.kind }} CR `{{ $labels.{{namespace_label}} }}/{{ $labels.name }}` has not been ready for 30 minutes + {{- else -}} + {{ $labels.kind }} CR `{{ $labels.name }}` has not been ready for 30 minutes + {{- end -}} + description: | + {{- if $labels.{{namespace_label}} -}} + Custom Resource of kind `{{ $labels.kind }}` named `{{ $labels.name }}` in namespace + `{{ $labels.{{namespace_label}} }}` has not been ready for more than 30 minutes. + {{- else -}} + Custom Resource of kind `{{ $labels.kind }}` named `{{ $labels.name }}` (cluster scoped + resource) has not been ready for more than 30 minutes. + {{- end }} + + The Ready condition last transitioned at {{ $value | humanizeTimestamp }}. + + Quick check with: + ``` + {{- if $labels.{{namespace_label}} }} + kubectl describe {{ $labels.kind }}/{{ $labels.name }} -n {{ $labels.{{namespace_label}} }} + {{- else }} + kubectl describe {{ $labels.kind }}/{{ $labels.name }} + {{- end }} + ``` + # Deep link into the CRD Conditions Browser dashboard shipped in this + # repository. The id label is "/", which for a cluster + # scoped resource is "/", so the empty namespace renders correctly + # here without a conditional. + dashboard_url: >- + /d/{{operator_namespace}}crd_conditions_browser/crd-conditions-browser?var-job={{ $labels.job }}&var-kind={{ $labels.kind + }}&var-condition=Ready&var-status=False&var-resource_id={{ $labels.{{namespace_label}} }}%2F{{ $labels.name }} + + # A condition the controller cannot determine the state of. Unlike the two + # rules around it this one is not scoped to a single condition: Unknown is + # bad whatever the condition's polarity is. + - alert: CustomResourceConditionUnknown + for: 30m + expr: > + max ( + {{operator_namespace}}controller_condition{status="Unknown"} + and + topk by (job, controller, kind, name, condition, {{namespace_label}}) ( + 1, {{operator_namespace}}controller_condition + ) + ) by (job, controller, kind, name, condition, {{namespace_label}}) + labels: + severity: warning + annotations: + summary: >- + {{- if $labels.{{namespace_label}} -}} + {{ $labels.kind }} CR `{{ $labels.{{namespace_label}} }}/{{ $labels.name }}` has had an unknown {{ $labels.condition }} condition for 30 minutes + {{- else -}} + {{ $labels.kind }} CR `{{ $labels.name }}` has had an unknown {{ $labels.condition }} condition for 30 minutes + {{- end -}} + description: | + {{- if $labels.{{namespace_label}} -}} + The `{{ $labels.condition }}` condition of `{{ $labels.kind }}` `{{ $labels.name }}` in + namespace `{{ $labels.{{namespace_label}} }}` has had status Unknown for more than 30 minutes. + {{- else -}} + The `{{ $labels.condition }}` condition of `{{ $labels.kind }}` `{{ $labels.name }}` + (cluster scoped resource) has had status Unknown for more than 30 minutes. + {{- end }} + + The controller is failing to determine the state of this resource, so neither a healthy + nor an unhealthy signal can be trusted for it. The condition last transitioned at + {{ $value | humanizeTimestamp }}. + + Quick check with: + ``` + {{- if $labels.{{namespace_label}} }} + kubectl describe {{ $labels.kind }}/{{ $labels.name }} -n {{ $labels.{{namespace_label}} }} + {{- else }} + kubectl describe {{ $labels.kind }}/{{ $labels.name }} + {{- end }} + ``` + dashboard_url: >- + /d/{{operator_namespace}}crd_conditions_browser/crd-conditions-browser?var-job={{ $labels.job }}&var-kind={{ $labels.kind }}&var-condition={{ + $labels.condition }}&var-status=Unknown&var-resource_id={{ $labels.{{namespace_label}} }}%2F{{ $labels.name }} + + # A resource that has sat in a non-ready state for hours. + # + # This rule is the reason the metric value is a lastTransitionTime. It needs + # no `for:` clause because the expression is itself a duration comparison, + # and that difference is the point: a `for:` clause measures how long the + # *alert* has been true, which is bounded by how long the series has been + # continuously present, whereas this measures how long the *resource* has + # actually been in this state according to its own status. It therefore + # survives scrape gaps, operator restarts and ruler restarts, any of which + # silently restart a `for:` clock, and reports the real age of the state. + # It also covers status Unknown, which CustomResourceNotReady does not. + # + # Scoped to Ready for the same polarity reason as CustomResourceNotReady. + # The condition label is kept in the aggregation so widening the matcher to + # your own conditions stays a one line change. + - alert: CustomResourceConditionStuck + expr: > + ( + time() + - + max ( + {{operator_namespace}}controller_condition{condition="Ready", status!="True"} + and + topk by (job, controller, kind, name, condition, {{namespace_label}}) ( + 1, {{operator_namespace}}controller_condition{condition="Ready"} + ) + ) by (job, controller, kind, name, condition, {{namespace_label}}) + ) > 21600 + labels: + severity: warning + annotations: + summary: >- + {{- if $labels.{{namespace_label}} -}} + {{ $labels.kind }} CR `{{ $labels.{{namespace_label}} }}/{{ $labels.name }}` has been stuck not ready for {{ $value | humanizeDuration }} + {{- else -}} + {{ $labels.kind }} CR `{{ $labels.name }}` has been stuck not ready for {{ $value | humanizeDuration }} + {{- end -}} + description: | + {{- if $labels.{{namespace_label}} -}} + The `{{ $labels.condition }}` condition of `{{ $labels.kind }}` `{{ $labels.name }}` in + namespace `{{ $labels.{{namespace_label}} }}` has not been True for + {{ $value | humanizeDuration }}. + {{- else -}} + The `{{ $labels.condition }}` condition of `{{ $labels.kind }}` `{{ $labels.name }}` + (cluster scoped resource) has not been True for {{ $value | humanizeDuration }}. + {{- end }} + + Nothing has moved for hours, so the controller is either wedged or waiting on something + that is never going to arrive. Reconciliation retries alone will not clear this. + + Quick check with: + ``` + {{- if $labels.{{namespace_label}} }} + kubectl describe {{ $labels.kind }}/{{ $labels.name }} -n {{ $labels.{{namespace_label}} }} + {{- else }} + kubectl describe {{ $labels.kind }}/{{ $labels.name }} + {{- end }} + ``` + dashboard_url: >- + /d/{{operator_namespace}}crd_conditions_browser/crd-conditions-browser?var-job={{ $labels.job }}&var-kind={{ $labels.kind }}&var-condition={{ + $labels.condition }}&var-resource_id={{ $labels.{{namespace_label}} }}%2F{{ $labels.name }} diff --git a/observability/alerts/managed_resources.yaml b/observability/alerts/managed_resources.yaml new file mode 100644 index 00000000..b5646d6c --- /dev/null +++ b/observability/alerts/managed_resources.yaml @@ -0,0 +1,112 @@ +# Alerting rules for the resource apply counters recorded by +# github.com/sourcehawk/operator-component-framework/pkg/metrics. +# +# These rules carry no metric namespace placeholder: they are shared by every +# operator in the cluster and installed once, as the PrometheusRule +# `ocf-managed-resources`. +# +# Neither counter carries an owner name or namespace, only the operator's static +# topology (controller, owner kind, component, resource identifier, kind), so +# these rules fire per resource *type* rather than per owner. Per-owner signal +# comes from the condition rules in crd_conditions.tpl.yaml. +# +# Both rules are ratios of the resource's own applies rather than absolute +# rates. At scale, legitimate spec changes across many owners keep the absolute +# `updated` rate above zero indefinitely; what distinguishes a resource that +# never converges is that more than half of its applies rewrite it, at a rate +# no human produces. See docs/observability.md for the thresholds to tune. +# +# `namespace` is the operator's own namespace and `job` its scrape job, both +# stamped by Prometheus (absent when scraping outside a cluster, which is +# harmless). Keeping both in every aggregation alerts on two installs of one +# operator separately, and stops two operators in one namespace whose +# controller and topology labels coincide from diluting each other's ratios. +groups: + - name: managed-resources + rules: + # A managed resource rewritten on every reconcile. A converged resource + # applies as `none`; one whose applies keep coming back `updated` is in a + # hot loop (apply updates the object, the watch event requeues the owner, + # the next apply updates it again) or in a fight with a webhook or another + # controller. Legitimate churn is followed by a `none` apply on the next + # reconcile, which keeps its ratio at or below one half. The floor keeps a + # single edit on an idle resource from producing a ratio of one over a + # handful of samples. + - alert: ManagedResourceNotConverging + for: 15m + expr: > + ( + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + rate(ocf_resource_apply_total{operation="updated"}[15m]) + ) + / + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + rate(ocf_resource_apply_total[15m]) + ) + ) > 0.5 + and + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + increase(ocf_resource_apply_total{operation="updated"}[15m]) + ) > 15 + labels: + severity: warning + annotations: + summary: >- + {{ $labels.kind }} `{{ $labels.resource }}` of component `{{ $labels.component }}` ({{ $labels.owner_kind }}) is rewritten on most reconciles + description: | + Over the last 15 minutes, {{ $value | humanizePercentage }} of the applies of the `{{ $labels.kind }}` resource `{{ $labels.resource }}` in component `{{ $labels.component }}` of owner kind `{{ $labels.owner_kind }}` (controller `{{ $labels.controller }}`) updated the object. In steady state a converged resource applies as `none` on every pass, so a resource whose applies are mostly updates is being rewritten on reconciles where nothing should have changed. + + Usual causes: a mutation that is not idempotent (a timestamp, a random value, an unsorted list), a defaulting webhook or another controller fighting the operator, or a field the API server normalises. Events report the same thing, but client-go's spam filter truncates them within seconds under exactly these conditions. + + Quick check with: + ``` + kubectl get events -A --field-selector reason=Updated{{ $labels.kind }} + ``` + + # Applies of one resource type mostly failing. Transient conflicts among + # successful applies stay under the threshold, and a resource failing for + # a single owner among many is the condition rules' job. The `or` keeps + # the ratio defined for a resource that has never applied successfully, + # where no success series exists to add. + - alert: ManagedResourceApplyFailing + for: 15m + expr: > + ( + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + rate(ocf_resource_apply_errors_total[15m]) + ) + / + ( + ( + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + rate(ocf_resource_apply_errors_total[15m]) + ) + + + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + rate(ocf_resource_apply_total[15m]) + ) + ) + or + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + rate(ocf_resource_apply_errors_total[15m]) + ) + ) + ) > 0.5 + and + sum by (namespace, job, controller, owner_kind, component, resource, kind) ( + increase(ocf_resource_apply_errors_total[15m]) + ) > 5 + labels: + severity: warning + annotations: + summary: >- + Applies of {{ $labels.kind }} `{{ $labels.resource }}` of component `{{ $labels.component }}` ({{ $labels.owner_kind }}) are failing + description: | + Over the last 15 minutes, {{ $value | humanizePercentage }} of the apply attempts of the `{{ $labels.kind }}` resource `{{ $labels.resource }}` in component `{{ $labels.component }}` of owner kind `{{ $labels.owner_kind }}` (controller `{{ $labels.controller }}`) failed. + + The counter covers every failure of an attempt: mutating the desired object, the server-side apply patch, and the classification that follows it. The owners whose resource fails report the error on their Ready condition, which is where the failure message lives; this rule says the failure is systematic for the resource type rather than one owner's bad day. + + Quick check with: + ``` + kubectl get {{ $labels.owner_kind }} -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[?(@.type=="Ready")].reason,MESSAGE:.status.conditions[?(@.type=="Ready")].message' + ``` diff --git a/observability/alerts/tests/controller_runtime_test.yaml b/observability/alerts/tests/controller_runtime_test.yaml new file mode 100644 index 00000000..ca274695 --- /dev/null +++ b/observability/alerts/tests/controller_runtime_test.yaml @@ -0,0 +1,240 @@ +# Unit tests for the controller-runtime alerting rules. Run with `make test-alerts`. +rule_files: + - ../controller_runtime.yaml + +evaluation_interval: 1m + +tests: + - name: TestControllerReconcileErrors + interval: 1m + input_series: + # database: 3 of 10 reconciles a minute fail + - series: 'controller_runtime_reconcile_total{controller="database",result="error"}' + values: '0+3x60' + - series: 'controller_runtime_reconcile_total{controller="database",result="success"}' + values: '0+7x60' + # webapp: 1 of 10 fails, under the threshold + - series: 'controller_runtime_reconcile_total{controller="webapp",result="error"}' + values: '0+1x60' + - series: 'controller_runtime_reconcile_total{controller="webapp",result="success"}' + values: '0+9x60' + alert_rule_test: + - eval_time: 10m + alertname: ControllerReconcileErrors + exp_alerts: [] + - eval_time: 26m + alertname: ControllerReconcileErrors + exp_alerts: + - exp_labels: + controller: database + severity: warning + exp_annotations: + summary: >- + Controller `database` fails 30% of its reconciles + description: | + Over the last 10 minutes, 30% of the reconciles of controller `database` returned an error. Errors are requeued with backoff, so a sustained ratio this high means the controller is stuck on something rather than riding out a blip. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep -i "error" | grep database + ``` + + # Two installs of the same operator in one cluster share controller names. + # The scrape job stamps each with its own namespace, so the broken install + # must fire on its own and the healthy one must not be dragged along. + - name: TestControllerReconcileErrorsPerInstall + interval: 1m + input_series: + - series: 'controller_runtime_reconcile_total{namespace="team-a",controller="database",result="error"}' + values: '0+10x60' + - series: 'controller_runtime_reconcile_total{namespace="team-a",controller="database",result="success"}' + values: '0+0x60' + - series: 'controller_runtime_reconcile_total{namespace="team-b",controller="database",result="error"}' + values: '0+0x60' + - series: 'controller_runtime_reconcile_total{namespace="team-b",controller="database",result="success"}' + values: '0+40x60' + alert_rule_test: + - eval_time: 26m + alertname: ControllerReconcileErrors + exp_alerts: + - exp_labels: + namespace: team-a + controller: database + severity: warning + exp_annotations: + summary: >- + Controller `database` fails 100% of its reconciles + description: | + Over the last 10 minutes, 100% of the reconciles of controller `database` returned an error. Errors are requeued with backoff, so a sustained ratio this high means the controller is stuck on something rather than riding out a blip. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep -i "error" | grep database + ``` + + - name: TestControllerReconcilePanics + interval: 1m + input_series: + - series: 'controller_runtime_reconcile_panics_total{controller="database"}' + values: '0 0 0 1+0x30' + alert_rule_test: + - eval_time: 2m + alertname: ControllerReconcilePanics + exp_alerts: [] + - eval_time: 5m + alertname: ControllerReconcilePanics + exp_alerts: + - exp_labels: + controller: database + severity: warning + exp_annotations: + summary: >- + Controller `database` panicked during reconcile + description: | + Controller `database` recovered from 1 panic(s) in the last 10 minutes. The reconcile that panicked was requeued, so the same owner will likely panic again on its next turn. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep -A 20 "Observed a panic" + ``` + # Out of the 10m window again + - eval_time: 20m + alertname: ControllerReconcilePanics + exp_alerts: [] + + - name: TestControllerWorkqueueBacklog + interval: 1m + input_series: + # database: items wait between 100s and 1000s before being picked up + - series: 'workqueue_queue_duration_seconds_bucket{controller="database",name="database",le="100"}' + values: '0+0x60' + - series: 'workqueue_queue_duration_seconds_bucket{controller="database",name="database",le="1000"}' + values: '0+10x60' + - series: 'workqueue_queue_duration_seconds_bucket{controller="database",name="database",le="+Inf"}' + values: '0+10x60' + # webapp: everything under 100s + - series: 'workqueue_queue_duration_seconds_bucket{controller="webapp",name="webapp",le="100"}' + values: '0+10x60' + - series: 'workqueue_queue_duration_seconds_bucket{controller="webapp",name="webapp",le="1000"}' + values: '0+10x60' + - series: 'workqueue_queue_duration_seconds_bucket{controller="webapp",name="webapp",le="+Inf"}' + values: '0+10x60' + alert_rule_test: + - eval_time: 10m + alertname: ControllerWorkqueueBacklog + exp_alerts: [] + - eval_time: 26m + alertname: ControllerWorkqueueBacklog + exp_alerts: + - exp_labels: + controller: database + severity: warning + exp_annotations: + summary: >- + Controller `database` cannot keep up with its workqueue + description: | + More than one percent of the items in the workqueue of controller `database` waited longer than 100 seconds before a worker picked them up (p99 wait 16m 31s, interpolated within the histogram bucket). The queue grows faster than the workers drain it: too few concurrent reconciles, reconciles that take too long, or a burst of events the controller cannot absorb. + + Quick check with: + ``` + kubectl top pod -l control-plane=controller-manager + ``` + + - name: TestControllerReconcileLatencyHigh + interval: 1m + input_series: + - series: 'controller_runtime_reconcile_time_seconds_bucket{controller="database",le="30"}' + values: '0+0x60' + - series: 'controller_runtime_reconcile_time_seconds_bucket{controller="database",le="60"}' + values: '0+10x60' + - series: 'controller_runtime_reconcile_time_seconds_bucket{controller="database",le="+Inf"}' + values: '0+10x60' + - series: 'controller_runtime_reconcile_time_seconds_bucket{controller="webapp",le="30"}' + values: '0+10x60' + - series: 'controller_runtime_reconcile_time_seconds_bucket{controller="webapp",le="60"}' + values: '0+10x60' + - series: 'controller_runtime_reconcile_time_seconds_bucket{controller="webapp",le="+Inf"}' + values: '0+10x60' + alert_rule_test: + - eval_time: 10m + alertname: ControllerReconcileLatencyHigh + exp_alerts: [] + - eval_time: 26m + alertname: ControllerReconcileLatencyHigh + exp_alerts: + - exp_labels: + controller: database + severity: warning + exp_annotations: + summary: >- + Controller `database` reconciles are slow, p99 above 30 seconds + description: | + The p99 reconcile time of controller `database` over the last 10 minutes is 59.7s, above the 30 second threshold. Every slow reconcile holds a worker, so sustained latency this high turns into a workqueue backlog. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep database | tail -50 + ``` + + - name: TestOperatorLeaderMissing + interval: 1m + input_series: + # the leader steps down at 10m and nobody takes over + - series: 'leader_election_master_status{name="demo-operator"}' + values: '1+0x10 0+0x50' + alert_rule_test: + - eval_time: 5m + alertname: OperatorLeaderMissing + exp_alerts: [] + - eval_time: 12m + alertname: OperatorLeaderMissing + exp_alerts: [] + - eval_time: 16m + alertname: OperatorLeaderMissing + exp_alerts: + - exp_labels: + name: demo-operator + severity: warning + exp_annotations: + summary: >- + Operator `demo-operator` has no leader + description: | + No replica of operator `demo-operator` has held the leader election lease for 5 minutes, so none of its controllers is reconciling. + + Quick check with: + ``` + kubectl get lease -A --field-selector metadata.name=demo-operator + ``` + + # Two operators scraped as different jobs in one namespace, both with a + # controller named `database`. Merged across jobs beta's 20 clean reconciles a + # minute would dilute alpha's 5 of 10 failures to 5/30 and mask them; keyed + # on `job` alpha fires on its own and beta stays silent. + - name: TestControllerReconcileErrorsKeyedOnJob + interval: 1m + input_series: + - series: 'controller_runtime_reconcile_total{namespace="operators",job="alpha",controller="database",result="error"}' + values: '0+5x60' + - series: 'controller_runtime_reconcile_total{namespace="operators",job="alpha",controller="database",result="success"}' + values: '0+5x60' + - series: 'controller_runtime_reconcile_total{namespace="operators",job="beta",controller="database",result="success"}' + values: '0+20x60' + alert_rule_test: + - eval_time: 26m + alertname: ControllerReconcileErrors + exp_alerts: + - exp_labels: + namespace: operators + job: alpha + controller: database + severity: warning + exp_annotations: + summary: >- + Controller `database` fails 50% of its reconciles + description: | + Over the last 10 minutes, 50% of the reconciles of controller `database` returned an error. Errors are requeued with backoff, so a sustained ratio this high means the controller is stuck on something rather than riding out a blip. + + Quick check with: + ``` + kubectl logs -l control-plane=controller-manager --all-containers | grep -i "error" | grep database + ``` diff --git a/observability/alerts/tests/crd_conditions_test.yaml b/observability/alerts/tests/crd_conditions_test.yaml new file mode 100644 index 00000000..423e80ac --- /dev/null +++ b/observability/alerts/tests/crd_conditions_test.yaml @@ -0,0 +1,357 @@ +# Unit tests for the generic CRD condition alerting rules. +# +# Run with `make test-alerts`, which renders ../crd_conditions.tpl.yaml with the +# metric namespace pinned to `test_operator` and the namespace label left at its +# default before handing both files to promtool. +rule_files: + - ../crd_conditions.yaml + +evaluation_interval: 1m + +tests: + # A reason change while a resource stays not ready must not restart the 30m + # clock, and a resource with no namespace label (cluster scoped) must produce + # the alternate wording. + - name: TestCustomResourceNotReady + interval: 1m + input_series: + # Backup Ready=False reason=Error1 [0m-19m] + - series: 'test_operator_controller_condition{controller="backup",kind="Backup",name="nightly-abc123",exported_namespace="backups",condition="Ready",status="False",reason="Error1",id="backups/nightly-abc123"}' + values: '1760534009+0x19' + # Backup Ready=False reason=Error2 [20m-44m]. A reason-only change keeps + # lastTransitionTime (meta.SetStatusCondition preserves it while the + # status stays False), so the value is the same as Error1's. The Error1 + # series is still within the 5m lookback window until 24m, so both are + # present for a while and the aggregation must collapse them to one. + - series: 'test_operator_controller_condition{controller="backup",kind="Backup",name="nightly-abc123",exported_namespace="backups",condition="Ready",status="False",reason="Error2",id="backups/nightly-abc123"}' + values: '_x20 1760534009+0x24' + # Backup Ready=True [45m-60m] + - series: 'test_operator_controller_condition{controller="backup",kind="Backup",name="nightly-abc123",exported_namespace="backups",condition="Ready",status="True",reason="Ready",id="backups/nightly-abc123"}' + values: '_x45 1760534420+0x15' + # RemoteStorage Ready=False [15m-60m], cluster scoped so no namespace label + - series: 'test_operator_controller_condition{controller="remotestorage",kind="RemoteStorage",name="document-store",condition="Ready",status="False",reason="Error",id="/document-store"}' + values: '_x15 1760582000+0x45' + + alert_rule_test: + # Nothing fires immediately (for: 30m) + - eval_time: 0m + alertname: CustomResourceNotReady + exp_alerts: [] + + # Still pending at 25m: under the threshold, and the Error1 -> Error2 + # change at 20m must not have restarted the clock. + - eval_time: 25m + alertname: CustomResourceNotReady + exp_alerts: [] + + # Backup has been not ready for 30m. RemoteStorage only started at 15m so + # it is still pending. + - eval_time: 30m + alertname: CustomResourceNotReady + exp_alerts: + - exp_labels: + controller: backup + kind: Backup + name: nightly-abc123 + exported_namespace: backups + severity: warning + exp_annotations: + summary: >- + Backup CR `backups/nightly-abc123` has not been ready for 30 minutes + description: | + Custom Resource of kind `Backup` named `nightly-abc123` in namespace + `backups` has not been ready for more than 30 minutes. + + The Ready condition last transitioned at 2025-10-15 13:13:29 +0000 UTC. + + Quick check with: + ``` + kubectl describe Backup/nightly-abc123 -n backups + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=&var-kind=Backup&var-condition=Ready&var-status=False&var-resource_id=backups%2Fnightly-abc123 + + # Backup recovered at 45m, so it has resolved. RemoteStorage has now been + # not ready for 30m and is the only alert left. + - eval_time: 50m + alertname: CustomResourceNotReady + exp_alerts: + - exp_labels: + controller: remotestorage + kind: RemoteStorage + name: document-store + severity: warning + exp_annotations: + summary: >- + RemoteStorage CR `document-store` has not been ready for 30 minutes + description: | + Custom Resource of kind `RemoteStorage` named `document-store` (cluster scoped + resource) has not been ready for more than 30 minutes. + + The Ready condition last transitioned at 2025-10-16 02:33:20 +0000 UTC. + + Quick check with: + ``` + kubectl describe RemoteStorage/document-store + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=&var-kind=RemoteStorage&var-condition=Ready&var-status=False&var-resource_id=%2Fdocument-store + + # Unknown is bad whatever the condition's polarity is, so this rule is not + # scoped to a single condition: both fixtures below must fire. + - name: TestCustomResourceConditionUnknown + interval: 1m + input_series: + - series: 'test_operator_controller_condition{controller="app",kind="App",name="checkout",exported_namespace="shop",condition="Synced",status="Unknown",reason="ProbeFailed",id="shop/checkout"}' + values: '1760600000+0x60' + - series: 'test_operator_controller_condition{controller="app",kind="App",name="checkout",exported_namespace="shop",condition="Ready",status="Unknown",reason="ProbeFailed",id="shop/checkout"}' + values: '1760600000+0x60' + + alert_rule_test: + - eval_time: 25m + alertname: CustomResourceConditionUnknown + exp_alerts: [] + + - eval_time: 30m + alertname: CustomResourceConditionUnknown + exp_alerts: + - exp_labels: + controller: app + kind: App + name: checkout + condition: Ready + exported_namespace: shop + severity: warning + exp_annotations: + summary: >- + App CR `shop/checkout` has had an unknown Ready condition for 30 minutes + description: | + The `Ready` condition of `App` `checkout` in + namespace `shop` has had status Unknown for more than 30 minutes. + + The controller is failing to determine the state of this resource, so neither a healthy + nor an unhealthy signal can be trusted for it. The condition last transitioned at + 2025-10-16 07:33:20 +0000 UTC. + + Quick check with: + ``` + kubectl describe App/checkout -n shop + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=&var-kind=App&var-condition=Ready&var-status=Unknown&var-resource_id=shop%2Fcheckout + - exp_labels: + controller: app + kind: App + name: checkout + condition: Synced + exported_namespace: shop + severity: warning + exp_annotations: + summary: >- + App CR `shop/checkout` has had an unknown Synced condition for 30 minutes + description: | + The `Synced` condition of `App` `checkout` in + namespace `shop` has had status Unknown for more than 30 minutes. + + The controller is failing to determine the state of this resource, so neither a healthy + nor an unhealthy signal can be trusted for it. The condition last transitioned at + 2025-10-16 07:33:20 +0000 UTC. + + Quick check with: + ``` + kubectl describe App/checkout -n shop + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=&var-kind=App&var-condition=Synced&var-status=Unknown&var-resource_id=shop%2Fcheckout + + # The stuck rule compares against the metric value, so the fixture timestamps + # have to live on promtool's clock, which starts at the unix epoch. A Ready + # condition that transitioned at t=60s is 6h59m old at an eval_time of 7h. + - name: TestCustomResourceConditionStuck + interval: 1m + input_series: + - series: 'test_operator_controller_condition{controller="cluster",kind="Cluster",name="prod",exported_namespace="clusters",condition="Ready",status="False",reason="Provisioning",id="clusters/prod"}' + values: '60+0x480' + + alert_rule_test: + # 5h since the transition, under the 6h threshold + - eval_time: 5h + alertname: CustomResourceConditionStuck + exp_alerts: [] + + - eval_time: 7h + alertname: CustomResourceConditionStuck + exp_alerts: + - exp_labels: + controller: cluster + kind: Cluster + name: prod + condition: Ready + exported_namespace: clusters + severity: warning + exp_annotations: + summary: >- + Cluster CR `clusters/prod` has been stuck not ready for 6h 59m 0s + description: | + The `Ready` condition of `Cluster` `prod` in + namespace `clusters` has not been True for + 6h 59m 0s. + + Nothing has moved for hours, so the controller is either wedged or waiting on something + that is never going to arrive. Reconciliation retries alone will not clear this. + + Quick check with: + ``` + kubectl describe Cluster/prod -n clusters + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=&var-kind=Cluster&var-condition=Ready&var-resource_id=clusters%2Fprod + + # Why CustomResourceConditionStuck exists alongside CustomResourceNotReady: + # the series disappears between 1h and 6h (operator down, scrape gap), which + # restarts the `for: 30m` clock on its return. 20m after it comes back the + # `for:` based rule is still pending, while the value based rule correctly + # reports the resource as stuck for 6h19m. + - name: TestConditionStuckSurvivesSeriesGaps + interval: 1m + input_series: + - series: 'test_operator_controller_condition{controller="cluster",kind="Cluster",name="staging",exported_namespace="clusters",condition="Ready",status="False",reason="Provisioning",id="clusters/staging"}' + values: '60+0x60 _x300 60+0x60' + + alert_rule_test: + - eval_time: 6h20m + alertname: CustomResourceNotReady + exp_alerts: [] + + - eval_time: 6h20m + alertname: CustomResourceConditionStuck + exp_alerts: + - exp_labels: + controller: cluster + kind: Cluster + name: staging + condition: Ready + exported_namespace: clusters + severity: warning + exp_annotations: + summary: >- + Cluster CR `clusters/staging` has been stuck not ready for 6h 19m 0s + description: | + The `Ready` condition of `Cluster` `staging` in + namespace `clusters` has not been True for + 6h 19m 0s. + + Nothing has moved for hours, so the controller is either wedged or waiting on something + that is never going to arrive. Reconciliation retries alone will not clear this. + + Quick check with: + ``` + kubectl describe Cluster/staging -n clusters + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=&var-kind=Cluster&var-condition=Ready&var-resource_id=clusters%2Fstaging + + # A former leader pod keeps exporting its last values until it restarts, so + # for a while two pods describe one owner. The stale series from the old pod + # must not fire while the current leader reports the owner as Ready, but a + # current leader that also reports it not ready must still fire, on the + # current leader's value. + - name: TestStaleFormerLeaderSeriesDoNotFire + interval: 1m + input_series: + # Old pod: Backup Ready=False, transitioned at 13:13:29, scraped throughout. + - series: 'test_operator_controller_condition{pod="operator-old",controller="backup",kind="Backup",name="weekly",exported_namespace="backups",condition="Ready",status="False",reason="Error",id="backups/weekly"}' + values: '1760534009+0x60' + # Current pod: Backup Ready=True, transitioned later at 13:20:20. + - series: 'test_operator_controller_condition{pod="operator-new",controller="backup",kind="Backup",name="weekly",exported_namespace="backups",condition="Ready",status="True",reason="Ready",id="backups/weekly"}' + values: '1760534420+0x60' + # Old pod: App Synced=Unknown, scraped throughout. + - series: 'test_operator_controller_condition{pod="operator-old",controller="app",kind="App",name="checkout",exported_namespace="shop",condition="Synced",status="Unknown",reason="ProbeFailed",id="shop/checkout"}' + values: '1760534009+0x60' + # Current pod: App Synced=True, transitioned later. + - series: 'test_operator_controller_condition{pod="operator-new",controller="app",kind="App",name="checkout",exported_namespace="shop",condition="Synced",status="True",reason="Synced",id="shop/checkout"}' + values: '1760534420+0x60' + # Old pod: Cluster Ready=False since t=60s, which would read as stuck at 7h. + - series: 'test_operator_controller_condition{pod="operator-old",controller="cluster",kind="Cluster",name="prod",exported_namespace="clusters",condition="Ready",status="False",reason="Provisioning",id="clusters/prod"}' + values: '60+0x420' + # Current pod: Cluster Ready=True since t=120s. + - series: 'test_operator_controller_condition{pod="operator-new",controller="cluster",kind="Cluster",name="prod",exported_namespace="clusters",condition="Ready",status="True",reason="Ready",id="clusters/prod"}' + values: '120+0x420' + # Both pods report Database not ready; the current pod transitioned later. + - series: 'test_operator_controller_condition{pod="operator-old",controller="database",kind="Database",name="main",exported_namespace="db",condition="Ready",status="False",reason="Error",id="db/main"}' + values: '1760534009+0x60' + - series: 'test_operator_controller_condition{pod="operator-new",controller="database",kind="Database",name="main",exported_namespace="db",condition="Ready",status="False",reason="Error",id="db/main"}' + values: '1760534348+0x60' + + alert_rule_test: + - eval_time: 40m + alertname: CustomResourceNotReady + exp_alerts: + - exp_labels: + controller: database + kind: Database + name: main + exported_namespace: db + severity: warning + exp_annotations: + summary: >- + Database CR `db/main` has not been ready for 30 minutes + description: | + Custom Resource of kind `Database` named `main` in namespace + `db` has not been ready for more than 30 minutes. + + The Ready condition last transitioned at 2025-10-15 13:19:08 +0000 UTC. + + Quick check with: + ``` + kubectl describe Database/main -n db + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=&var-kind=Database&var-condition=Ready&var-status=False&var-resource_id=db%2Fmain + + - eval_time: 40m + alertname: CustomResourceConditionUnknown + exp_alerts: [] + + - eval_time: 7h + alertname: CustomResourceConditionStuck + exp_alerts: [] + + # Two installs of the operator export the same metric namespace and, by + # misconfiguration or a split watch, the same owner identity. The freshest + # series join and the aggregation both keep job, so install alpha's older + # False series is not deduped away by install beta's newer True series, and + # alpha fires with its job label. + - name: TestCrossJobSeriesStayApart + interval: 1m + input_series: + - series: 'test_operator_controller_condition{job="alpha",controller="backup",kind="Backup",name="shared",exported_namespace="backups",condition="Ready",status="False",reason="Error",id="backups/shared"}' + values: '1760534009+0x60' + - series: 'test_operator_controller_condition{job="beta",controller="backup",kind="Backup",name="shared",exported_namespace="backups",condition="Ready",status="True",reason="Ready",id="backups/shared"}' + values: '1760534420+0x60' + alert_rule_test: + - eval_time: 40m + alertname: CustomResourceNotReady + exp_alerts: + - exp_labels: + job: alpha + controller: backup + kind: Backup + name: shared + exported_namespace: backups + severity: warning + exp_annotations: + summary: >- + Backup CR `backups/shared` has not been ready for 30 minutes + description: | + Custom Resource of kind `Backup` named `shared` in namespace + `backups` has not been ready for more than 30 minutes. + + The Ready condition last transitioned at 2025-10-15 13:13:29 +0000 UTC. + + Quick check with: + ``` + kubectl describe Backup/shared -n backups + ``` + dashboard_url: >- + /d/test_operator_crd_conditions_browser/crd-conditions-browser?var-job=alpha&var-kind=Backup&var-condition=Ready&var-status=False&var-resource_id=backups%2Fshared diff --git a/observability/alerts/tests/managed_resources_test.yaml b/observability/alerts/tests/managed_resources_test.yaml new file mode 100644 index 00000000..b2f35485 --- /dev/null +++ b/observability/alerts/tests/managed_resources_test.yaml @@ -0,0 +1,233 @@ +# Unit tests for the managed resource alerting rules. +# +# Run with `make test-alerts`. The counters carry no owner identity, so every +# series below is one resource *type* of one component, aggregated over all of +# its owners. Time series values are cumulative counter samples one minute apart. +rule_files: + - ../managed_resources.yaml + +evaluation_interval: 1m + +tests: + # A hot loop: every apply of the configmap rewrites it, 4 updates a second. + # The other resources of the component converge with `none`, which must not + # mask the configmap's ratio. + - name: TestManagedResourceNotConvergingHotLoop + interval: 1m + input_series: + - series: 'ocf_resource_apply_total{controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="updated"}' + values: '0+240x60' + - series: 'ocf_resource_apply_total{controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="none"}' + values: '0+1x60' + - series: 'ocf_resource_apply_total{controller="webapp",owner_kind="WebApp",component="server",resource="deployment",kind="Deployment",operation="none"}' + values: '0+240x60' + alert_rule_test: + - eval_time: 10m + alertname: ManagedResourceNotConverging + exp_alerts: [] + # ratio 240/241 over the window, floor satisfied, for: 15m elapsed + - eval_time: 31m + alertname: ManagedResourceNotConverging + exp_alerts: + - exp_labels: + controller: webapp + owner_kind: WebApp + component: server + resource: configmap + kind: ConfigMap + severity: warning + exp_annotations: + summary: >- + ConfigMap `configmap` of component `server` (WebApp) is rewritten on most reconciles + description: | + Over the last 15 minutes, 99.59% of the applies of the `ConfigMap` resource `configmap` in component `server` of owner kind `WebApp` (controller `webapp`) updated the object. In steady state a converged resource applies as `none` on every pass, so a resource whose applies are mostly updates is being rewritten on reconciles where nothing should have changed. + + Usual causes: a mutation that is not idempotent (a timestamp, a random value, an unsorted list), a defaulting webhook or another controller fighting the operator, or a field the API server normalises. Events report the same thing, but client-go's spam filter truncates them within seconds under exactly these conditions. + + Quick check with: + ``` + kubectl get events -A --field-selector reason=UpdatedConfigMap + ``` + + # Legitimate churn at scale: many owners change spec, every `updated` apply is + # followed by a `none` apply on the reconcile the status update triggers. The + # absolute rate is high (2 updates a minute, well over the floor) but the ratio + # sits at 0.5, so the rule must stay silent. + - name: TestManagedResourceNotConvergingChurnDoesNotFire + interval: 1m + input_series: + - series: 'ocf_resource_apply_total{controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="updated"}' + values: '0+2x60' + - series: 'ocf_resource_apply_total{controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="none"}' + values: '0+2x60' + alert_rule_test: + - eval_time: 45m + alertname: ManagedResourceNotConverging + exp_alerts: [] + + # One edit on an otherwise idle resource: ratio is 1 but only one update in + # the window, under the floor. + - name: TestManagedResourceNotConvergingSingleEditDoesNotFire + interval: 1m + input_series: + - series: 'ocf_resource_apply_total{controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="updated"}' + values: '0 0 0 1+0x57' + alert_rule_test: + - eval_time: 30m + alertname: ManagedResourceNotConverging + exp_alerts: [] + + # A PVC whose applies fail three times out of four, on a resource that also + # has the occasional successful apply. + - name: TestManagedResourceApplyFailing + interval: 1m + input_series: + - series: 'ocf_resource_apply_errors_total{controller="database",owner_kind="Database",component="storage",resource="pvc",kind="PersistentVolumeClaim"}' + values: '0+3x60' + - series: 'ocf_resource_apply_total{controller="database",owner_kind="Database",component="storage",resource="pvc",kind="PersistentVolumeClaim",operation="none"}' + values: '0+1x60' + alert_rule_test: + - eval_time: 10m + alertname: ManagedResourceApplyFailing + exp_alerts: [] + - eval_time: 31m + alertname: ManagedResourceApplyFailing + exp_alerts: + - exp_labels: + controller: database + owner_kind: Database + component: storage + resource: pvc + kind: PersistentVolumeClaim + severity: warning + exp_annotations: + summary: >- + Applies of PersistentVolumeClaim `pvc` of component `storage` (Database) are failing + description: | + Over the last 15 minutes, 75% of the apply attempts of the `PersistentVolumeClaim` resource `pvc` in component `storage` of owner kind `Database` (controller `database`) failed. + + The counter covers every failure of an attempt: mutating the desired object, the server-side apply patch, and the classification that follows it. The owners whose resource fails report the error on their Ready condition, which is where the failure message lives; this rule says the failure is systematic for the resource type rather than one owner's bad day. + + Quick check with: + ``` + kubectl get Database -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[?(@.type=="Ready")].reason,MESSAGE:.status.conditions[?(@.type=="Ready")].message' + ``` + + # A resource that has only ever failed: no success series exists at all. The + # ratio must still evaluate (to 1) rather than vanish on a missing vector. + - name: TestManagedResourceApplyFailingWithoutSuccesses + interval: 1m + input_series: + - series: 'ocf_resource_apply_errors_total{controller="database",owner_kind="Database",component="storage",resource="pvc",kind="PersistentVolumeClaim"}' + values: '0+2x60' + alert_rule_test: + - eval_time: 31m + alertname: ManagedResourceApplyFailing + exp_alerts: + - exp_labels: + controller: database + owner_kind: Database + component: storage + resource: pvc + kind: PersistentVolumeClaim + severity: warning + exp_annotations: + summary: >- + Applies of PersistentVolumeClaim `pvc` of component `storage` (Database) are failing + description: | + Over the last 15 minutes, 100% of the apply attempts of the `PersistentVolumeClaim` resource `pvc` in component `storage` of owner kind `Database` (controller `database`) failed. + + The counter covers every failure of an attempt: mutating the desired object, the server-side apply patch, and the classification that follows it. The owners whose resource fails report the error on their Ready condition, which is where the failure message lives; this rule says the failure is systematic for the resource type rather than one owner's bad day. + + Quick check with: + ``` + kubectl get Database -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[?(@.type=="Ready")].reason,MESSAGE:.status.conditions[?(@.type=="Ready")].message' + ``` + + # Sporadic conflicts among many successful applies must not fire. + - name: TestManagedResourceApplyFailingSporadicDoesNotFire + interval: 1m + input_series: + - series: 'ocf_resource_apply_errors_total{controller="database",owner_kind="Database",component="storage",resource="pvc",kind="PersistentVolumeClaim"}' + values: '0+1x60' + - series: 'ocf_resource_apply_total{controller="database",owner_kind="Database",component="storage",resource="pvc",kind="PersistentVolumeClaim",operation="none"}' + values: '0+9x60' + alert_rule_test: + - eval_time: 45m + alertname: ManagedResourceApplyFailing + exp_alerts: [] + + # Two operators scraped as different jobs in one namespace, with a controller + # and resource topology that happens to coincide. Aggregated across jobs the + # healthy beta (every apply `none`) would dilute alpha's hot loop to a ratio + # of 240/(240+1+2400) and mask it; keyed on `job` alpha fires on its own. + - name: TestManagedResourceNotConvergingKeyedOnJob + interval: 1m + input_series: + - series: 'ocf_resource_apply_total{namespace="operators",job="alpha",controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="updated"}' + values: '0+240x60' + - series: 'ocf_resource_apply_total{namespace="operators",job="alpha",controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="none"}' + values: '0+1x60' + - series: 'ocf_resource_apply_total{namespace="operators",job="beta",controller="webapp",owner_kind="WebApp",component="server",resource="configmap",kind="ConfigMap",operation="none"}' + values: '0+2400x60' + alert_rule_test: + - eval_time: 31m + alertname: ManagedResourceNotConverging + exp_alerts: + - exp_labels: + namespace: operators + job: alpha + controller: webapp + owner_kind: WebApp + component: server + resource: configmap + kind: ConfigMap + severity: warning + exp_annotations: + summary: >- + ConfigMap `configmap` of component `server` (WebApp) is rewritten on most reconciles + description: | + Over the last 15 minutes, 99.59% of the applies of the `ConfigMap` resource `configmap` in component `server` of owner kind `WebApp` (controller `webapp`) updated the object. In steady state a converged resource applies as `none` on every pass, so a resource whose applies are mostly updates is being rewritten on reconciles where nothing should have changed. + + Usual causes: a mutation that is not idempotent (a timestamp, a random value, an unsorted list), a defaulting webhook or another controller fighting the operator, or a field the API server normalises. Events report the same thing, but client-go's spam filter truncates them within seconds under exactly these conditions. + + Quick check with: + ``` + kubectl get events -A --field-selector reason=UpdatedConfigMap + ``` + + # Same shape for the failure ratio: alpha's applies all fail, beta's all + # succeed. Merged across jobs the ratio would be 6/(6+6+600); keyed on `job` + # alpha's ratio is 1. + - name: TestManagedResourceApplyFailingKeyedOnJob + interval: 1m + input_series: + - series: 'ocf_resource_apply_errors_total{namespace="operators",job="alpha",controller="database",owner_kind="Database",component="storage",resource="pvc",kind="PersistentVolumeClaim"}' + values: '0+6x60' + - series: 'ocf_resource_apply_total{namespace="operators",job="beta",controller="database",owner_kind="Database",component="storage",resource="pvc",kind="PersistentVolumeClaim",operation="none"}' + values: '0+600x60' + alert_rule_test: + - eval_time: 31m + alertname: ManagedResourceApplyFailing + exp_alerts: + - exp_labels: + namespace: operators + job: alpha + controller: database + owner_kind: Database + component: storage + resource: pvc + kind: PersistentVolumeClaim + severity: warning + exp_annotations: + summary: >- + Applies of PersistentVolumeClaim `pvc` of component `storage` (Database) are failing + description: | + Over the last 15 minutes, 100% of the apply attempts of the `PersistentVolumeClaim` resource `pvc` in component `storage` of owner kind `Database` (controller `database`) failed. + + The counter covers every failure of an attempt: mutating the desired object, the server-side apply patch, and the classification that follows it. The owners whose resource fails report the error on their Ready condition, which is where the failure message lives; this rule says the failure is systematic for the resource type rather than one owner's bad day. + + Quick check with: + ``` + kubectl get Database -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[?(@.type=="Ready")].reason,MESSAGE:.status.conditions[?(@.type=="Ready")].message' + ``` diff --git a/observability/dashboards/crd_conditions_browser.tpl.json b/observability/dashboards/crd_conditions_browser.tpl.json new file mode 100644 index 00000000..ae110408 --- /dev/null +++ b/observability/dashboards/crd_conditions_browser.tpl.json @@ -0,0 +1,796 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "description": "Dashboard to monitor operator custom resource conditions", + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "links": [ + { + "asDropdown": false, + "icon": "external link", + "includeVars": false, + "keepTime": true, + "tags": [ + "ocf" + ], + "targetBlank": false, + "title": "OCF dashboards", + "tooltip": "", + "type": "dashboards", + "url": "" + } + ], + "liveNow": true, + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 10, + "panels": [], + "title": "Operator Conditions", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Conditions matching the given filter criteria ", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "semi-dark-blue", + "mode": "fixed" + }, + "mappings": [], + "noValue": "0", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 5, + "x": 0, + "y": 1 + }, + "id": 6, + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "12.1.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=~\"$status\", reason=~\"$reason\", id=~\"$resource_id\", {{namespace_label}}=~\"$namespace\"} > 0\n and\n topk by (id, condition) (1, {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", {{namespace_label}}=~\"$namespace\"})\n)", + "legendFormat": "Total", + "range": true, + "refId": "A" + } + ], + "title": "Total Conditions Matching", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Table of $kind conditions of type $condition", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "transparent", + "mode": "fixed" + }, + "custom": { + "align": "auto", + "cellOptions": { + "applyToRow": false, + "mode": "basic", + "type": "color-background", + "wrapText": true + }, + "filterable": true, + "inspect": true + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Namespace" + }, + "properties": [ + { + "id": "custom.width", + "value": 397 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Since" + }, + "properties": [ + { + "id": "unit", + "value": "dateTimeFromNow" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Status" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "options": { + "False": { + "color": "red", + "index": 1, + "text": "False" + }, + "True": { + "color": "semi-dark-green", + "index": 0, + "text": "True" + }, + "Unknown": { + "color": "semi-dark-yellow", + "index": 2, + "text": "Unknown" + } + }, + "type": "value" + } + ] + } + ] + } + ] + }, + "gridPos": { + "h": 13, + "w": 24, + "x": 0, + "y": 5 + }, + "id": 5, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [] + }, + "pluginVersion": "12.1.1", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum(\n {{operator_namespace}}controller_condition{job=~\"$job\", \n kind=\"$kind\",\n condition=~\"$condition\",\n status=~\"$status\",\n reason=~\"$reason\",\n id=~\"$resource_id\",\n {{namespace_label}}=~\"$namespace\"\n }\n and\n topk by (id, condition) (1, {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", {{namespace_label}}=~\"$namespace\"})\n) by (name, {{namespace_label}}, condition, status, reason)", + "format": "table", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "$kind Conditions", + "transformations": [ + { + "id": "calculateField", + "options": { + "alias": "value_ms", + "binary": { + "left": { + "matcher": { + "id": "byName", + "options": "Value" + } + }, + "operator": "*", + "right": { + "fixed": "1000" + } + }, + "mode": "binary", + "reduce": { + "reducer": "sum" + } + } + }, + { + "id": "convertFieldType", + "options": { + "conversions": [ + { + "destinationType": "time", + "targetField": "value_ms" + } + ], + "fields": {} + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "Value": true + }, + "includeByName": {}, + "indexByName": { + "Time": 0, + "{{namespace_label}}": 1, + "name": 2, + "condition": 3, + "status": 4, + "reason": 5, + "value_ms": 6, + "Value": 7 + }, + "renameByName": { + "Value": "", + "condition": "Condition", + "{{namespace_label}}": "Namespace", + "name": "Name", + "reason": "Reason", + "status": "Status", + "value_ms": "Since" + } + } + } + ], + "type": "table" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 18 + }, + "id": 9, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Number of $kind $condition Conditions with status False", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "semi-dark-red", + "mode": "thresholds" + }, + "mappings": [], + "noValue": "0", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "semi-dark-red", + "value": 1 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 24, + "x": 0, + "y": 19 + }, + "id": 7, + "options": { + "colorMode": "background_solid", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "12.1.1", + "repeat": "condition", + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=\"False\", reason=~\"$reason\", id=~\"$resource_id\", {{namespace_label}}=~\"$namespace\"} > 0\n and\n topk by (id, condition) (1, {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", {{namespace_label}}=~\"$namespace\"})\n)", + "legendFormat": "Total", + "range": false, + "refId": "A", + "exemplar": false, + "instant": true + } + ], + "title": "$condition=False", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Number of $kind $condition Conditions with status Unknown", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "semi-dark-red", + "mode": "thresholds" + }, + "mappings": [], + "noValue": "0", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "semi-dark-yellow", + "value": 1 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 24, + "x": 0, + "y": 24 + }, + "id": 11, + "options": { + "colorMode": "background_solid", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "12.1.1", + "repeat": "condition", + "repeatDirection": "h", + "targets": [ + { + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=\"Unknown\", reason=~\"$reason\", id=~\"$resource_id\", {{namespace_label}}=~\"$namespace\"} > 0\n and\n topk by (id, condition) (1, {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", {{namespace_label}}=~\"$namespace\"})\n)", + "legendFormat": "Total", + "range": false, + "refId": "A", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "exemplar": false, + "instant": true + } + ], + "title": "$condition=Unknown", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Number of $kind $condition Conditions with status True", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "semi-dark-red", + "mode": "thresholds" + }, + "mappings": [], + "noValue": "0", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 24, + "x": 0, + "y": 29 + }, + "id": 8, + "options": { + "colorMode": "background_solid", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "12.1.1", + "repeat": "condition", + "repeatDirection": "h", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=\"True\", reason=~\"$reason\", id=~\"$resource_id\", {{namespace_label}}=~\"$namespace\"} > 0\n and\n topk by (id, condition) (1, {{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", {{namespace_label}}=~\"$namespace\"})\n)", + "legendFormat": "Total", + "range": false, + "refId": "A", + "exemplar": false, + "instant": true + } + ], + "title": "$condition=True", + "type": "stat" + } + ], + "title": "Status Counts Per Condition", + "type": "row" + } + ], + "preload": false, + "refresh": "", + "schemaVersion": 41, + "tags": [ + "ocf" + ], + "templating": { + "list": [ + { + "current": { + "text": "", + "value": "" + }, + "description": "The data source to get the metrics from", + "label": "Datasource", + "name": "datasource", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "type": "datasource" + }, + { + "current": { + "selected": true, + "text": "All", + "value": "$__all" + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values({{operator_namespace}}controller_condition,job)", + "description": "The Prometheus scrape job of the operator install, usually the name of its metrics Service or ServiceMonitor. Narrows every panel to one install when two installs export the same metric namespace; the operator dashboard and the condition alerts link here with it set.", + "includeAll": true, + "allValue": ".*", + "label": "Operator (scrape job)", + "multi": false, + "name": "job", + "options": [], + "query": { + "query": "label_values({{operator_namespace}}controller_condition,job)", + "refId": "job" + }, + "refresh": 2, + "regex": "", + "sort": 1, + "type": "query" + }, + { + "current": { + "text": "", + "value": "" + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values({{operator_namespace}}controller_condition{job=~\"$job\"},kind)", + "description": "The resource kind to view", + "label": "Kind", + "name": "kind", + "options": [], + "query": { + "qryType": 1, + "query": "label_values({{operator_namespace}}controller_condition{job=~\"$job\"},kind)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 2, + "regex": "", + "type": "query" + }, + { + "current": { + "text": "All", + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\"},condition)", + "description": "The condition type to display", + "includeAll": true, + "label": "Condition", + "multi": true, + "name": "condition", + "options": [], + "query": { + "qryType": 1, + "query": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\"},condition)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 2, + "regex": "", + "type": "query", + "allValue": ".*" + }, + { + "allowCustomValue": true, + "current": { + "text": "All", + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\"},status)", + "description": "Filter for a specific status", + "includeAll": true, + "label": "Status", + "multi": true, + "name": "status", + "options": [], + "query": { + "qryType": 1, + "query": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\"},status)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query", + "allValue": ".*" + }, + { + "current": { + "text": "All", + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=~\"$status\"},reason)", + "description": "Filter for condition reason", + "includeAll": true, + "label": "Reason", + "multi": true, + "name": "reason", + "options": [], + "query": { + "qryType": 1, + "query": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=~\"$status\"},reason)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query", + "allValue": ".*" + }, + { + "current": { + "text": "All", + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "query_result(sum ({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\"} > 0) by ({{namespace_label}}))", + "description": "The namespaces in which to list the resources", + "includeAll": true, + "label": "Namespace", + "multi": true, + "name": "namespace", + "options": [], + "query": { + "qryType": 3, + "query": "query_result(sum ({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\"} > 0) by ({{namespace_label}}))", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 2, + "regex": "^.*{{namespace_label}}=\"([^\"]+)\".*$", + "sort": 1, + "type": "query", + "allValue": ".*" + }, + { + "allowCustomValue": false, + "current": { + "text": "All", + "value": "$__all" + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=~\"$status\", reason=~\"$reason\", {{namespace_label}}=~\"$namespace\"},id)", + "description": "Namespace and resource name combination", + "hide": 0, + "includeAll": true, + "multi": true, + "name": "resource_id", + "options": [], + "query": { + "qryType": 1, + "query": "label_values({{operator_namespace}}controller_condition{job=~\"$job\", kind=\"$kind\", condition=~\"$condition\", status=~\"$status\", reason=~\"$reason\", {{namespace_label}}=~\"$namespace\"},id)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query", + "allValue": ".*", + "label": "Resource" + }, + { + "baseFilters": [], + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "filters": [], + "name": "Filters", + "type": "adhoc" + } + ] + }, + "time": { + "from": "now-5m", + "to": "now" + }, + "timepicker": { + "hidden": true + }, + "timezone": "browser", + "title": "CRD Conditions Browser", + "uid": "{{operator_namespace}}crd_conditions_browser", + "version": 1 +} diff --git a/observability/dashboards/ocf_operator.tpl.json b/observability/dashboards/ocf_operator.tpl.json new file mode 100644 index 00000000..1c0d0b0b --- /dev/null +++ b/observability/dashboards/ocf_operator.tpl.json @@ -0,0 +1,3605 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "description": "Health of an operator built on the operator-component-framework: reconciliation, workqueue, managed resource applies, custom resource (CR) conditions, API client and process.", + "editable": true, + "graphTooltip": 1, + "links": [ + { + "asDropdown": false, + "icon": "external link", + "includeVars": false, + "keepTime": true, + "tags": [ + "ocf" + ], + "targetBlank": false, + "title": "OCF dashboards", + "tooltip": "", + "type": "dashboards", + "url": "" + } + ], + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 2, + "panels": [], + "title": "Overview", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 0, + "y": 1 + }, + "id": 3, + "options": { + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(rate(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "range": true, + "instant": false + } + ], + "title": "Reconciles/s", + "type": "stat", + "description": "Reconciles per second across the selected controllers, with the trend over the dashboard range." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 0.05 + }, + { + "color": "red", + "value": 0.25 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 6, + "y": 1 + }, + "id": 4, + "options": { + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(rate(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\", result=\"error\"}[$__rate_interval]))\n/\nsum(rate(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "range": true, + "instant": false + } + ], + "title": "Reconcile error ratio", + "type": "stat", + "description": "Share of reconciles returning an error. Orange from 5%, red from 25%, the ControllerReconcileErrors threshold." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 10 + }, + { + "color": "red", + "value": 60 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 12, + "y": 1 + }, + "id": 5, + "options": { + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (le) (rate(controller_runtime_reconcile_time_seconds_bucket{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])))", + "refId": "A", + "range": true, + "instant": false + } + ], + "title": "p99 reconcile time", + "type": "stat", + "description": "p99 reconcile time. Orange from 10s, red from 60s." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 30 + }, + { + "color": "red", + "value": 300 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 18, + "y": 1 + }, + "id": 6, + "options": { + "colorMode": "background", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (le) (rate(workqueue_queue_duration_seconds_bucket{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])))", + "refId": "A", + "range": true, + "instant": false + } + ], + "title": "p99 queue wait", + "type": "stat", + "description": "p99 time an item waited in the workqueue before a worker picked it up. Orange from 30s, red from 5m." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short", + "noValue": "0" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Not Ready > 2m" + }, + "properties": [ + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unknown" + }, + "properties": [ + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 4, + "w": 12, + "x": 0, + "y": 6 + }, + "id": 9, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\", status=\"True\"}\n and\n topk by (kind, id) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\"})\n)", + "refId": "A", + "legendFormat": "Ready", + "range": true, + "instant": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n ({{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\", status=\"False\"} < (time() - 120))\n and\n topk by (kind, id) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\"})\n) or vector(0)", + "refId": "B", + "legendFormat": "Not Ready > 2m", + "range": true, + "instant": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\", status=\"Unknown\"}\n and\n topk by (kind, id) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\"})\n) or vector(0)", + "refId": "C", + "legendFormat": "Unknown", + "range": true, + "instant": false + } + ], + "title": "CRs by Ready status", + "type": "stat", + "description": "CRs counted once each on their freshest Ready series. Not Ready counts only CRs whose Ready condition has been False for more than two minutes, debounced on lastTransitionTime, so a rollout in progress does not count." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "options": { + "0": { + "color": "red", + "index": 1, + "text": "Standby" + }, + "1": { + "color": "green", + "index": 0, + "text": "Leader" + } + }, + "type": "value" + }, + { + "options": { + "match": "null", + "result": { + "color": "text", + "index": 2, + "text": "n/a" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 12, + "y": 6 + }, + "id": 8, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max(leader_election_master_status{namespace=~\"$namespace\", job=\"$job\"})", + "refId": "A", + "range": false, + "exemplar": false, + "instant": true + } + ], + "title": "Leader", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 18, + "y": 6 + }, + "id": 7, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "same_as_value", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(controller_runtime_active_workers{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"})", + "refId": "A", + "legendFormat": "active", + "range": false, + "exemplar": false, + "instant": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(controller_runtime_max_concurrent_reconciles{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"})", + "refId": "B", + "legendFormat": "max", + "range": false, + "exemplar": false, + "instant": true + } + ], + "title": "Active / max workers", + "type": "stat" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 10 + }, + "id": 31, + "panels": [], + "title": "Conditions", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short", + "min": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Ready" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Not Ready" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unknown" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 11 + }, + "id": 41, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\", status=\"True\"}\n and\n topk by (kind, id) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\"})\n) or vector(0)", + "refId": "A", + "legendFormat": "Ready", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\", status=\"False\"}\n and\n topk by (kind, id) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\"})\n) or vector(0)", + "refId": "B", + "legendFormat": "Not Ready", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(\n {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\", status=\"Unknown\"}\n and\n topk by (kind, id) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\"})\n) or vector(0)", + "refId": "C", + "legendFormat": "Unknown", + "range": true + } + ], + "title": "CRs by Ready status over time", + "type": "timeseries", + "description": "CRs counted once each on their freshest Ready series, stacked by status. Not Ready here is every CR whose Ready condition is False, without the two minute debounce the Overview tile applies, so a rollout shows as a short red band." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "CR counts per kind, condition and status for every condition that is not True, largest first. Empty when every condition of every CR is True. Unlike the Ready panels this covers all condition types, so a negative polarity condition such as Degraded=False does not appear while Degraded=True does.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + } + ] + }, + "unit": "short", + "noValue": "No unhealthy conditions" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/ False$/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/ Unknown$/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 11 + }, + "id": 32, + "options": { + "displayMode": "gradient", + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "maxVizHeight": 32, + "minVizHeight": 24, + "minVizWidth": 8, + "namePlacement": "top", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showUnfilled": true, + "sizing": "manual", + "valueMode": "color", + "text": { + "titleSize": 13, + "valueSize": 18 + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sort_desc(\n count by (kind, condition, status) (\n {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", status!=\"True\"}\n and\n topk by (kind, id, condition) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\"})\n )\n)", + "refId": "A", + "legendFormat": "{{kind}} {{condition}} {{status}}", + "exemplar": false, + "instant": true, + "range": false + } + ], + "title": "Unhealthy conditions", + "type": "bargauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "CRs whose Ready condition is not True. The Name links into the conditions browser filtered to that CR.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "filterable": true, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Status" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "options": { + "False": { + "color": "red", + "index": 1, + "text": "False" + }, + "True": { + "color": "semi-dark-green", + "index": 0, + "text": "True" + }, + "Unknown": { + "color": "semi-dark-yellow", + "index": 2, + "text": "Unknown" + } + }, + "type": "value" + } + ] + }, + { + "id": "custom.cellOptions", + "value": { + "type": "color-background" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Since" + }, + "properties": [ + { + "id": "unit", + "value": "dateTimeFromNow" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Name" + }, + "properties": [ + { + "id": "links", + "value": [ + { + "title": "Open in the conditions browser", + "url": "/d/{{operator_namespace}}crd_conditions_browser/crd-conditions-browser?var-job=$job&var-kind=${__data.fields.Kind}&var-condition=Ready&var-resource_id=${__data.fields.id:percentencode}&${__url_time_range}" + } + ] + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "id" + }, + "properties": [ + { + "id": "custom.hidden", + "value": true + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 20 + }, + "id": 33, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [] + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (kind, {{namespace_label}}, name, reason, status, id) (\n {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\", status!=\"True\"}\n and\n topk by (kind, id) (1, {{operator_namespace}}controller_condition{job=\"$job\", controller=~\"$controller\", condition=\"Ready\"})\n)", + "refId": "A", + "exemplar": false, + "instant": true, + "range": false, + "format": "table" + } + ], + "title": "CRs not Ready", + "transformations": [ + { + "id": "calculateField", + "options": { + "alias": "since_ms", + "binary": { + "left": { + "matcher": { + "id": "byName", + "options": "Value" + } + }, + "operator": "*", + "right": { + "fixed": "1000" + } + }, + "mode": "binary", + "reduce": { + "reducer": "sum" + } + } + }, + { + "id": "convertFieldType", + "options": { + "conversions": [ + { + "destinationType": "time", + "targetField": "since_ms" + } + ], + "fields": {} + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "Value": true + }, + "includeByName": {}, + "indexByName": { + "Time": 0, + "kind": 1, + "{{namespace_label}}": 2, + "name": 3, + "status": 4, + "reason": 5, + "since_ms": 6, + "id": 7, + "Value": 8 + }, + "renameByName": { + "kind": "Kind", + "name": "Name", + "reason": "Reason", + "since_ms": "Since", + "status": "Status", + "{{namespace_label}}": "Namespace" + } + } + } + ], + "type": "table" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 30 + }, + "id": 26, + "panels": [], + "title": "Managed resources", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Applies per second that found a diff and updated each managed resource, over time. A resource that stays high is rewritten on every reconcile; the Not-converging ratio next to it puts that against the resource's own apply rate. Sort the legend by Last to rank resources as the previous table did. Dashed lines mark the 0.05 and 0.5 updates/s bands.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "orange", + "value": 0.05 + }, + { + "color": "red", + "value": 0.5 + } + ] + }, + "unit": "reqps", + "min": 0 + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 31 + }, + "id": 28, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller, owner_kind, component, resource, kind) (rate(ocf_resource_apply_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\", operation=\"updated\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{controller}}/{{component}}/{{resource}} ({{kind}})", + "range": true + } + ], + "title": "Updated rate per resource", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 30, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "created" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "updated" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "none" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "error" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 41 + }, + "id": 27, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (operation) (rate(ocf_resource_apply_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{operation}}", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(rate(ocf_resource_apply_errors_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "B", + "legendFormat": "error", + "range": true + } + ], + "title": "Apply rate by operation", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 0.5 + } + ] + }, + "unit": "percentunit", + "min": 0, + "max": 1 + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 41 + }, + "id": 29, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller, owner_kind, component, resource, kind) (rate(ocf_resource_apply_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\", operation=\"updated\"}[15m]))\n/\nsum by (controller, owner_kind, component, resource, kind) (rate(ocf_resource_apply_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[15m]))", + "refId": "A", + "legendFormat": "{{controller}}/{{component}}/{{resource}} ({{kind}})", + "range": true + } + ], + "title": "Not-converging ratio", + "type": "timeseries", + "description": "Share of applies over 15m that found a diff and updated the resource. The ManagedResourceNotConverging alert uses the same expression with a request-rate floor." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit", + "min": 0, + "max": 1 + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 41 + }, + "id": 30, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller, owner_kind, component, resource, kind) (rate(ocf_resource_apply_errors_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))\n/\n(\n sum by (controller, owner_kind, component, resource, kind) (rate(ocf_resource_apply_errors_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))\n +\n (sum by (controller, owner_kind, component, resource, kind) (rate(ocf_resource_apply_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])) or sum by (controller, owner_kind, component, resource, kind) (rate(ocf_resource_apply_errors_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])) * 0)\n)", + "refId": "A", + "legendFormat": "{{controller}}/{{component}}/{{resource}} ({{kind}})", + "range": true + } + ], + "title": "Apply error ratio per resource", + "type": "timeseries", + "description": "Failed applies as a share of all apply attempts per resource. A resource whose every apply fails records no ocf_resource_apply_total sample, so the denominator falls back to the error rate alone and the ratio reads 1." + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 50 + }, + "id": 12, + "panels": [], + "title": "Reconciliation", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 30, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 11, + "w": 24, + "x": 0, + "y": 51 + }, + "id": 13, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller, result) (rate(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{controller}} {{result}}", + "range": true + } + ], + "title": "Reconcile rate by result", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 0.25 + } + ] + }, + "unit": "percentunit", + "min": 0, + "max": 1 + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 62 + }, + "id": 14, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (rate(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\", result=\"error\"}[$__rate_interval]))\n/\nsum by (controller) (rate(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Reconcile error ratio", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 62 + }, + "id": 15, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (controller, le) (rate(controller_runtime_reconcile_time_seconds_bucket{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])))", + "refId": "A", + "legendFormat": "p50 {{controller}}", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.90, sum by (controller, le) (rate(controller_runtime_reconcile_time_seconds_bucket{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])))", + "refId": "B", + "legendFormat": "p90 {{controller}}", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (controller, le) (rate(controller_runtime_reconcile_time_seconds_bucket{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])))", + "refId": "C", + "legendFormat": "p99 {{controller}}", + "range": true + } + ], + "title": "Reconcile latency", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "bars", + "fillOpacity": 60, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 62 + }, + "id": 16, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (increase(controller_runtime_reconcile_panics_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Reconcile panics", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 71 + }, + "id": 17, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (controller) (workqueue_longest_running_processor_seconds{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"})", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Max in-progress reconcile age", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/^max .*/" + }, + "properties": [ + { + "id": "custom.lineStyle", + "value": { + "dash": [ + 10, + 10 + ], + "fill": "dash" + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 71 + }, + "id": 18, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (controller_runtime_active_workers{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"})", + "refId": "A", + "legendFormat": "active {{controller}}", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (controller_runtime_max_concurrent_reconciles{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"})", + "refId": "B", + "legendFormat": "max {{controller}}", + "range": true + } + ], + "title": "Workers", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 80 + }, + "id": 19, + "panels": [], + "title": "Workqueue", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 81 + }, + "id": 20, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (workqueue_depth{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"})", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Depth", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 81 + }, + "id": 21, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (rate(workqueue_adds_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Adds/s", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "dashed" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 300 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 81 + }, + "id": 22, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (controller, le) (rate(workqueue_queue_duration_seconds_bucket{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Queue wait p99", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 90 + }, + "id": 23, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (controller, le) (rate(workqueue_work_duration_seconds_bucket{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval])))", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Work duration p99", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 90 + }, + "id": 24, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (rate(workqueue_retries_total{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Retries/s", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 90 + }, + "id": 25, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (controller) (workqueue_unfinished_work_seconds{namespace=~\"$namespace\", job=\"$job\", controller=~\"$controller\"})", + "refId": "A", + "legendFormat": "{{controller}}", + "range": true + } + ], + "title": "Unfinished work", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 99 + }, + "id": 34, + "panels": [], + "title": "API client", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 100 + }, + "id": 35, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (method) (rate(rest_client_requests_total{namespace=~\"$namespace\", job=\"$job\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{method}}", + "range": true + } + ], + "title": "Requests/s by method", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 100 + }, + "id": 36, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (code) (rate(rest_client_requests_total{namespace=~\"$namespace\", job=\"$job\", code!~\"2..\"}[$__rate_interval]))", + "refId": "A", + "legendFormat": "{{code}}", + "range": true + } + ], + "title": "Non-2xx responses/s by code", + "type": "timeseries" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 109 + }, + "id": 40, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/ request$/" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.lineStyle", + "value": { + "fill": "dash", + "dash": [ + 10, + 10 + ] + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/ limit$/" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + }, + { + "id": "custom.lineStyle", + "value": { + "fill": "dash", + "dash": [ + 10, + 10 + ] + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 110 + }, + "id": 37, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "rate(process_cpu_seconds_total{namespace=~\"$namespace\", job=\"$job\"}[$__rate_interval])", + "refId": "A", + "legendFormat": "{{instance}}", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (pod) (kube_pod_container_resource_requests{resource=\"cpu\"} and on (namespace, pod) up{namespace=~\"$namespace\", job=\"$job\"})", + "refId": "B", + "legendFormat": "{{pod}} request", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (pod) (kube_pod_container_resource_limits{resource=\"cpu\"} and on (namespace, pod) up{namespace=~\"$namespace\", job=\"$job\"})", + "refId": "C", + "legendFormat": "{{pod}} limit", + "range": true + } + ], + "title": "CPU", + "type": "timeseries", + "description": "CPU usage of the operator process; 1 is one full core. Dashed lines are the container request and limit from kube-state-metrics (kube_pod_container_resource_requests and _limits), joined to the operator pod through the scrape target labels; they are absent when kube-state-metrics is not scraped or the target carries no namespace and pod labels. A pod with several containers shows its largest request and limit." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/ request$/" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + }, + { + "id": "custom.lineStyle", + "value": { + "fill": "dash", + "dash": [ + 10, + 10 + ] + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/ limit$/" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "red" + } + }, + { + "id": "custom.lineStyle", + "value": { + "fill": "dash", + "dash": [ + 10, + 10 + ] + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 110 + }, + "id": 38, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "process_resident_memory_bytes{namespace=~\"$namespace\", job=\"$job\"}", + "refId": "A", + "legendFormat": "{{instance}}", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (pod) (kube_pod_container_resource_requests{resource=\"memory\"} and on (namespace, pod) up{namespace=~\"$namespace\", job=\"$job\"})", + "refId": "B", + "legendFormat": "{{pod}} request", + "range": true + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (pod) (kube_pod_container_resource_limits{resource=\"memory\"} and on (namespace, pod) up{namespace=~\"$namespace\", job=\"$job\"})", + "refId": "C", + "legendFormat": "{{pod}} limit", + "range": true + } + ], + "title": "Memory RSS", + "type": "timeseries", + "description": "Resident memory of the operator process. Dashed lines are the container request and limit from kube-state-metrics (kube_pod_container_resource_requests and _limits), joined to the operator pod through the scrape target labels; they are absent when kube-state-metrics is not scraped or the target carries no namespace and pod labels. A pod with several containers shows its largest request and limit." + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 110 + }, + "id": 39, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "go_goroutines{namespace=~\"$namespace\", job=\"$job\"}", + "refId": "A", + "legendFormat": "{{instance}}", + "range": true + } + ], + "title": "Goroutines", + "type": "timeseries" + } + ], + "title": "Process", + "type": "row" + } + ], + "refresh": "", + "schemaVersion": 41, + "tags": [ + "ocf" + ], + "templating": { + "list": [ + { + "current": {}, + "hide": 0, + "label": "Datasource", + "name": "datasource", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "type": "datasource" + }, + { + "current": { + "selected": true, + "text": "All", + "value": "$__all" + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(controller_runtime_reconcile_total,namespace)", + "description": "Kubernetes namespace the operator runs in, as stamped on its series by the scrape. All also covers series without a namespace label, such as an operator scraped outside a cluster.", + "includeAll": true, + "allValue": ".*", + "label": "Operator namespace", + "multi": false, + "name": "namespace", + "options": [], + "query": { + "query": "label_values(controller_runtime_reconcile_total,namespace)", + "refId": "namespace" + }, + "refresh": 2, + "regex": "", + "sort": 1, + "type": "query" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(controller_runtime_reconcile_total{namespace=~\"$namespace\"},job)", + "includeAll": false, + "label": "Operator (scrape job)", + "multi": false, + "name": "job", + "options": [], + "query": { + "query": "label_values(controller_runtime_reconcile_total{namespace=~\"$namespace\"},job)", + "refId": "job" + }, + "refresh": 2, + "regex": "", + "sort": 1, + "type": "query", + "description": "The Prometheus scrape job of the operator, usually the name of its metrics Service or ServiceMonitor. Every controller-runtime operator exports the same metric names, so this is what selects one operator." + }, + { + "current": { + "selected": true, + "text": [ + "All" + ], + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\"},controller)", + "includeAll": true, + "label": "Controller", + "multi": true, + "name": "controller", + "options": [], + "query": { + "query": "label_values(controller_runtime_reconcile_total{namespace=~\"$namespace\", job=\"$job\"},controller)", + "refId": "controller" + }, + "refresh": 2, + "regex": "", + "sort": 1, + "type": "query" + } + ] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": { + "refresh_intervals": [ + "30s", + "1m", + "5m", + "15m", + "1h" + ] + }, + "timezone": "browser", + "title": "OCF Operator", + "uid": "{{operator_namespace}}ocf_operator", + "version": 1 +} diff --git a/observability/dev/docker-compose.yaml b/observability/dev/docker-compose.yaml new file mode 100644 index 00000000..c692d022 --- /dev/null +++ b/observability/dev/docker-compose.yaml @@ -0,0 +1,29 @@ +# Local Prometheus + Grafana for looking at the dashboards and alerts with the +# simulator's data behind them. Start with `make observability-up`. +services: + prometheus: + image: prom/prometheus:v3.14.0 + command: + - --config.file=/etc/prometheus/prometheus.yml + - --web.enable-lifecycle + ports: + - "127.0.0.1:9090:9090" + volumes: + - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro + - ../generated/dev/alerts:/etc/prometheus/rules:ro + extra_hosts: + - "host.docker.internal:host-gateway" + + grafana: + image: grafana/grafana:13.1.4 + ports: + - "127.0.0.1:3000:3000" + environment: + GF_AUTH_ANONYMOUS_ENABLED: "true" + GF_AUTH_ANONYMOUS_ORG_ROLE: Admin + GF_AUTH_DISABLE_LOGIN_FORM: "true" + volumes: + - ./grafana/provisioning:/etc/grafana/provisioning:ro + - ../generated/dev/dashboards:/var/lib/grafana/dashboards:ro + depends_on: + - prometheus diff --git a/observability/dev/grafana/provisioning/dashboards/ocf.yaml b/observability/dev/grafana/provisioning/dashboards/ocf.yaml new file mode 100644 index 00000000..0d6093b6 --- /dev/null +++ b/observability/dev/grafana/provisioning/dashboards/ocf.yaml @@ -0,0 +1,8 @@ +apiVersion: 1 +providers: + - name: ocf + folder: OCF + type: file + disableDeletion: true + options: + path: /var/lib/grafana/dashboards diff --git a/observability/dev/grafana/provisioning/datasources/prometheus.yaml b/observability/dev/grafana/provisioning/datasources/prometheus.yaml new file mode 100644 index 00000000..c988f2da --- /dev/null +++ b/observability/dev/grafana/provisioning/datasources/prometheus.yaml @@ -0,0 +1,9 @@ +apiVersion: 1 +datasources: + - name: Prometheus + type: prometheus + uid: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true + editable: false diff --git a/observability/dev/prometheus/prometheus.yml b/observability/dev/prometheus/prometheus.yml new file mode 100644 index 00000000..0c2538a0 --- /dev/null +++ b/observability/dev/prometheus/prometheus.yml @@ -0,0 +1,27 @@ +# Scrapes the simulator running on the host. The static `namespace` and `pod` +# target labels collide with the namespace label the condition gauge exports, +# which Prometheus resolves by renaming the exported one to +# `exported_namespace`, exactly as a ServiceMonitor scrape does in a cluster. +global: + scrape_interval: 5s + evaluation_interval: 10s + +rule_files: + - /etc/prometheus/rules/*.yaml + +scrape_configs: + - job_name: demo-operator + static_configs: + - targets: ["host.docker.internal:8080"] + labels: + namespace: operators + pod: demo-operator-0 + # The simulator's kube-state-metrics lookalikes (container requests and + # limits) on their own job, with honor_labels so the namespace and pod labels + # the series carry survive, as they do under kube-state-metrics' own + # ServiceMonitor. + - job_name: kube-state-metrics + honor_labels: true + metrics_path: /ksm/metrics + static_configs: + - targets: ["host.docker.internal:8080"] diff --git a/observability/dev/simulator/kubestate.go b/observability/dev/simulator/kubestate.go new file mode 100644 index 00000000..96c76509 --- /dev/null +++ b/observability/dev/simulator/kubestate.go @@ -0,0 +1,48 @@ +package main + +import ( + "github.com/prometheus/client_golang/prometheus" +) + +// Container request and limit values the simulated operator pod runs with, +// picked so that the Process panels show usage well inside the request and +// the limit at a visible distance above it. +const ( + simPodNamespace = "operators" + simPodName = "demo-operator-0" + simContainer = "manager" + simCPURequest = 0.25 + simCPULimit = 1 + simMemRequest = 128 << 20 + simMemLimit = 512 << 20 +) + +// newKubeStateMetrics returns a registry holding lookalikes of the two +// kube-state-metrics families the OCF Operator dashboard joins against for +// container requests and limits, with the label set kube-state-metrics gives +// them. The pod and namespace match the target labels the dev Prometheus +// stamps on the simulator's own scrape, which is what the join keys on. +func newKubeStateMetrics() *prometheus.Registry { + labels := []string{"namespace", "pod", "container", "node", "resource", "unit"} + requests := prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "kube_pod_container_resource_requests", + Help: "The number of requested request resource by a container.", + }, labels) + limits := prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "kube_pod_container_resource_limits", + Help: "The number of requested limit resource by a container.", + }, labels) + for _, r := range []struct { + resource, unit string + request, limit float64 + }{ + {"cpu", "core", simCPURequest, simCPULimit}, + {"memory", "byte", simMemRequest, simMemLimit}, + } { + requests.WithLabelValues(simPodNamespace, simPodName, simContainer, "node-a", r.resource, r.unit).Set(r.request) + limits.WithLabelValues(simPodNamespace, simPodName, simContainer, "node-a", r.resource, r.unit).Set(r.limit) + } + reg := prometheus.NewRegistry() + reg.MustRegister(requests, limits) + return reg +} diff --git a/observability/dev/simulator/kubestate_test.go b/observability/dev/simulator/kubestate_test.go new file mode 100644 index 00000000..8dd4545b --- /dev/null +++ b/observability/dev/simulator/kubestate_test.go @@ -0,0 +1,35 @@ +package main + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// The dashboards join kube_pod_container_resource_* to the operator's scrape +// on (namespace, pod), so the lookalikes must carry kube-state-metrics' label +// set and name the pod the dev Prometheus stamps on the simulator's target. +func TestKubeStateMetricsLookalikes(t *testing.T) { + mfs, err := newKubeStateMetrics().Gather() + require.NoError(t, err) + got := map[string]int{} + for _, mf := range mfs { + for _, m := range mf.GetMetric() { + labels := map[string]string{} + for _, l := range m.GetLabel() { + labels[l.GetName()] = l.GetValue() + } + assert.Equal(t, simPodNamespace, labels["namespace"], "%s namespace", mf.GetName()) + assert.Equal(t, simPodName, labels["pod"], "%s pod", mf.GetName()) + assert.Contains(t, []string{"cpu", "memory"}, labels["resource"], "%s resource", mf.GetName()) + assert.NotEmpty(t, labels["container"]) + assert.NotEmpty(t, labels["unit"]) + got[mf.GetName()]++ + } + } + assert.Equal(t, map[string]int{ + "kube_pod_container_resource_requests": 2, + "kube_pod_container_resource_limits": 2, + }, got) +} diff --git a/observability/dev/simulator/main.go b/observability/dev/simulator/main.go new file mode 100644 index 00000000..2b124bc4 --- /dev/null +++ b/observability/dev/simulator/main.go @@ -0,0 +1,82 @@ +// Command simulator exposes a synthetic operator's metrics on /metrics for the +// local observability stack under observability/dev, and the kube-state-metrics +// series the dashboards join against on /ksm/metrics. +// +// The framework's own series (resource applies, conditions) are recorded +// through the real pkg/metrics recorder; controller-runtime, workqueue, REST +// client and leader election series are lookalikes guarded by runtime_test.go. +// The world it plays is described in docs/observability.md. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "log" + "net/http" + "os" + "os/signal" + "syscall" + "time" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promhttp" + ocm "github.com/sourcehawk/go-crd-condition-metrics/pkg/crd-condition-metrics" + + "github.com/sourcehawk/operator-component-framework/pkg/metrics" +) + +func main() { + if err := run(); err != nil { + log.Fatal(err) + } +} + +// run wires the registry the way an operator would, serves /metrics and plays +// the world until the process is signalled. It returns an error when the +// metrics endpoint failed to serve, so main exits non-zero and +// `make observability-up` fails visibly instead of idling without metrics. +func run() error { + listen := flag.String("listen", ":8080", "address to serve /metrics on") + metricNamespace := flag.String("metric-namespace", "demo", "metric namespace of the condition gauge") + leader := flag.Bool("leader", true, "report this replica as the leader; false shows OperatorLeaderMissing") + flag.Parse() + + reg := prometheus.NewRegistry() + conditions := ocm.NewOperatorConditionsGauge(*metricNamespace) + collectors := metrics.NewCollectors() + reg.MustRegister(conditions, collectors) + rt := newRuntimeMetrics(reg) + + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + mux := http.NewServeMux() + mux.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{})) + mux.Handle("/ksm/metrics", promhttp.HandlerFor(newKubeStateMetrics(), promhttp.HandlerOpts{})) + srv := &http.Server{Addr: *listen, Handler: mux, ReadHeaderTimeout: 5 * time.Second} + serveErr := make(chan error, 1) + go func() { + log.Printf("serving metrics on %s/metrics", *listen) + if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + serveErr <- err + stop() + } + }() + + newWorld(conditions, collectors, rt, *leader).run(ctx) + + shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + shutdownErr := srv.Shutdown(shutdown) + select { + case err := <-serveErr: + return fmt.Errorf("serving metrics: %w", err) + default: + } + if shutdownErr != nil { + return fmt.Errorf("shutting down metrics server: %w", shutdownErr) + } + return nil +} diff --git a/observability/dev/simulator/runtime.go b/observability/dev/simulator/runtime.go new file mode 100644 index 00000000..f04b7975 --- /dev/null +++ b/observability/dev/simulator/runtime.go @@ -0,0 +1,142 @@ +package main + +import ( + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/collectors" + ctrlmetrics "sigs.k8s.io/controller-runtime/pkg/metrics" +) + +// runtimeMetrics defines the controller-runtime, workqueue, REST client and +// leader election series an operator exposes, with the same names, label sets +// and bucket layouts as the real ones. The real vectors live in +// controller-runtime's internal packages; runtime_test.go guards this copy +// against drift. +type runtimeMetrics struct { + reconcileTotal *prometheus.CounterVec + reconcileErrors *prometheus.CounterVec + reconcilePanics *prometheus.CounterVec + reconcileTime *prometheus.HistogramVec + maxConcurrent *prometheus.GaugeVec + activeWorkers *prometheus.GaugeVec + queueDepth *prometheus.GaugeVec + queueAdds *prometheus.CounterVec + queueDuration *prometheus.HistogramVec + workDuration *prometheus.HistogramVec + queueUnfinished *prometheus.GaugeVec + queueLongestRunning *prometheus.GaugeVec + queueRetries *prometheus.CounterVec + restRequests *prometheus.CounterVec + leader *prometheus.GaugeVec +} + +// Label names and the reconcile result value the lookalike series share with +// controller-runtime. +const ( + labelController = "controller" + labelName = "name" + resultError = "error" +) + +// reconcileTimeBuckets mirrors controller-runtime's ReconcileTime histogram. +var reconcileTimeBuckets = []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, + 1.25, 1.5, 1.75, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60} + +func newRuntimeMetrics(reg prometheus.Registerer) *runtimeMetrics { + m := &runtimeMetrics{ + reconcileTotal: prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "controller_runtime_reconcile_total", + Help: "Total number of reconciliations per controller", + }, []string{labelController, "result"}), + reconcileErrors: prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "controller_runtime_reconcile_errors_total", + Help: "Total number of reconciliation errors per controller", + }, []string{labelController}), + reconcilePanics: prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "controller_runtime_reconcile_panics_total", + Help: "Total number of reconciliation panics per controller", + }, []string{labelController}), + reconcileTime: prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Name: "controller_runtime_reconcile_time_seconds", + Help: "Length of time per reconciliation per controller", + Buckets: reconcileTimeBuckets, + }, []string{labelController}), + maxConcurrent: prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "controller_runtime_max_concurrent_reconciles", + Help: "Maximum number of concurrent reconciles per controller", + }, []string{labelController}), + activeWorkers: prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "controller_runtime_active_workers", + Help: "Number of currently used workers per controller", + }, []string{labelController}), + queueDepth: prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Subsystem: ctrlmetrics.WorkQueueSubsystem, Name: ctrlmetrics.DepthKey, + Help: "Current depth of workqueue by workqueue and priority", + }, []string{labelName, labelController, "priority"}), + queueAdds: prometheus.NewCounterVec(prometheus.CounterOpts{ + Subsystem: ctrlmetrics.WorkQueueSubsystem, Name: ctrlmetrics.AddsKey, + Help: "Total number of adds handled by workqueue", + }, []string{labelName, labelController}), + queueDuration: prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Subsystem: ctrlmetrics.WorkQueueSubsystem, Name: ctrlmetrics.QueueLatencyKey, + Help: "How long in seconds an item stays in workqueue before being requested", + Buckets: prometheus.ExponentialBuckets(10e-9, 10, 12), + }, []string{labelName, labelController}), + workDuration: prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Subsystem: ctrlmetrics.WorkQueueSubsystem, Name: ctrlmetrics.WorkDurationKey, + Help: "How long in seconds processing an item from workqueue takes.", + Buckets: prometheus.ExponentialBuckets(10e-9, 10, 12), + }, []string{labelName, labelController}), + queueUnfinished: prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Subsystem: ctrlmetrics.WorkQueueSubsystem, Name: ctrlmetrics.UnfinishedWorkKey, + Help: "How many seconds of work has been done that " + + "is in progress and hasn't been observed by work_duration. Large " + + "values indicate stuck threads. One can deduce the number of stuck " + + "threads by observing the rate at which this increases.", + }, []string{labelName, labelController}), + queueLongestRunning: prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Subsystem: ctrlmetrics.WorkQueueSubsystem, Name: ctrlmetrics.LongestRunningProcessorKey, + Help: "How many seconds has the longest running " + + "processor for workqueue been running.", + }, []string{labelName, labelController}), + queueRetries: prometheus.NewCounterVec(prometheus.CounterOpts{ + Subsystem: ctrlmetrics.WorkQueueSubsystem, Name: ctrlmetrics.RetriesKey, + Help: "Total number of items added to the workqueue with a non-zero delay (rate-limited requeues, explicit RequeueAfter or AddAfter calls)", + }, []string{labelName, labelController}), + restRequests: prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "rest_client_requests_total", + Help: "Number of HTTP requests, partitioned by status code, method, and host.", + }, []string{"code", "method", "host"}), + leader: prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "leader_election_master_status", + Help: "Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.", + }, []string{labelName}), + } + reg.MustRegister( + m.reconcileTotal, m.reconcileErrors, m.reconcilePanics, m.reconcileTime, m.maxConcurrent, m.activeWorkers, + m.queueDepth, m.queueAdds, m.queueDuration, m.workDuration, m.queueUnfinished, m.queueLongestRunning, m.queueRetries, + m.restRequests, m.leader, + collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}), + collectors.NewGoCollector(), + ) + return m +} + +// initController creates the zero-valued series controller-runtime creates +// when a controller starts, so panels show a flat line instead of no data. +func (m *runtimeMetrics) initController(name string, maxConcurrent int) { + for _, result := range []string{resultError, "requeue_after", "requeue", "success"} { + m.reconcileTotal.WithLabelValues(name, result).Add(0) + } + m.reconcileErrors.WithLabelValues(name).Add(0) + m.reconcilePanics.WithLabelValues(name).Add(0) + m.reconcileTime.WithLabelValues(name) + m.maxConcurrent.WithLabelValues(name).Set(float64(maxConcurrent)) + m.activeWorkers.WithLabelValues(name).Set(0) + m.queueDepth.WithLabelValues(name, name, "0").Set(0) + m.queueAdds.WithLabelValues(name, name).Add(0) + m.queueDuration.WithLabelValues(name, name) + m.workDuration.WithLabelValues(name, name) + m.queueUnfinished.WithLabelValues(name, name).Set(0) + m.queueLongestRunning.WithLabelValues(name, name).Set(0) + m.queueRetries.WithLabelValues(name, name).Add(0) +} diff --git a/observability/dev/simulator/runtime_test.go b/observability/dev/simulator/runtime_test.go new file mode 100644 index 00000000..f1705c32 --- /dev/null +++ b/observability/dev/simulator/runtime_test.go @@ -0,0 +1,142 @@ +package main + +import ( + "context" + "sort" + "strings" + "sync" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus" + dto "github.com/prometheus/client_model/go" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/types" + clientmetrics "k8s.io/client-go/tools/metrics" + "k8s.io/client-go/util/workqueue" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/controller" + ctrlmetrics "sigs.k8s.io/controller-runtime/pkg/metrics" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + "sigs.k8s.io/controller-runtime/pkg/source" +) + +// family is the shape of a metric family that the dashboards and alerts +// depend on: its type, its label names and, for histograms, its buckets. +type family struct { + kind dto.MetricType + labels []string + buckets []float64 +} + +func families(t *testing.T, g prometheus.Gatherer, prefixes ...string) map[string]family { + t.Helper() + mfs, err := g.Gather() + require.NoError(t, err) + out := map[string]family{} + for _, mf := range mfs { + name := mf.GetName() + matched := false + for _, p := range prefixes { + if strings.HasPrefix(name, p) { + matched = true + } + } + if !matched || len(mf.GetMetric()) == 0 { + continue + } + m := mf.GetMetric()[0] + f := family{kind: mf.GetType()} + for _, lp := range m.GetLabel() { + f.labels = append(f.labels, lp.GetName()) + } + sort.Strings(f.labels) + if h := m.GetHistogram(); h != nil { + for _, b := range h.GetBucket() { + f.buckets = append(f.buckets, b.GetUpperBound()) + } + } + out[name] = f + } + return out +} + +// TestRuntimeMetricsMatchControllerRuntime starts a real, unmanaged +// controller-runtime controller (no API server involved) and drives one +// request through its queue so that controller-runtime initialises its +// reconcile and workqueue series, then checks that every lookalike series the +// simulator defines exists in controller-runtime with the same type, label +// names and buckets. +func TestRuntimeMetricsMatchControllerRuntime(t *testing.T) { + reconciled := make(chan struct{}) + var once sync.Once + c, err := controller.NewUnmanaged("parity", controller.Options{ + Reconciler: reconcile.Func(func(context.Context, reconcile.Request) (reconcile.Result, error) { + once.Do(func() { close(reconciled) }) + return reconcile.Result{}, nil + }), + SkipNameValidation: ptr.To(true), + }) + require.NoError(t, err) + require.NoError(t, c.Watch(source.Func(func(_ context.Context, q workqueue.TypedRateLimitingInterface[reconcile.Request]) error { + q.Add(reconcile.Request{NamespacedName: types.NamespacedName{Namespace: "default", Name: "parity"}}) + return nil + }))) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + done := make(chan error, 1) + go func() { done <- c.Start(ctx) }() + select { + case <-reconciled: + case <-time.After(10 * time.Second): + require.FailNow(t, "the controller never reconciled the enqueued request") + } + // ReconcileTime is observed after the reconciler returns; wait for the + // family to appear before snapshotting the registry. + require.Eventually(t, func() bool { + mfs, err := ctrlmetrics.Registry.Gather() + if err != nil { + return false + } + for _, mf := range mfs { + if mf.GetName() == "controller_runtime_reconcile_time_seconds" && len(mf.GetMetric()) > 0 { + return true + } + } + return false + }, 10*time.Second, 10*time.Millisecond, "controller-runtime never observed a reconcile duration") + cancel() + require.NoError(t, <-done) + + // client-go creates rest_client_requests_total lazily on the first request; + // controller-runtime's pkg/metrics init installed the adapter, so one + // recorded result materialises the real family in the registry. + clientmetrics.RequestResult.Increment(context.Background(), "200", "GET", "10.96.0.1:443") + + want := families(t, ctrlmetrics.Registry, "controller_runtime_", "workqueue_", "rest_client_", "leader_election_") + + reg := prometheus.NewRegistry() + rm := newRuntimeMetrics(reg) + rm.initController("parity", 1) + rm.restRequests.WithLabelValues("200", "GET", "https://example").Add(0) + rm.leader.WithLabelValues("parity").Set(1) + got := families(t, reg, "controller_runtime_", "workqueue_", "rest_client_", "leader_election_") + + require.NotEmpty(t, got) + for name, g := range got { + w, ok := want[name] + if !ok && strings.HasPrefix(name, "leader_election_") { + // The leader elector creates leader_election_master_status only + // when it runs against an API server. Its definition is kept in + // step with controller-runtime's leaderelection.go by hand. + continue + } + if !assert.True(t, ok, "simulator metric %s does not exist in controller-runtime", name) { + continue + } + assert.Equal(t, w.kind, g.kind, "%s type", name) + assert.Equal(t, w.labels, g.labels, "%s labels", name) + assert.Equal(t, w.buckets, g.buckets, "%s buckets", name) + } +} diff --git a/observability/dev/simulator/world.go b/observability/dev/simulator/world.go new file mode 100644 index 00000000..80d4bfda --- /dev/null +++ b/observability/dev/simulator/world.go @@ -0,0 +1,327 @@ +package main + +import ( + "context" + "fmt" + "math/rand/v2" + "time" + + ocm "github.com/sourcehawk/go-crd-condition-metrics/pkg/crd-condition-metrics" + + "github.com/sourcehawk/operator-component-framework/pkg/component" + "github.com/sourcehawk/operator-component-framework/pkg/component/concepts" + "github.com/sourcehawk/operator-component-framework/pkg/metrics" +) + +// owner is a simulated custom resource. It satisfies ocm.ObjectLike. +type owner struct { + namespace, name string +} + +func (o owner) GetName() string { return o.name } +func (o owner) GetNamespace() string { return o.namespace } + +// resource is one managed resource of a component, as the framework labels it. +type resource struct { + component, identifier, kind string +} + +// controllerSim drives one controller's worth of series: a framework recorder +// for conditions and applies, and the controller-runtime lookalikes. +type controllerSim struct { + name string + ownerKind string + rec *metrics.Recorder + rt *runtimeMetrics +} + +func (c *controllerSim) labels(r resource) component.ResourceMetricLabels { + return component.ResourceMetricLabels{ + OwnerKind: c.ownerKind, Component: r.component, Identifier: r.identifier, Kind: r.kind, + } +} + +// reconcile records one reconcile: the workqueue hand-off, the reconcile +// result and duration, and a few REST calls. apply and condition recording is +// left to the caller, which knows the scenario. +func (c *controllerSim) reconcile(result string, queueWait, duration time.Duration) { + c.rt.queueAdds.WithLabelValues(c.name, c.name).Inc() + c.rt.queueDuration.WithLabelValues(c.name, c.name).Observe(queueWait.Seconds()) + c.rt.workDuration.WithLabelValues(c.name, c.name).Observe(duration.Seconds()) + c.rt.reconcileTime.WithLabelValues(c.name).Observe(duration.Seconds()) + c.rt.reconcileTotal.WithLabelValues(c.name, result).Inc() + if result == resultError { + c.rt.reconcileErrors.WithLabelValues(c.name).Inc() + c.rt.queueRetries.WithLabelValues(c.name, c.name).Inc() + } + c.rt.restRequests.WithLabelValues("200", "GET", "10.96.0.1:443").Add(float64(2 + rand.IntN(3))) + c.rt.restRequests.WithLabelValues("200", "PATCH", "10.96.0.1:443").Add(float64(1 + rand.IntN(4))) + if rand.IntN(20) == 0 { + c.rt.restRequests.WithLabelValues("409", "PATCH", "10.96.0.1:443").Inc() + } +} + +func (c *controllerSim) apply(r resource, op concepts.ConvergingOperation) { + c.rec.RecordResourceApply(c.labels(r), op) +} + +func (c *controllerSim) applyError(r resource) { + c.rec.RecordResourceApplyError(c.labels(r)) +} + +func (c *controllerSim) condition(o owner, conditionType, status, reason string, since time.Time) { + c.rec.RecordConditionFor(c.ownerKind, o, conditionType, status, reason, since) +} + +// world holds every scenario and runs them until ctx is done. +type world struct { + webapp, database *controllerSim + rt *runtimeMetrics + leader bool + start time.Time + // Scenario timing, overridable so the scripted-world test can drive the + // slow scenarios through a full cycle in milliseconds. + backlogInterval time.Duration + panicDelay time.Duration + panicInterval time.Duration +} + +func newWorld(conditions *ocm.OperatorConditionsGauge, collectors *metrics.Collectors, rt *runtimeMetrics, leader bool) *world { + w := &world{ + rt: rt, leader: leader, start: time.Now(), + backlogInterval: 5 * time.Second, panicDelay: time.Minute, panicInterval: 20 * time.Minute, + } + w.webapp = &controllerSim{ + name: "webapp", ownerKind: "WebApp", rt: rt, + rec: metrics.NewRecorder("webapp", conditions, collectors), + } + w.database = &controllerSim{ + name: "database", ownerKind: "Database", rt: rt, + rec: metrics.NewRecorder("database", conditions, collectors), + } + rt.initController("webapp", 4) + rt.initController("database", 2) + return w +} + +// Component and namespace names reused across the scripted world. +const ( + componentServer = "server" + namespaceShop = "shop" +) + +// Condition reasons the simulated owners report. +const ( + reasonHealthy = "Healthy" + reasonFailing = "Failing" + reasonCreating = "Creating" + reasonUnknown = "Unknown" +) + +var ( + webappDeployment = resource{componentServer, "deployment", "Deployment"} + webappService = resource{componentServer, "service", "Service"} + webappConfigMap = resource{componentServer, "configmap", "ConfigMap"} + webappIngress = resource{"ingress", "ingress", "Ingress"} + + dbStatefulSet = resource{"storage", "statefulset", "StatefulSet"} + dbPVC = resource{"storage", "pvc", "PersistentVolumeClaim"} + dbCronJob = resource{"backup", "cronjob", "CronJob"} +) + +// run starts every scenario goroutine and blocks until ctx is done. +func (w *world) run(ctx context.Context) { + w.rt.leader.WithLabelValues("demo-operator").Set(boolToFloat(w.leader)) + + go w.healthyWebApps(ctx) + go w.hotLoop(ctx) + go w.databases(ctx) + go w.workqueueBacklog(ctx) + go w.panics(ctx) + <-ctx.Done() +} + +func boolToFloat(b bool) float64 { + if b { + return 1 + } + return 0 +} + +// healthyWebApps: fifty owners; one random owner reconciles every six to +// eleven seconds, so each of the fifty is reconciled roughly every five to ten +// minutes on average. All resources converged, Ready=True since the simulator +// started. The hot-loop owner (webapp-01) is among them and is also healthy by +// its conditions. +func (w *world) healthyWebApps(ctx context.Context) { + c := w.webapp + owners := make([]owner, 50) + for i := range owners { + owners[i] = owner{namespace: fmt.Sprintf("team-%c", 'a'+rune(i%5)), name: fmt.Sprintf("webapp-%02d", i+1)} + } + tick := func(o owner) { + c.reconcile("success", time.Duration(rand.IntN(50))*time.Millisecond, time.Duration(50+rand.IntN(250))*time.Millisecond) + for _, r := range []resource{webappDeployment, webappService, webappConfigMap, webappIngress} { + c.apply(r, concepts.ConvergingOperationNone) + } + c.condition(o, "ServerReady", "True", reasonHealthy, w.start) + c.condition(o, "IngressReady", "True", reasonHealthy, w.start) + c.condition(o, "Ready", "True", reasonHealthy, w.start) + } + for _, o := range owners { + tick(o) + } + for { + select { + case <-ctx.Done(): + return + case <-time.After(time.Duration(6+rand.IntN(6)) * time.Second): + tick(owners[rand.IntN(len(owners))]) + } + } +} + +// hotLoop: webapp-01's configmap is rewritten on every reconcile, four times a +// second, the way a non-idempotent mutation behaves once the watch event +// requeues the owner. The other resources of that owner converge. +func (w *world) hotLoop(ctx context.Context) { + c := w.webapp + ticker := time.NewTicker(250 * time.Millisecond) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + c.reconcile("success", 5*time.Millisecond, time.Duration(30+rand.IntN(40))*time.Millisecond) + c.apply(webappConfigMap, concepts.ConvergingOperationUpdated) + c.apply(webappDeployment, concepts.ConvergingOperationNone) + c.apply(webappService, concepts.ConvergingOperationNone) + } + } +} + +// databases: six owners. orders-db has been Ready=False (Failing) for eight +// hours; users-db is Ready=Unknown; reports-db stays Ready=False while its +// reason flips between Failing and Creating; the rest are healthy, among them +// the cluster-scoped shared-gateway, whose empty namespace makes the condition +// gauge export series without a namespace label. PVC applies fail three times +// out of four, thirty percent of reconciles error, and reconcile durations are +// slow enough for the p99 to cross a minute. +func (w *world) databases(ctx context.Context) { + c := w.database + orders := owner{namespaceShop, "orders-db"} + users := owner{namespaceShop, "users-db"} + reports := owner{"analytics", "reports-db"} + healthy := []owner{{namespaceShop, "carts-db"}, {"analytics", "events-db"}} + clusterOwner := owner{namespace: "", name: "shared-gateway"} + stuckSince := w.start.Add(-8 * time.Hour) + unknownSince := w.start.Add(-1 * time.Hour) + flipSince := w.start + flipReason := reasonFailing + + i := 0 + tick := func() { + i++ + result := "success" + if rand.IntN(10) < 3 { + result = resultError + } + duration := time.Duration(5+rand.IntN(60)) * time.Second + if rand.IntN(10) < 2 { + duration = time.Duration(70+rand.IntN(50)) * time.Second + } + c.reconcile(result, time.Duration(rand.IntN(400))*time.Millisecond, duration) + c.apply(dbStatefulSet, concepts.ConvergingOperationNone) + c.apply(dbCronJob, concepts.ConvergingOperationNone) + if i%4 == 0 { + c.apply(dbPVC, concepts.ConvergingOperationNone) + } else { + c.applyError(dbPVC) + } + + if i%60 == 0 { // every three minutes + // A reason-only change keeps lastTransitionTime, as + // meta.SetStatusCondition does while the status stays False. + if flipReason == reasonFailing { + flipReason = reasonCreating + } else { + flipReason = reasonFailing + } + } + c.condition(orders, "StorageReady", "False", reasonFailing, stuckSince) + c.condition(orders, "BackupReady", "True", reasonHealthy, stuckSince) + c.condition(orders, "Ready", "False", reasonFailing, stuckSince) + c.condition(users, "StorageReady", "Unknown", reasonUnknown, unknownSince) + c.condition(users, "BackupReady", "True", reasonHealthy, unknownSince) + c.condition(users, "Ready", "Unknown", reasonUnknown, unknownSince) + c.condition(reports, "StorageReady", "False", flipReason, flipSince) + c.condition(reports, "BackupReady", "True", reasonHealthy, w.start) + c.condition(reports, "Ready", "False", flipReason, flipSince) + for _, o := range healthy { + c.condition(o, "StorageReady", "True", reasonHealthy, w.start) + c.condition(o, "BackupReady", "True", reasonHealthy, w.start) + c.condition(o, "Ready", "True", reasonHealthy, w.start) + } + c.condition(clusterOwner, "StorageReady", "True", reasonHealthy, w.start) + c.condition(clusterOwner, "Ready", "True", reasonHealthy, w.start) + } + tick() + ticker := time.NewTicker(3 * time.Second) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + tick() + } + } +} + +// workqueueBacklog: the database queue is deep, items wait minutes, and both +// of the database controller's workers stay busy chewing through it, while +// webapp's workers idle between zero and two. A reconcile-scoped flip of +// active_workers would almost never be caught by a 5s scrape, so the gauges +// are modelled as persistent levels instead. +func (w *world) workqueueBacklog(ctx context.Context) { + c := w.database + ticker := time.NewTicker(w.backlogInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + depth := 30 + rand.IntN(20) + c.rt.queueDepth.WithLabelValues(c.name, c.name, "0").Set(float64(depth)) + c.rt.queueDuration.WithLabelValues(c.name, c.name).Observe(float64(200 + rand.IntN(400))) + c.rt.queueUnfinished.WithLabelValues(c.name, c.name).Set(float64(60 + rand.IntN(120))) + c.rt.queueLongestRunning.WithLabelValues(c.name, c.name).Set(float64(30 + rand.IntN(90))) + c.rt.queueDepth.WithLabelValues(w.webapp.name, w.webapp.name, "0").Set(float64(rand.IntN(3))) + c.rt.activeWorkers.WithLabelValues(c.name).Set(2) + c.rt.activeWorkers.WithLabelValues(w.webapp.name).Set(float64(rand.IntN(3))) + } + } +} + +// panics: the database controller panics once every 20 minutes, starting +// one minute in, so ControllerReconcilePanics is visible without waiting long. +func (w *world) panics(ctx context.Context) { + select { + case <-ctx.Done(): + return + case <-time.After(w.panicDelay): + } + ticker := time.NewTicker(w.panicInterval) + defer ticker.Stop() + for { + w.database.rt.reconcilePanics.WithLabelValues("database").Inc() + w.database.rt.reconcileTotal.WithLabelValues("database", "error").Inc() + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + } +} diff --git a/observability/dev/simulator/world_test.go b/observability/dev/simulator/world_test.go new file mode 100644 index 00000000..a5dbbfef --- /dev/null +++ b/observability/dev/simulator/world_test.go @@ -0,0 +1,145 @@ +package main + +import ( + "context" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus" + dto "github.com/prometheus/client_model/go" + ocm "github.com/sourcehawk/go-crd-condition-metrics/pkg/crd-condition-metrics" + "github.com/stretchr/testify/require" + + "github.com/sourcehawk/operator-component-framework/pkg/metrics" +) + +// seriesValue finds the first series of the named family whose labels are a +// superset of want and returns its gauge or counter value. +func seriesValue(mfs []*dto.MetricFamily, name string, want map[string]string) (float64, bool) { + for _, mf := range mfs { + if mf.GetName() != name { + continue + } + for _, m := range mf.GetMetric() { + labels := map[string]string{} + for _, lp := range m.GetLabel() { + labels[lp.GetName()] = lp.GetValue() + } + matched := true + for k, v := range want { + if labels[k] != v { + matched = false + break + } + } + if !matched { + continue + } + switch { + case m.GetGauge() != nil: + return m.GetGauge().GetValue(), true + case m.GetCounter() != nil: + return m.GetCounter().GetValue(), true + } + return 0, true + } + } + return 0, false +} + +// TestWorldScriptedScenarios wires a registry the way main does, runs the +// world and checks that every scenario's series appear with the shapes the +// dev stack's alerts key on: the stuck Ready condition with its eight hour +// old transition, the failing PVC applies, the hot loop's rising updated +// counter, the cluster-scoped owner without a namespace label, both +// controllers reconciling and the leader gauge at one. It then cancels the +// context and requires run to return. +func TestWorldScriptedScenarios(t *testing.T) { + reg := prometheus.NewRegistry() + conditions := ocm.NewOperatorConditionsGauge("demo") + collectors := metrics.NewCollectors() + reg.MustRegister(conditions, collectors) + rt := newRuntimeMetrics(reg) + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + done := make(chan struct{}) + w := newWorld(conditions, collectors, rt, true) + // The backlog and panic scenarios run on minute-scale timers in the dev + // stack; shorten them so the test sees both fire more than once. + w.backlogInterval = 50 * time.Millisecond + w.panicDelay = 50 * time.Millisecond + w.panicInterval = 50 * time.Millisecond + go func() { + w.run(ctx) + close(done) + }() + + var firstUpdated float64 + require.Eventually(t, func() bool { + mfs, err := reg.Gather() + if err != nil { + return false + } + stuck, ok := seriesValue(mfs, "demo_controller_condition", map[string]string{ + "kind": "Database", "name": "orders-db", "condition": "Ready", "status": "False", "reason": "Failing", + }) + if !ok || time.Since(time.Unix(int64(stuck), 0)) < 7*time.Hour { + return false + } + if _, ok := seriesValue(mfs, "ocf_resource_apply_errors_total", map[string]string{"resource": "pvc"}); !ok { + return false + } + updated, ok := seriesValue(mfs, "ocf_resource_apply_total", map[string]string{ + "resource": "configmap", "operation": "updated", + }) + if !ok || updated == 0 { + return false + } + if firstUpdated == 0 { + firstUpdated = updated + return false // seen once; wait until it has risen + } + if updated <= firstUpdated { + return false + } + // The cluster-scoped owner: client_golang exports the declared + // namespace label with an empty value, which Prometheus ingests as no + // namespace label at all, and the id keeps its "/" shape. + if _, ok := seriesValue(mfs, "demo_controller_condition", map[string]string{ + "kind": "Database", "name": "shared-gateway", "namespace": "", + "id": "/shared-gateway", "condition": "Ready", "status": "True", + }); !ok { + return false + } + for _, controller := range []string{"webapp", "database"} { + if _, ok := seriesValue(mfs, "controller_runtime_reconcile_total", map[string]string{"controller": controller}); !ok { + return false + } + } + // The backlog scenario keeps the database queue deep with items waiting + // and both workers busy, and the panic scenario keeps counting. + depth, ok := seriesValue(mfs, "workqueue_depth", map[string]string{"controller": "database"}) + if !ok || depth < 30 { + return false + } + if _, ok := seriesValue(mfs, "workqueue_queue_duration_seconds", map[string]string{"controller": "database"}); !ok { + return false + } + if active, ok := seriesValue(mfs, "controller_runtime_active_workers", map[string]string{"controller": "database"}); !ok || active != 2 { + return false + } + if panics, ok := seriesValue(mfs, "controller_runtime_reconcile_panics_total", map[string]string{"controller": "database"}); !ok || panics < 2 { + return false + } + leader, ok := seriesValue(mfs, "leader_election_master_status", map[string]string{"name": "demo-operator"}) + return ok && leader == 1 + }, 5*time.Second, 100*time.Millisecond, "the scripted world never produced the expected series") + + cancel() + select { + case <-done: + case <-time.After(2 * time.Second): + require.FailNow(t, "run did not return after the context was cancelled") + } +} diff --git a/observability/observability_test.go b/observability/observability_test.go new file mode 100644 index 00000000..b74c4c57 --- /dev/null +++ b/observability/observability_test.go @@ -0,0 +1,219 @@ +// Package observability_test checks the dashboard and alert templates: that +// they render to valid JSON/YAML, keep their uids, leave no placeholder behind +// and reference only metric names that actually exist. +package observability_test + +import ( + "encoding/json" + "os" + "path/filepath" + "regexp" + "strconv" + "strings" + "testing" + + "github.com/prometheus/client_golang/prometheus" + ocm "github.com/sourcehawk/go-crd-condition-metrics/pkg/crd-condition-metrics" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "sigs.k8s.io/yaml" + + "github.com/sourcehawk/operator-component-framework/pkg/metrics" +) + +const ( + lintNamespace = "lint_operator" + // maxMetricNamespaceLen is the cap the Makefile's require_metric_namespace + // enforces on METRIC_NAMESPACE, derived from maxGrafanaUIDLen and the + // longest dashboard file name. + maxMetricNamespaceLen = 17 + // maxGrafanaUIDLen is Grafana's limit on a dashboard uid. + maxGrafanaUIDLen = 40 + nsLabel = "exported_namespace" +) + +// render mirrors the Makefile's render_template. +func render(t *testing.T, path string) string { + t.Helper() + b, err := os.ReadFile(path) + require.NoError(t, err) + s := strings.ReplaceAll(string(b), "{{operator_namespace}}", lintNamespace+"_") + return strings.ReplaceAll(s, "{{namespace_label}}", nsLabel) +} + +// knownMetrics is every metric name the templates may reference: the +// framework's own, the condition gauge, and the controller-runtime, workqueue, +// client-go, leader election, process and Go runtime families. +func knownMetrics(t *testing.T) map[string]bool { + t.Helper() + known := map[string]bool{} + descs := make(chan *prometheus.Desc, 64) + go func() { + metrics.NewCollectors().Describe(descs) + ocm.NewOperatorConditionsGauge(lintNamespace).Describe(descs) + close(descs) + }() + fq := regexp.MustCompile(`fqName: "([^"]+)"`) + for d := range descs { + m := fq.FindStringSubmatch(d.String()) + require.Len(t, m, 2, d.String()) + known[m[1]] = true + } + for _, n := range []string{ + "controller_runtime_reconcile_total", "controller_runtime_reconcile_errors_total", + "controller_runtime_reconcile_panics_total", "controller_runtime_reconcile_time_seconds", + "controller_runtime_max_concurrent_reconciles", "controller_runtime_active_workers", + "workqueue_depth", "workqueue_adds_total", "workqueue_queue_duration_seconds", + "workqueue_work_duration_seconds", "workqueue_unfinished_work_seconds", + "workqueue_longest_running_processor_seconds", "workqueue_retries_total", + "rest_client_requests_total", "leader_election_master_status", + "process_cpu_seconds_total", "process_resident_memory_bytes", "go_goroutines", + "kube_pod_container_resource_requests", "kube_pod_container_resource_limits", "up", + } { + known[n] = true + } + return known +} + +// metricNameRe pulls every identifier that looks like one of the metric +// families we care about out of a PromQL expression, including the +// `_bucket`/`_sum`/`_count` suffixes of histograms. +var metricNameRe = regexp.MustCompile(`\b((?:ocf|controller_runtime|workqueue|rest_client|leader_election|process|go|kube)_[a-z0-9_]+|up|` + lintNamespace + `_controller_condition)\b`) + +// selectorRe pulls the metric name out of every selector-shaped token +// (`name{...}`). It catches typos and unknown metric families that the prefix +// whitelist of metricNameRe skips over. +var selectorRe = regexp.MustCompile(`([a-zA-Z_:][a-zA-Z0-9_:]*)\s*\{`) + +// promqlKeywords are identifiers that may legally precede `{` in PromQL +// without naming a metric. +var promqlKeywords = map[string]bool{ + "by": true, "on": true, "ignoring": true, + "group_left": true, "group_right": true, "without": true, +} + +// placeholderRe matches a `{{name}}` template placeholder. After rendering, +// any hit in a PromQL expression is a mis-spelled placeholder the render left +// behind. +var placeholderRe = regexp.MustCompile(`\{\{ *[a-zA-Z_]+ *\}\}`) + +// assertExprClean checks a rendered PromQL expression: every metric-family +// token and every selector names a known metric, and no template placeholder +// survived rendering. +func assertExprClean(t *testing.T, known map[string]bool, expr, context string) { + t.Helper() + for _, m := range metricNameRe.FindAllString(expr, -1) { + assert.True(t, known[baseName(m)], "unknown metric %q in %s", m, context) + } + for _, m := range selectorRe.FindAllStringSubmatch(expr, -1) { + if promqlKeywords[m[1]] { + continue + } + assert.True(t, known[baseName(m[1])], "unknown selector metric %q in %s", m[1], context) + } + assert.NotRegexp(t, placeholderRe, expr, "unrendered placeholder in %s", context) +} + +// baseName strips the histogram sample suffixes off a series name so that it +// can be looked up as a metric family. +func baseName(n string) string { + for _, s := range []string{"_bucket", "_sum", "_count"} { + n = strings.TrimSuffix(n, s) + } + return n +} + +// exprsFromJSON collects every PromQL-carrying string (panel targets, variable +// queries and definitions) from a decoded dashboard. +func exprsFromJSON(v any, out *[]string) { + switch x := v.(type) { + case map[string]any: + for k, val := range x { + if k == "expr" || k == "query" || k == "definition" { + if s, ok := val.(string); ok { + *out = append(*out, s) + } + } + exprsFromJSON(val, out) + } + case []any: + for _, val := range x { + exprsFromJSON(val, out) + } + } +} + +func TestDashboards(t *testing.T) { + files, err := filepath.Glob("dashboards/*.tpl.json") + require.NoError(t, err) + require.NotEmpty(t, files) + known := knownMetrics(t) + for _, f := range files { + t.Run(filepath.Base(f), func(t *testing.T) { + rendered := render(t, f) + assert.NotContains(t, rendered, "{{operator_namespace}}") + assert.NotContains(t, rendered, "{{namespace_label}}") + var d map[string]any + require.NoError(t, json.Unmarshal([]byte(rendered), &d), "valid JSON") + want := strings.TrimSuffix(filepath.Base(f), ".tpl.json") + assert.Equal(t, lintNamespace+"_"+want, d["uid"], "uid is the file name prefixed with the metric namespace") + // Grafana limits a uid to 40 characters of [A-Za-z0-9_-]. The Makefile + // caps METRIC_NAMESPACE at maxMetricNamespaceLen on that basis, so a + // dashboard file name that no longer leaves room for it must fail here. + assert.Regexp(t, `^[A-Za-z0-9_-]+$`, want, "dashboard file name is made of Grafana uid characters") + assert.LessOrEqual(t, maxMetricNamespaceLen+1+len(want), maxGrafanaUIDLen, + "a METRIC_NAMESPACE of %d characters must still render a uid within Grafana's %d character limit; shorten the file name or lower the cap in the Makefile", maxMetricNamespaceLen, maxGrafanaUIDLen) + assert.Equal(t, "", d["refresh"], "auto refresh is off by default") + var exprs []string + exprsFromJSON(d, &exprs) + require.NotEmpty(t, exprs) + for _, e := range exprs { + assertExprClean(t, known, e, strconv.Quote(e)) + } + }) + } +} + +func TestAlerts(t *testing.T) { + // The glob matches the shared files and, because `.tpl.yaml` also ends in + // `.yaml`, the templated ones too; render handles both. + files, err := filepath.Glob("alerts/*.yaml") + require.NoError(t, err) + require.NotEmpty(t, files) + known := knownMetrics(t) + for _, f := range files { + t.Run(filepath.Base(f), func(t *testing.T) { + rendered := render(t, f) + assert.NotContains(t, rendered, "{{operator_namespace}}") + assert.NotContains(t, rendered, "{{namespace_label}}") + var doc struct { + Groups []struct { + Name string `json:"name"` + Rules []struct { + Alert string `json:"alert"` + Expr string `json:"expr"` + Labels map[string]string `json:"labels"` + } `json:"rules"` + } `json:"groups"` + } + require.NoError(t, yaml.Unmarshal([]byte(rendered), &doc)) + require.NotEmpty(t, doc.Groups) + for _, g := range doc.Groups { + for _, r := range g.Rules { + assert.Len(t, r.Labels, 1, "%s carries only the severity label", r.Alert) + assert.Contains(t, []string{"warning", "critical"}, r.Labels["severity"], "%s severity", r.Alert) + assertExprClean(t, known, r.Expr, r.Alert) + } + } + // Every alert in the file has a promtool unit test. + base := strings.TrimSuffix(strings.TrimSuffix(filepath.Base(f), ".yaml"), ".tpl") + tests, err := os.ReadFile(filepath.Join("alerts", "tests", base+"_test.yaml")) + require.NoError(t, err, "every rule file has a promtool test file") + for _, g := range doc.Groups { + for _, r := range g.Rules { + assert.Contains(t, string(tests), "alertname: "+r.Alert+"\n", "%s has a unit test", r.Alert) + } + } + }) + } +} diff --git a/pkg/metrics/metrics.go b/pkg/metrics/metrics.go index b316d6f4..0d194767 100644 --- a/pkg/metrics/metrics.go +++ b/pkg/metrics/metrics.go @@ -17,9 +17,14 @@ // // rec := component.ReconcileContext{ // // ... -// Metrics: metrics.NewRecorder("webapp-controller", conditions, collectors), +// Metrics: metrics.NewRecorder("webapp", conditions, collectors), // } // +// The controller name must match the controller-runtime controller name (the +// lower-cased kind passed to For, unless Named overrides it) so the dashboards +// and alerts shipped under observability/ correlate the framework's series +// with controller-runtime's. +// // [go-crd-condition-metrics]: https://github.com/sourcehawk/go-crd-condition-metrics package metrics @@ -112,10 +117,12 @@ var _ component.MetricsRecorder = (*Recorder)(nil) // NewRecorder creates a Recorder for the named controller. // // controller is the value of the `controller` label on every series the -// recorder emits, condition metrics included, so it must match the name used -// for that controller elsewhere. conditions and collectors are the shared, -// registered collectors; passing nil for either disables that half of the -// recording rather than panicking at reconcile time. +// recorder emits, condition metrics included, so it must match the name +// controller-runtime uses for that controller (the lower-cased kind passed to +// For, unless Named overrides it); the shipped dashboards and alerts filter +// both families with one controller label. conditions and collectors are the +// shared, registered collectors; passing nil for either disables that half of +// the recording rather than panicking at reconcile time. func NewRecorder( controller string, conditions *ocm.OperatorConditionsGauge, collectors *Collectors, ) *Recorder { diff --git a/plugin/skills/building-components/SKILL.md b/plugin/skills/building-components/SKILL.md index c2fdb474..daf407a7 100644 --- a/plugin/skills/building-components/SKILL.md +++ b/plugin/skills/building-components/SKILL.md @@ -6,7 +6,9 @@ description: gates, prerequisites, the reconciliation lifecycle, conditions and the status model, reading a component's condition with GetCondition, aggregating component conditions into an owner-level condition, grace periods, suspension, ReconcileContext, how a controller writes the owner CR's status (including a status-only controller that manages no - resources) and where observedGeneration is set, FlushStatus, guards, and declared data cells. + resources) and where observedGeneration is set, FlushStatus, guards, declared data cells, wiring metrics.NewRecorder + (its controller name must match the controller-runtime controller name), the shipped Grafana dashboards and Prometheus + alert rules, and what an alert such as ManagedResourceNotConverging means. --- # Building Components @@ -316,3 +318,7 @@ with `go doc`, `go doc` wins. - `references/component.md`: full component documentation. Read when you need exact builder signatures, status constants, lifecycle phase details, or guard semantics. +- `references/observability.md`: the Grafana dashboards and Prometheus alert rules shipped for the metrics a + `ReconcileContext.Metrics` recorder emits. Read when wiring `metrics.NewRecorder` (its controller name must match + controller-runtime's controller name), rendering the dashboards and alerts for an operator, or explaining what an + alert such as `ManagedResourceNotConverging` means. diff --git a/plugin/skills/building-components/references/component.md b/plugin/skills/building-components/references/component.md index 9d21e7ae..a0f674b2 100644 --- a/plugin/skills/building-components/references/component.md +++ b/plugin/skills/building-components/references/component.md @@ -842,12 +842,15 @@ func init() { recCtx := component.ReconcileContext{ // ... - Metrics: metrics.NewRecorder("webapp-controller", conditions, collectors), + Metrics: metrics.NewRecorder("webapp", conditions, collectors), } ``` -The controller name becomes the `controller` label on every series the recorder emits. Passing `nil` for either -collector disables that family; passing `nil` for `Metrics` itself disables both. +The controller name becomes the `controller` label on every series the recorder emits. It must match the +controller-runtime controller name (the lower-cased kind passed to `For`, unless `Named` overrides it) so that the +shipped [dashboards and alerts](observability.md) correlate the framework's series with controller-runtime's reconcile +and workqueue series. Passing `nil` for either collector disables that family; passing `nil` for `Metrics` itself +disables both. ### Resource metrics diff --git a/plugin/skills/building-components/references/observability.md b/plugin/skills/building-components/references/observability.md new file mode 100644 index 00000000..7da5d072 --- /dev/null +++ b/plugin/skills/building-components/references/observability.md @@ -0,0 +1,357 @@ +# Observability + +The framework ships Grafana dashboards and Prometheus alert rules for the metrics an operator built on it exposes: the +[condition and resource apply metrics](component.md#metrics) recorded through `pkg/metrics`, and the reconcile, +workqueue, REST client, leader election and process series every controller-runtime operator exports. They live under +`observability/` in the repository as templates, keyed on the metric namespace of your operator, and render with `make`. +A local Prometheus and Grafana stack fed by a simulator lets you look at every panel and every alert without a cluster. + +## What ships + +| Artifact | File | Scope | +| ------------------------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| OCF Operator dashboard | `dashboards/ocf_operator.tpl.json` | Per operator. Rendered uid `_ocf_operator`. Operator health end to end: reconciliation, workqueue, managed resource applies, condition summary, API client, process. | +| CRD Conditions Browser | `dashboards/crd_conditions_browser.tpl.json` | Per operator. Rendered uid `_crd_conditions_browser`. Per-owner condition drill-down; the target of the condition alerts' `dashboard_url` links. | +| Condition alerts | `alerts/crd_conditions.tpl.yaml` | Per operator, rendered per metric namespace as the `PrometheusRule` `-crd-conditions`. `CustomResourceNotReady`, `CustomResourceConditionUnknown`, `CustomResourceConditionStuck`. | +| Managed resource alerts | `alerts/managed_resources.yaml` | Shared, cluster-wide. Installed once as the `PrometheusRule` `ocf-managed-resources`, whichever operator rendered it. `ManagedResourceNotConverging`, `ManagedResourceApplyFailing`. | +| Controller-runtime alerts | `alerts/controller_runtime.yaml` | Shared, cluster-wide. Installed once as the `PrometheusRule` `ocf-controller-runtime`. `ControllerReconcileErrors`, `ControllerReconcilePanics`, `ControllerWorkqueueBacklog`, `ControllerReconcileLatencyHigh`, `OperatorLeaderMissing`. | + +The split follows the metrics. The condition gauge is named after the metric namespace +(`_controller_condition`), so its rules and both dashboards carry a placeholder and render per +operator. The apply counters (`ocf_resource_apply_total`, `ocf_resource_apply_errors_total`) and the controller-runtime +families have fixed names shared by every operator in the cluster, so their rules contain no placeholder, tell operators +apart by label, and are installed once. `make alerts` writes the shared files alongside the per-operator one on every +render; apply them from whichever operator's render you like, the content is identical. + +## Rendering + +The render pipeline is `make` and `sed`; it needs no Go toolchain. Three different things are called a namespace on this +page, so to be precise: the **metric namespace** is the string your operator passed to `ocm.NewOperatorConditionsGauge` +(the prefix of the condition gauge's name), the **owner namespace** is the Kubernetes namespace of a custom resource, +carried as a label on its condition series, and the **operator namespace** is the Kubernetes namespace the operator pod +runs in, stamped on every series by the scrape job. + +Clone the repository and, from its root, render with your metric namespace: + +```bash +make dashboards METRIC_NAMESPACE=myoperator +make alerts METRIC_NAMESPACE=myoperator +``` + +Output lands in `observability/generated/`, which is gitignored. Each target removes the files it previously rendered +before writing, so a dashboard or rule file that a framework upgrade renamed or dropped does not linger and get +installed again: + +``` +observability/generated/ +├── alerts/ +│ ├── controller_runtime.yaml PrometheusRule ocf-controller-runtime (shared) +│ ├── crd_conditions.yaml PrometheusRule myoperator-crd-conditions +│ └── managed_resources.yaml PrometheusRule ocf-managed-resources (shared) +└── dashboards/ + ├── crd_conditions_browser.json + └── ocf_operator.json +``` + +Two placeholders are substituted: `{{operator_namespace}}` becomes `_`, so every reference to the +condition gauge reads `myoperator_controller_condition` (the placeholder is named for the metric namespace, not a +Kubernetes one), and `{{namespace_label}}` becomes the value of `NAMESPACE_LABEL`. Pass the same variables to both +`make dashboards` and `make alerts`. Both placeholders and every variable below are the same as in +[go-crd-condition-metrics](https://github.com/sourcehawk/go-crd-condition-metrics), so a build that already renders that +repository's artifacts needs no change. + +| Variable | Default | Effect | +| -------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `METRIC_NAMESPACE` | required | The argument to `ocm.NewOperatorConditionsGauge`. Names the condition gauge, the dashboard uids and the condition `PrometheusRule`. A letter followed by metric name characters (`[A-Za-z0-9_]`), at most 17 characters: it names the `PrometheusRule` objects with `_` mapped to `-`, so it must start with a letter, and the longest rendered uid must fit Grafana's 40 character limit; rendering fails otherwise. | +| `NAMESPACE_LABEL` | `exported_namespace` | The label carrying the owner's namespace on condition series, see below. Must be a Prometheus label name (`[A-Za-z0-9_]`, not starting with a digit); rendering fails otherwise. | +| `ALERT_FORMAT` | `prometheusrule` | `prometheusrule` wraps each rule file in a `monitoring.coreos.com/v1` `PrometheusRule` for the Prometheus Operator; `rules` writes the plain `groups:` file that Prometheus loads through `rule_files`. | +| `PROMETHEUSRULE_NAMESPACE` | unset | `metadata.namespace` of the `PrometheusRule` objects. Unset leaves it to `kubectl apply -n`. | +| `PROMETHEUSRULE_LABELS` | unset | Comma-separated `key=value` pairs written to `metadata.labels`. An entry without `=` fails the render. A kube-prometheus-stack install selects rules by its release label, so pass `PROMETHEUSRULE_LABELS=release=`. | +| `OBS_OUT` | `observability/generated` | Render output directory. | + +`PrometheusRule` names are the metric namespace or `ocf` prefix joined to the file name, lower-cased, with `_` and `:` +folded to `-`, so a namespace of `My_Operator` renders as `my-operator-crd-conditions`. + +### The namespace label + +The condition gauge exports the owner's namespace as a `namespace` label. When a ServiceMonitor or PodMonitor scrapes +the operator, Prometheus stamps the scrape target's own labels on every series, and the target's `namespace` label (the +operator pod's namespace) collides with the exported one. Prometheus resolves the collision by renaming the exported +label to `exported_namespace`, which is why that is the default. Pass `NAMESPACE_LABEL=namespace` when your scrape sets +`honorLabels: true`, or does not stamp a `namespace` target label at all, so the exported label arrives unchanged. + +The setting affects the condition rules and both dashboards. Nothing else is namespace-scoped by owner: the apply +counters carry no owner namespace by design, and the `namespace` and `job` the controller-runtime and managed-resource +rules aggregate by are the operator's own namespace and scrape job, stamped by Prometheus. Together they let two +installs of one operator in a cluster alert separately, and keep two operators in one namespace that happen to share a +controller name from merging into one ratio, where a healthy operator would dilute a failing one below the threshold. +Outside a cluster both labels are simply absent, which is harmless. + +### Installing + +With the default `ALERT_FORMAT`, apply the rendered rules into the namespace your Prometheus Operator watches: + +```bash +make alerts METRIC_NAMESPACE=myoperator PROMETHEUSRULE_NAMESPACE=monitoring PROMETHEUSRULE_LABELS=release=kube-prometheus-stack +kubectl apply -f observability/generated/alerts/ +``` + +With `ALERT_FORMAT=rules`, add the three files to your Prometheus `rule_files`. + +For the dashboards, either import the two JSON files through Grafana's UI or API, or, with the Grafana sidecar that +kube-prometheus-stack deploys, ship them as a ConfigMap carrying the sidecar's label (`grafana_dashboard` by default): + +```bash +kubectl create configmap myoperator-dashboards -n monitoring --from-file=observability/generated/dashboards/ +kubectl label configmap myoperator-dashboards -n monitoring grafana_dashboard=1 +``` + +Check the rendered files into the repository that deploys your operator, and re-render when you upgrade the framework. +Tune thresholds and `for:` durations in the rendered files, or with a kustomize patch over them; the sections below say +what each threshold means so the change is deliberate. Give the two shared `PrometheusRule` objects one owner in the +cluster: two teams applying differently tuned copies under the same name overwrite each other, and applying them into +two namespaces installs every shared alert twice. + +## Naming the controller + +The OCF Operator dashboard filters four metric families with one `controller` variable: controller-runtime's reconcile +series, the workqueue series, the framework's apply counters and the condition gauge. controller-runtime labels the +first two with the name of the controller, which is the lower-cased kind passed to `For` unless `Named` overrides it. +The framework labels the last two with the name you pass to `metrics.NewRecorder`. For the dashboard to correlate them, +the two names must be the same: + +```go +ctrl.NewControllerManagedBy(mgr). + For(&v1.WebApp{}). + Named("webapp"). // optional, "webapp" is the default for kind WebApp + Complete(r) + +recCtx := component.ReconcileContext{ + // ... + Metrics: metrics.NewRecorder("webapp", conditions, collectors), +} +``` + +The alerts key on the same label, so the `controller` in a `ManagedResourceNotConverging` notification and the +`controller` in a `ControllerReconcileErrors` notification then name the same thing. `OperatorLeaderMissing` is the one +exception: leader election is per operator, not per controller, and its `name` label is the lease name. + +## Alerts + +Every rule ships with `severity: warning` and no routing labels; severity, thresholds and routing are yours to tune. No +rule on the apply counters or the controller-runtime metrics creates a series per owner: they aggregate by the +operator's static topology, so the same rules hold whether the operator manages three owners or three thousand. The +per-owner signal comes from the condition rules. + +### Managed resources + +Shared, installed once as `ocf-managed-resources`. Both rules key on +`(namespace, job, controller, owner_kind, component, resource, kind)`: the labels of `ocf_resource_apply_total` plus the +scrape namespace and job. They fire per resource type, not per owner, because the counters carry no owner identity. + +| Alert | Fires when | Threshold | `for` | +| ------------------------------ | ---------------------------------------------------- | ----------------------------------------------------------------------------- | ----- | +| `ManagedResourceNotConverging` | most applies of one resource type rewrite the object | `updated / all applies` over 15m `> 0.5`, and more than 15 updates in 15m | 15m | +| `ManagedResourceApplyFailing` | most apply attempts of one resource type fail | `errors / (errors + applies)` over 15m `> 0.5`, and more than 5 errors in 15m | 15m | + +`ManagedResourceNotConverging` is the alert the apply counters were built for: a managed resource rewritten on most +reconcile. [Resource metrics](component.md#resource-metrics) explains what an `updated` apply on every reconcile means +and why events do not catch it. + +The rule measures the share of a resource's own applies that rewrote it, not the absolute `updated` rate. A bare +`rate(updated) > 0` is wrong at scale: legitimate spec changes across many owners keep the aggregate `updated` rate +above zero indefinitely. Legitimate churn is followed by a `none` apply on the next reconcile, which keeps its ratio at +or below one half; a hot loop pushes the ratio to one. The floor of 15 updates in 15 minutes keeps a single edit on an +otherwise idle resource from producing a ratio of one over a handful of samples. Raise the floor if your operator's +resources are edited in bursts; lower the ratio only if you are sure your reconcile cadence never produces a `none` +between two legitimate updates. + +`ManagedResourceApplyFailing` counts every failure of an attempt: mutating the desired object, the server-side apply +patch, and the classification after it. Transient conflicts among successful applies stay under the ratio, and a +resource failing for one owner among many stays under the floor; that owner's `Ready` condition goes `False` and the +condition rules catch it. The `or` in the denominator keeps the ratio defined for a resource that has never applied +successfully, where no success series exists to add. The framework records no event for a failed apply, so the `kubectl` +command in the notification's description lists the owners' `Ready` condition reason and message, which is where the +failure lives. + +### Controller-runtime + +Shared, installed once as `ocf-controller-runtime`. All but the last rule aggregate by `(namespace, job, controller)`; +`OperatorLeaderMissing` keys on the lease name, which is unique within a namespace. + +| Alert | Fires when | Threshold | `for` | +| -------------------------------- | ---------------------------------------------- | -------------------------------------------------------------------------------- | ----- | +| `ControllerReconcileErrors` | a controller's reconciles mostly return errors | `result="error"` share of `controller_runtime_reconcile_total` over 10m `> 0.25` | 15m | +| `ControllerReconcilePanics` | a reconcile panicked | `increase(controller_runtime_reconcile_panics_total[10m]) > 0` | none | +| `ControllerWorkqueueBacklog` | items wait too long for a worker | p99 of `workqueue_queue_duration_seconds` over 10m `> 100` seconds | 15m | +| `ControllerReconcileLatencyHigh` | reconciles are slow | p99 of `controller_runtime_reconcile_time_seconds` over 10m `> 30` seconds | 15m | +| `OperatorLeaderMissing` | no replica holds the leader lease | `max by (namespace, name) (leader_election_master_status) == 0` | 5m | + +The thresholds are ratios and quantiles rather than absolute rates for the same reason as above: they hold at any scale. +The backlog rule uses queue wait time rather than queue depth, because no depth is right for every operator, whereas +items waiting minutes for a worker is wrong at any scale. + +Both quantile thresholds sit on histogram bucket bounds so that they mean what they say. controller-runtime's reconcile +time histogram has 60 seconds as its largest finite bucket, and `histogram_quantile` never returns more than the last +finite bound, so a threshold of 60 or above could never fire; 30 is the highest bound that leaves room above it. The +workqueue histogram has one bucket per decade, so `> 100` means more than one percent of items waited longer than 100 +seconds, and the p99 value reported in the notification is interpolated within that bucket. Move these thresholds only +to another bucket bound: the reconcile histogram's bounds from ten seconds up are 10, 15, 20, 25, 30, 40, 50 and 60, and +the workqueue histogram's are 1, 10, 100 and 1000. A threshold between two bounds, such as 45, fires exactly like the +bound below it and only reads as if it were stricter. + +`OperatorLeaderMissing` is silent in two cases: when leader election is off, because the gauge is not exported at all, +and when no replica is alive to export it, for example a crash loop before the elector starts. Pair it with your +platform's target-down alert for the operator's scrape job. + +### Conditions + +Per operator, rendered as `-crd-conditions`. The metric value of +`_controller_condition` is the condition's `lastTransitionTime`, which the rules rely on. + +| Alert | Fires when | Threshold | `for` | +| -------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----- | +| `CustomResourceNotReady` | an owner's `Ready` condition is `False` | `max by (controller, kind, name, )` of `condition="Ready", status="False"` | 30m | +| `CustomResourceConditionUnknown` | any condition of an owner is `Unknown` | `max by (controller, kind, name, condition, )` of `status="Unknown"` | 30m | +| `CustomResourceConditionStuck` | an owner's `Ready` condition has not been `True` for six hours | `time() - max(...)` of `condition="Ready", status!="True"` `> 21600` | none | + +Every rule aggregates with `max()` instead of matching series directly, and that is load bearing twice over. The +aggregation drops the `reason`, `status` and `id` labels, so a controller that keeps changing the reason while an owner +stays unhealthy does not restart the `for:` clock every time. And `max()` keeps the freshest `lastTransitionTime`, so +two series for one owner, such as a reason change still inside the lookback window, collapse to the freshest rather than +adding up the way `sum()` would. + +The status matcher on its own would still fire on a former leader pod's stale series (see [Stale series](#stale-series) +below), so before applying it every rule joins on the freshest series per owner across every status, with the same +`and topk by (...) (1, ...)` join the dashboards use. A stale `False` or `Unknown` series loses that join as soon as the +current leader exports a later `lastTransitionTime` for the owner, whatever its status. + +`CustomResourceNotReady` and `CustomResourceConditionStuck` are scoped to `Ready` on purpose. Matching `status="False"` +across every condition type would fire forever on negative-polarity conditions such as `Degraded`, where `False` is the +healthy state. To cover your own positive-polarity conditions, widen the `condition` matcher in the rendered file, for +example `condition=~"Ready|CertificateReady"`. The stuck rule keeps `condition` in its `by` clause, so that edit alone +gives one alert per owner and condition. `CustomResourceNotReady` aggregates without `condition`, so add it to the `by` +clause as well if you want a separate alert per condition rather than one per owner. `CustomResourceConditionUnknown` is +not scoped, because `Unknown` is bad whatever the polarity. + +`CustomResourceConditionStuck` has no `for:` clause because its expression is itself a duration comparison. A `for:` +clause measures how long the alert has been true, which is bounded by how long the series has been continuously present; +a scrape gap, an operator restart or a ruler restart silently restarts that clock. The stuck rule measures how long the +owner has been in its state according to its own status, so it survives all three and reports the real age. It also +covers `Unknown`, which `CustomResourceNotReady` does not. Tune the `21600` (six hours) to the longest time an owner of +yours can legitimately take to become ready. + +Each condition alert carries a `dashboard_url` annotation that deep-links into the CRD Conditions Browser rendered for +the same metric namespace, narrowed to the one owner. The `id` label is `/`, or `/` for a +cluster-scoped owner, so the link works without a conditional. + +## Dashboards + +Both dashboards are Grafana JSON at schema version 41 with a `datasource` variable, auto-refresh off, and the `ocf` tag. +A dashboard link at the top of each lists every dashboard carrying that tag, which is how they cross-reference each +other regardless of the folder or sub-path Grafana serves them from. + +The uids are templated with the metric namespace because Grafana upserts dashboards by uid: with fixed uids, importing a +second operator's render into a shared Grafana would overwrite the first. Two operators in one Grafana therefore get +`alpha_ocf_operator` and `beta_ocf_operator`, and each operator's condition alerts link to its own browser. + +### OCF Operator + +Variables, in cascade: `namespace` (Operator namespace), `job` (Operator, the scrape job) and `controller` (multi, All). +Every controller-runtime operator on a cluster exports the same metric names, so nothing in `controller_runtime_*`, +`workqueue_*`, `rest_client_*`, `process_*` or the apply counters says which operator a series belongs to; the scrape +labels do. `namespace` is the namespace the operator pod runs in and `job` its scrape job, usually the name of the +metrics Service or ServiceMonitor. Together they select one install of one operator, the same pair the shared alert +rules aggregate by, so two installs of the same operator in different namespaces stay apart. `namespace` defaults to +All, which also matches series that carry no namespace label at all, so an operator scraped outside a cluster still +renders. `namespace` scopes the controller-runtime, workqueue, apply and process panels, whose `namespace` label is +always the pod's; the condition panels are scoped by `job` alone, because on condition series that label is the CR's +namespace when rendered with `NAMESPACE_LABEL=namespace`. `controller` narrows within that operator, see +[Naming the controller](#naming-the-controller). Panel titles say CR for what the framework calls the owner: one custom +resource instance the controller reconciles, identified by its kind, namespace and name. Rows are ordered by what an +on-call reader asks first: is the operator healthy right now, are the owners healthy, is anything being rewritten or +failing, then the controller's own reconciliation and workqueue internals, the API client, and the process. + +| Row | What it answers | +| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Overview | Is the operator healthy right now? Four stat tiles with sparklines over the dashboard range: reconciles per second, reconcile error ratio, p99 reconcile time and p99 queue wait. Below them, owner counts on one panel, Ready, not Ready for more than two minutes (debounced on `lastTransitionTime`, so a rollout in progress does not count) and Unknown, then leader status and active against max workers. All three owner counts are scoped to the `Ready` condition; unlike `CustomResourceConditionUnknown`, the Unknown count does not cover other conditions. | +| Conditions | Which CRs are unhealthy? CRs by Ready status over time, stacked True, False and Unknown so a rollout or an outage shows as a band; a bar gauge of unhealthy conditions counting owners per kind, condition and status for every condition that is not True (empty when everything is healthy, and independent of how many kinds or condition types the operator has); and a full width CRs not Ready table (kind, namespace, name, reason, since, status) whose rows link into the CRD Conditions Browser filtered to that owner. | +| Managed resources | Is anything being rewritten or failing? Apply rate by operation with the error rate on the same panel, the `updated` rate per resource over time with a legend table sorted by last value so the worst offender is on top, the not-converging ratio per resource, and the apply error ratio per resource. The not-converging panel plots the `ManagedResourceNotConverging` expression without its floor, so a resource heading for the alert is visible before it fires; the error ratio's denominator falls back to the error rate alone when no success series exists, as the alert does. | +| Reconciliation | Where does reconcile time go? Reconcile rate by result, error ratio over time, latency at p50, p90 and p99, panics, the age of the longest in-progress reconcile (`workqueue_longest_running_processor_seconds`), and workers. | +| Workqueue | Is the operator keeping up? Depth, adds per second, queue wait p99, work duration p99, retries per second, unfinished work. | +| API client | Is the API server pushing back? `rest_client_requests_total` rate by method and non-2xx responses by code. | +| Process | Collapsed. CPU, resident memory and goroutines for the `job`. The CPU and memory panels draw the container request and limit as dashed lines when kube-state-metrics is scraped, joining `kube_pod_container_resource_requests` and `_limits` to the operator's scrape target on its `namespace` and `pod` labels; a pod with several containers shows its largest. | + +Reason strings are operator-specific, so colour is driven by `status` (`True` green, `False` red, `Unknown` yellow) and +reasons appear as text. + +### CRD Conditions Browser + +The browser answers "which owners of this kind are in this state, and since when". Its variables narrow each other from +left to right: `kind` (single choice), then `condition`, `status`, `reason`, `namespace` and `resource_id`, all +multi-select with All, plus ad hoc filters. The Operator Conditions row shows the count of matching conditions and a +table of them by name, namespace, condition, status and reason with a since column; the collapsed Status Counts row +breaks the count down by `False`, `Unknown` and `True`. + +`resource_id` is what the alerts drill into: a `CustomResourceNotReady` notification opens the browser with `kind`, +`condition`, `status` and `resource_id` preset, and the CRs not Ready table on the operator dashboard does the same for +the row you click. + +Every multi variable answers All with the regular expression `.*` rather than a list of every value. That keeps the +query size constant however many owners exist, and it keeps cluster-scoped owners visible: their condition series carry +no namespace label, and a `=~".*"` matcher on an absent label matches, whereas a list of observed namespaces would not. + +### Stale series + +The condition gauge is exported by whichever pod recorded it. After a leader change the former leader keeps exporting +its last values until it restarts, so for a while two series describe one owner, and a plain `count()` double counts. +Every condition query in both dashboards joins on the freshest series per owner: + +```promql + and topk by (kind, id) (1, ) +``` + +The metric value is the `lastTransitionTime`, so `topk` keeps the most recently transitioned series for each owner and +drops the stale duplicate. The join carries `kind` because `id` is only `/`, and two owners of +different kinds can share it. Queries that span more than one condition type, such as the Unhealthy conditions panel, +join on `topk by (kind, id, condition)` instead, so each owner keeps one freshest series per condition rather than one +overall. The browser pins `kind` through its variable, so its queries join on `topk by (id, condition)`. The condition +alerts apply the same join before their status matcher, keyed on `(job, controller, kind, name, )` and, +for the rules that keep `condition` in their `by` clause, `condition`. `job` is in every grouping so two installs +exporting the same metric namespace never dedupe or merge across each other, and the browser link in each notification +carries it. `job` is the whole install identity these rules need: two installs only collide on a series when both export +the same CR (same controller, kind, namespace and name), which means both are reconciling the same object, a deployment +the framework does not support rather than one to alert on. + +#### Known limitation: equal timestamps + +A reason-only update keeps `lastTransitionTime` (`meta.SetStatusCondition` preserves it while the status is unchanged), +so after a leader change a former leader that is still being scraped can export the previous reason with exactly the +same value as the current leader's series. The alerts are unaffected, because every rule drops `reason` before it +aggregates and a tie implies the same status. The dashboards' reason-filtered panels can pick either series for as long +as both are scraped. The window is short: controller-runtime stops the manager when it loses the lease, so the old pod +restarts and its series disappear within a scrape interval or two. Dedupe on a leadership signal would close it, at the +cost of a fallback for operators that run without leader election, and is not done. + +## Previewing locally + +A clone of the repository can bring up Prometheus and Grafana with simulated operator data behind them, so you can look +at every panel and every alert before installing anything in a cluster. You need docker with the compose plugin and Go. + +```bash +make observability-up +``` + +Grafana serves on `http://localhost:3000` with anonymous admin access and both dashboards provisioned; Prometheus serves +on `http://localhost:9090`, with the alerts on `http://localhost:9090/alerts`. The simulator plays a scripted world in +which every panel is populated and every alert fires within a few minutes (`OperatorLeaderMissing` only with +`make observability-up SIMULATOR_ARGS="-leader=false"`). The simulator also serves kube-state-metrics lookalikes for its +own pod's requests and limits on `/ksm/metrics`, scraped as a separate `kube-state-metrics` job, so the Process panels +show their dashed lines. Stop the simulator with Ctrl-C, then remove the containers: + +```bash +make observability-down +``` + +Maintainers find the stack's internals, the scripted world and the tests that guard the templates in +`observability/README.md` in the repository.