From 3108ac62749bdf83cb78b461d57972d16fe68a60 Mon Sep 17 00:00:00 2001 From: Bartek Tofel Date: Fri, 11 Sep 2026 16:37:14 +0200 Subject: [PATCH] chore: adjust grafana alertcheck docs --- grafana-alertcheck/docs/advanced.md | 6 +++--- grafana-alertcheck/docs/architecture.md | 6 +++--- grafana-alertcheck/docs/how-alerts-are-evaluated.md | 2 +- grafana-alertcheck/docs/index.md | 4 +++- 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/grafana-alertcheck/docs/advanced.md b/grafana-alertcheck/docs/advanced.md index 073292cd6..445738268 100644 --- a/grafana-alertcheck/docs/advanced.md +++ b/grafana-alertcheck/docs/advanced.md @@ -10,7 +10,7 @@ description: Why grafana-alertcheck schedules per rule, how the request budget w ## Per-rule schedules, never a global cycle -Each rule polls at its **own** cadence, `--poll-interval` (default: half the rule's own evaluation interval). There is deliberately no single global minimum-interval cycle. +Each rule polls at its **own** cadence (default: half the rule's own evaluation interval). There is deliberately no single global minimum-interval cycle. Overwrite with `--poll-interval`. One rule at `intervalSeconds=10` beside twenty at `300` keeps a 5 s cadence for itself and 150 s for the other twenty — not a 5 s cycle for all of them, which would be a 60× request bloat at ~1.8 s per request and would fail to start on a reasonable fleet. @@ -18,7 +18,7 @@ The scheduler staggers each rule's initial next-due time across its cadence, and ## The check budget -The gate records one observation of every rule up front and checks the schedule against those **measured** latencies (payload sizes vary ~230× across rules, so a fixed estimate is meaningless). It errors at start — before waiting — if any of three conditions hold: +The gate records one observation of every rule up front and checks the schedule against those **measured** latencies (payload sizes varied ~230× across existing rules, so a fixed estimate would be meaningless). It errors at start — before waiting — if any of three conditions hold: - **Utilization** — total request rate exceeds `--concurrency`. - **Per-rule** — one rule's request can't fit its own cadence. @@ -38,4 +38,4 @@ Querying Grafana's alert state history after the fact fails closed *in the wrong Instead, `watch` records its own evidence live and the log becomes the source of truth. The trade-off: the gate can miss an episode shorter than a rule's poll interval, though `activeAt` still surfaces sub-interval onsets for instances still active at a poll. -A corollary of recording fresh: there is no replay. Re-running a failed job is a new deploy with a new `from` and a new recording — never a re-classification of old evidence. +A corollary of recording fresh: there is no replay. Re-running a failed job is a new piece of work with a new `from` and a new recording — never a re-classification of old evidence. diff --git a/grafana-alertcheck/docs/architecture.md b/grafana-alertcheck/docs/architecture.md index ca493220e..455053163 100644 --- a/grafana-alertcheck/docs/architecture.md +++ b/grafana-alertcheck/docs/architecture.md @@ -12,15 +12,15 @@ This page documents the invariants and seams a maintainer must not break. It exi ## Fail-closed invariants -The gate must stop the release if it cannot get an answer. Every rule below is a specific instance of that: +The gate must fail if it cannot get an answer. Every rule below is a specific instance of that: - **An error is never a pass.** A pass is exactly `len(Violations) == 0 && err == nil`. Every error path leaves `err` non-nil, and the CLI maps that to exit `2` unconditionally. - **Inability beats violation.** Any `unobservable` rule is exit `2`, even alongside a real violation found first. - **Absent never means normal.** An instance that leaves the bad set is looked up in the *same* response: present as `normal` → cleared; absent (or `MissingSeries`) → vanished (a discontinuity, not a recovery). -- **Staleness is absolute.** `grafana_now − lastEvaluation` is compared against a threshold, never "did it increase since the last poll" — a delta check reports stale on ~half the polls of a healthy rule. +- **Staleness is absolute.** `grafana_now − lastEvaluation` is compared against a threshold, never "did it increase since the last poll" — a delta check reports stale on ~half the polls of a healthy rule (we poll at half of `intervalSeconds` of each rule). - **`grafana_now` is the response `Date` header.** Never the runner clock, in any comparison against a Grafana timestamp. - **No early exit.** `check` collects to `to + transitionGrace` before classifying once. -- **No replay.** No run-id key, no artifact download, no state between attempts. A retry is a new deploy. +- **No replay.** No run-id key, no artifact download, no state between attempts. A retry is a new piece of work and observation. ## The pure-function seam diff --git a/grafana-alertcheck/docs/how-alerts-are-evaluated.md b/grafana-alertcheck/docs/how-alerts-are-evaluated.md index 7c9cb3b6a..9e09d4cbc 100644 --- a/grafana-alertcheck/docs/how-alerts-are-evaluated.md +++ b/grafana-alertcheck/docs/how-alerts-are-evaluated.md @@ -40,7 +40,7 @@ For each instance the gate builds a timeline of bad spans over `[from, to]`, the | `skipped` | Paused **before** the window opened | reported, not observable | | `unobservable` | Coverage gap / sustained `health=error` / stale / absent | → 2 | -`recovered` has **no deadline** — an alert that clears at minute 58 of a 60-minute window still passes. The total bad time is reported as `BadFor`; the removed deadline is replaced by that measured value rather than a derived limit. +`recovered` has **no deadline** — an alert that clears at minute 58 of a 60-minute window still passes. The total bad time is reported as `BadFor`. ### Preexisting policy diff --git a/grafana-alertcheck/docs/index.md b/grafana-alertcheck/docs/index.md index 95d413772..e8ec76928 100644 --- a/grafana-alertcheck/docs/index.md +++ b/grafana-alertcheck/docs/index.md @@ -35,6 +35,8 @@ export GRAFANA_TOKEN=… Requires Grafana >= 13.0.0 and < 14.0.0. Outside that range the gate exits `2`. +Grafana API token needs to have `fixed:alerting:reader` permissions. Ask the o11y team for your token. + ## Quickstart — recorder mode ```bash @@ -74,7 +76,7 @@ An error is never a pass: `2` wins over any violation found alongside it. - A fix that **stops emitting a metric is not a recovery** — the instance vanishes, which is a discontinuity, not health. - The gate checks alert **state and health**, not notification delivery — a silenced alert that still fires fails. - `recovered` has **no deadline** — a bad-at-`from` alert that clears by `to` passes; set `--preexisting fail` to forbid it. -- A **retry is a new deploy**, not a replay — re-running the job re-records against a new `from`. +- If you retry the check, then the work also needs to be retried - there is no way to check the past. - `watch` and `check` must run in **one job, one runner, one filesystem** — nothing persists across jobs or attempts. - The gate **never exits early** — a violation at minute 2 still holds the runner to `to + transitionGrace + drainTimeout`; size the job timeout to the planned run time the gate prints at start.