Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions grafana-alertcheck/docs/advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ description: Why grafana-alertcheck schedules per rule, how the request budget w

## Per-rule schedules, never a global cycle

Each rule polls at its **own** cadence, `--poll-interval` (default: half the rule's own evaluation interval). There is deliberately no single global minimum-interval cycle.
Each rule polls at its **own** cadence (default: half the rule's own evaluation interval). There is deliberately no single global minimum-interval cycle. Overwrite with `--poll-interval`.

One rule at `intervalSeconds=10` beside twenty at `300` keeps a 5 s cadence for itself and 150 s for the other twenty — not a 5 s cycle for all of them, which would be a 60× request bloat at ~1.8 s per request and would fail to start on a reasonable fleet.

The scheduler staggers each rule's initial next-due time across its cadence, and serves due rules **earliest-due-first**, so a tight rule never queues behind slack ones.

## The check budget

The gate records one observation of every rule up front and checks the schedule against those **measured** latencies (payload sizes vary ~230× across rules, so a fixed estimate is meaningless). It errors at start — before waiting — if any of three conditions hold:
The gate records one observation of every rule up front and checks the schedule against those **measured** latencies (payload sizes varied ~230× across existing rules, so a fixed estimate would be meaningless). It errors at start — before waiting — if any of three conditions hold:

- **Utilization** — total request rate exceeds `--concurrency`.
- **Per-rule** — one rule's request can't fit its own cadence.
Expand All @@ -38,4 +38,4 @@ Querying Grafana's alert state history after the fact fails closed *in the wrong

Instead, `watch` records its own evidence live and the log becomes the source of truth. The trade-off: the gate can miss an episode shorter than a rule's poll interval, though `activeAt` still surfaces sub-interval onsets for instances still active at a poll.

A corollary of recording fresh: there is no replay. Re-running a failed job is a new deploy with a new `from` and a new recording — never a re-classification of old evidence.
A corollary of recording fresh: there is no replay. Re-running a failed job is a new piece of work with a new `from` and a new recording — never a re-classification of old evidence.
6 changes: 3 additions & 3 deletions grafana-alertcheck/docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ This page documents the invariants and seams a maintainer must not break. It exi

## Fail-closed invariants

The gate must stop the release if it cannot get an answer. Every rule below is a specific instance of that:
The gate must fail if it cannot get an answer. Every rule below is a specific instance of that:

- **An error is never a pass.** A pass is exactly `len(Violations) == 0 && err == nil`. Every error path leaves `err` non-nil, and the CLI maps that to exit `2` unconditionally.
- **Inability beats violation.** Any `unobservable` rule is exit `2`, even alongside a real violation found first.
- **Absent never means normal.** An instance that leaves the bad set is looked up in the *same* response: present as `normal` → cleared; absent (or `MissingSeries`) → vanished (a discontinuity, not a recovery).
- **Staleness is absolute.** `grafana_now − lastEvaluation` is compared against a threshold, never "did it increase since the last poll" — a delta check reports stale on ~half the polls of a healthy rule.
- **Staleness is absolute.** `grafana_now − lastEvaluation` is compared against a threshold, never "did it increase since the last poll" — a delta check reports stale on ~half the polls of a healthy rule (we poll at half of `intervalSeconds` of each rule).
- **`grafana_now` is the response `Date` header.** Never the runner clock, in any comparison against a Grafana timestamp.
- **No early exit.** `check` collects to `to + transitionGrace` before classifying once.
- **No replay.** No run-id key, no artifact download, no state between attempts. A retry is a new deploy.
- **No replay.** No run-id key, no artifact download, no state between attempts. A retry is a new piece of work and observation.

## The pure-function seam

Expand Down
2 changes: 1 addition & 1 deletion grafana-alertcheck/docs/how-alerts-are-evaluated.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ For each instance the gate builds a timeline of bad spans over `[from, to]`, the
| `skipped` | Paused **before** the window opened | reported, not observable |
| `unobservable` | Coverage gap / sustained `health=error` / stale / absent | → 2 |

`recovered` has **no deadline** — an alert that clears at minute 58 of a 60-minute window still passes. The total bad time is reported as `BadFor`; the removed deadline is replaced by that measured value rather than a derived limit.
`recovered` has **no deadline** — an alert that clears at minute 58 of a 60-minute window still passes. The total bad time is reported as `BadFor`.

### Preexisting policy

Expand Down
4 changes: 3 additions & 1 deletion grafana-alertcheck/docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ export GRAFANA_TOKEN=…

Requires Grafana >= 13.0.0 and < 14.0.0. Outside that range the gate exits `2`.

Grafana API token needs to have `fixed:alerting:reader` permissions. Ask the o11y team for your token.

## Quickstart — recorder mode

```bash
Expand Down Expand Up @@ -74,7 +76,7 @@ An error is never a pass: `2` wins over any violation found alongside it.
- A fix that **stops emitting a metric is not a recovery** — the instance vanishes, which is a discontinuity, not health.
- The gate checks alert **state and health**, not notification delivery — a silenced alert that still fires fails.
- `recovered` has **no deadline** — a bad-at-`from` alert that clears by `to` passes; set `--preexisting fail` to forbid it.
- A **retry is a new deploy**, not a replay — re-running the job re-records against a new `from`.
- If you retry the check, then the work also needs to be retried - there is no way to check the past.
- `watch` and `check` must run in **one job, one runner, one filesystem** — nothing persists across jobs or attempts.
- The gate **never exits early** — a violation at minute 2 still holds the runner to `to + transitionGrace + drainTimeout`; size the job timeout to the planned run time the gate prints at start.

Expand Down
Loading