Route every severity a rule claims to a contact point that exists - #242
Merged
Conversation
Every rule under dashboards/base/alerting/ labels itself `severity: page` or `severity: ticket`. Grafana routes a firing alert by matching those labels against a notification policy tree, and the catalog declared no GrafanaNotificationPolicy and no GrafanaContactPoint — so all 27 rules across 7 groups fell through to the workspace default policy and its empty default contact point. A rule labelled to page is asking for a human to be woken. Unrouted, the promise is kept by nobody, and the failure is silent in exactly the moment it matters: OperatorMetricsAbsent fires when the metrics path is dead, which is the condition under which nobody is looking at a dashboard. ──────────────────────── The routing tree ──────────────────────── A GrafanaNotificationPolicy matching `severity` to two contact points, one per urgency, carrying the same instanceSelector the rule groups use. The root receiver is the low-urgency one rather than a dedicated fallback. An alert whose severity matches no route is a rule the tree has not been taught about, and the two answers to that are "deliver it somewhere a person reads" and "drop it silently". The gate below makes that case impossible to arrive at unnoticed; the root route makes it survivable if it does. Grouped by service and component rather than by rule, so a fleet-wide failure raising six rules against one service is one notification. The page repeats hourly while firing; the ticket repeats every twelve, because a ticket that re-notifies like a page teaches people to mute the channel both arrive in. Credentials arrive the way the Grafana service-account token's do: an ExternalSecret over the aws-secrets-manager ClusterSecretStore, with the remote key patched per cluster by the dashboards ApplicationSet from the cluster_name label — the secret is cluster-scoped so co-located siblings in one account do not collide. Neither receiver's secret is marked optional. An optional missing secret produces a contact point with no integration key: Grafana accepts it, the route resolves, and nothing is delivered — indistinguishable from a working destination by every signal except an incident nobody answered. Required, the contact point reports NotReady and says so. ──────────────────────── The gate ──────────────────────── scripts/check-alert-severity-routes.py, wired into `task validate` and CI. The routing keys come from the POLICY, not from a list: whatever label keys the tree matches on are the keys that make a claim about delivery. Then every value a rule carries for one of those keys must match a route for that key; every rule must carry every key the policy routes on, because a rule missing one lands in the root bucket by a different path; and every receiver a route names must be a contact point this catalog declares, in both directions — a route to an undeclared receiver delivers nothing, and a contact point no route names is a destination that rots the way an unread exemption does. Regex matchers count as a route for their key and vouch for no particular value. Deciding which values a pattern admits means running the pattern, and reporting a value as routed because it looked like it might match would be asserting the thing this gate exists to check. It states its own scope. Two facts must also hold before a page arrives and neither is readable from this tree: whether Amazon Managed Grafana ACCEPTS the contact point, which its workspace configuration decides, and whether the credential is seeded, which Secrets Manager holds. A green run says the catalog routes every urgency it claims. It does not say a page arrived. ──────────────────────── What proves it ──────────────────────── A positive control in scripts/tests/controls.py relabels one rule with a severity the tree does not route — the rule still parses, still evaluates, still changes state in the alert list, and reaches the root receiver instead of the destination it asked for. Eleven mutants, each killed and each naming a test: the keys hardcoded to `severity` instead of read from the policy; only top-level routes treated as delivery decisions; a regex matcher vouching for its pattern as a value; a missing label, an unrouted value, an undeclared receiver and an unreferenced contact point each not a finding; no policy, two policies, and a policy matching on nothing each accepted; and an empty rule corpus reading as a routed catalog. Two probes in scripts/tests/reverify-gates.sh, floor 35 → 38, one per direction: a severity the tree does not route, and a route to a receiver nobody declares. ──────────── Two things the repository's own gates caught ──────────── The ExternalSecret's first filename tripped the `*credentials*` ignore rule that keeps secrets out of git — renamed rather than force-added. check-externalsecret-keys.py then rejected it for holding an unpatched remote key, which is a real per-cluster requirement: the committed name is a placeholder, and unpatched it resolves to a secret that exists in no account. The gate's own refusal for a missing directory composed its message with `relative_to`, which RAISES for a path outside the repository. A refusal that crashes while writing its own message exits with a traceback and the status of a rejection, which is the reading these exit codes exist to keep apart. `under_root` returns the path as given when it is outside, and the test that found it asserts the refusal rather than the message. ──────────────────────── Verification ──────────────────────── ruff clean; mypy clean over 48 files; 493 tests across 19 modules; coverage ratchet 42.6% (floor 40 → 42), 14 per-gate floors held, this gate at 97% against a floor of 95; controls.py 18 controls; empty-corpus.py 30 probed gates; reverify-gates.sh 41/41; task validate exit 0; check-named-things.py 195 references resolve; yamllint clean; check-workflows.sh clean at MEDIUM and above. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
`mutateDigest: true` is the whole reason the enforcing tier enforces. Verification resolves a tag to a digest at admission; the kubelet resolves the same tag again at pull time. Pinning at admission is what stops anyone who can move a tag between those two moments from running an unverified image under a policy reporting success. In this repository that guarantee was a field value other files read, never a behaviour anything ran. Every Kyverno suite loads a BASE policy, where the field is false because Kyverno rejects true under an Audit failure action — so the only rendition carrying it was under no test at all. check-image-verification.py declines to assert the field and delegates the pairing; check-policy-validity.py asserts Enforce implies true and then applies a probe pod whose image is outside `ghcr.io/nanohype/*`, so the rule never matches and the mutation path never runs. ──────────────────────── What now runs ──────────────────────── scripts/check-digest-rewrite.sh renders each ENFORCING overlay and admits a pod through it — a real signed `ghcr.io/nanohype/*` release, referenced by TAG, because a digest-pinned fixture needs no rewrite and would satisfy every check against a policy that had stopped doing it. That fixture property is asserted rather than assumed. The overlays are enumerated from the tree, not named in the script. An overlay added later is one more tier depending on a step, and it has to be executed too. It reaches the registry for the manifest and Rekor for the transparency-log entry, which is the round trip admission makes — this exercises the path rather than a model of it. A verification that could not REACH those is told apart from a rewrite that did not happen: the first exits 2, because it observed nothing. ──────────── How the result is observed, and why by report ──────────── The Kyverno CLI offers no machine-readable channel carrying the object a verifyImages mutation admitted. Three were tried and each is recorded in the script. `kyverno test`'s `patchedResources` is compared against EVERY engine response for the rule, and a verifyImages rule produces two — one holding the resource as submitted, one holding it mutated — so one comparison always diffs whichever file is supplied, and supplying both produces a cross product with two failures. `kyverno apply -o` writes a file for mutate rules and leaves it empty for this one. The JSON policy report carries the verdict and the message and not the object. So the reading is the CLI's own detailed report. The expected patch supplied to it is the pod AS SUBMITTED, and that direction is what makes the reading non-circular: the mutated response differs from it, the report prints what the engine produced, and a digest in that output can only have come from the engine. An expectation carrying the digest would have the report echo the fixture back. That is text, and text is the weaker reading — but it fails in the safe direction. The assertion is that one reference token appears, so a report that stops printing it, wraps it, or renames the column makes the token absent and this gate red. It cannot go quiet. ──────────────────────── What it does not establish ──────────────────────── That a cluster admits what the CLI admits. The engine is the same code and the policy is the rendered one, and no environment reachable from this repository runs an API server. docs/threat-model.md now says that in those words, in place of a paragraph that had been mangled by an earlier edit into a duplicated fragment and an orphaned sentence. ci.yml carried a claim this work disproves — that `kyverno test` cannot reach Fulcio/Rekor and never verifies a signature. It does; the reason the suites never verified one is that they load base policies whose rule the fixtures do not match. Corrected rather than left standing beside a step that contradicts it. ──────────────────────── What proves it ──────────────────────── Both directions on the real tree. Unmutated: two enforcing overlays each admit the tagged pod carrying the digest verification resolved, exit 0. With the production overlay's patch flipped to `mutateDigest: false` — a change every YAML-reading gate still passes, because the field is simply gone rather than wrong — this exits 1 naming the overlay and what the tag then means. That is a probe in scripts/tests/reverify-gates.sh, floor 38 → 40, and the harness runs 43 checks. Registered in controls.py's NEEDS_NETWORK_SH rather than carrying a positive control: the round trip IS what it executes, and a control for it offline would be a control for a verification that did not happen. ──────────────────────── Verification ──────────────────────── ruff clean; mypy clean over 48 files; 493 tests across 19 modules; controls.py 18 controls; empty-corpus.py 31 probed gates; reverify-gates.sh 43/43; task validate exit 0; check-named-things.py 196 references resolve; yamllint clean; check-workflows.sh clean at MEDIUM and above. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
It landed here by mistake and belongs to its own issue and its own branch, where it is now — execute-the-digest-rewrite, cherry-picked onto main with the reverify floor set for that branch's probe count. Reverted rather than rewritten, because this branch is pushed and carries an open pull request: a force-push would move commits a reviewer may already have read. The pair nets to no change at squash, and the tree here is identical to the one before the mistake. Co-authored-by: stxkxsbot <275011021+stxkxsbot@users.noreply.github.com>
CI Results
All checks passed. |
Brings in the digest-rewrite gate. One collision, in the reverify harness's floor on how many checks it ran. ─── scripts/tests/reverify-gates.sh ─── Both sides raised MIN_CHECKS off the same base of 35, each by the number of `run` lines it added: the routing work to 38 for three, the digest work to 37 for two. Neither number is right for a tree holding both. Resolved to 40. The floor tracks the harness's own size — it sits three under the `run` count on both parents and on the base, which is the margin that lets a check be retired without the harness turning red on the next run while still catching a harness that stopped executing most of itself. The merged file carries 43 `run` lines. Taking either side verbatim would have been a floor a truncated harness clears: at 37, three of the checks this tree adds could stop running and the floor would still pass.
The routing gate read `dashboards/base/alerting/*.yaml`. A cluster receives what
`dashboards/base/kustomization.yaml` lists in `resources`, and that is an
explicit list a line can leave.
─── The defect ───
Delete `- alerting/notification-policy.yaml` from the kustomization and the
render carries seven GrafanaAlertRuleGroups, 27 rules labelled `severity: page`,
and no routing tree. Every alert falls to the workspace default policy and its
empty default contact point. The gate written to make exactly that impossible
reported the tree as fully routed, because the file it read was still on disk
and still described complete delivery.
Nothing else in the catalog sees it. `kustomize build` succeeds — a resource
list is shorter, not malformed. kubeconform SKIPs the grafana.integreatly.org
CRDs. No other gate reads a notification policy. The symptom is an alert that
changes state in the Grafana list and reaches nobody, which is indistinguishable
from an alert that has not fired.
The gate closed the case of a routing tree that is wrong. This is the case of a
routing tree that is not there, arriving by a different door.
─── scripts/check-alert-severity-routes.py ───
`shipped(root)` renders the kustomization and returns (kind, metadata.name) for
every alerting object the render carries. The root is `ALERTING.parent` rather
than a second literal, so the two readings cannot come to describe different
trees.
`main()` compares that against the same set read off disk, and reports both
directions:
* on disk and not rendered — routing this gate checked and no cluster
receives;
* rendered and not on disk — an alerting object the gate never opened. That
direction decides which tree Grafana obeys: a second GrafanaNotificationPolicy
entering the render from elsewhere under dashboards/base reconciles too, and
whichever lands last wins.
Both readings are kept rather than one replacing the other. The render decides
what a cluster gets, but a rendered document no longer carries the file it came
from, and "a rule is unrouted" across eight near-identical files is not
actionable without one.
A render that fails and a `kustomize` that is not installed are both exit 2. A
kustomization that does not build says nothing about what a cluster receives,
and the files on disk say nothing about it either — that is a gate that could
not run, not a tree it rejected.
─── scripts/tests/test_alert_severity_routes.py ───
`TheVerdict`'s planted fixtures now write a real kustomization and are rendered
rather than compared against a stub, so the cases that predate this change
exercise the render path as well. `policy()` takes a name because kustomize
rejects two resources sharing an id, and the two-policy case has to plant two
distinct ones to reach the gate.
`WhatShipsDecides` covers the new verdicts: each of the three object kinds left
out of `resources`, an alerting object rendered from outside the directory, a
render that fails, and that the second reading shells out rather than globbing
the same directory twice. That last one is what a gate faking this would pass.
`TheShippedRootIsTheOneDelivered` asserts over the tree that an ApplicationSet
delivers the root the gate renders — in the tests rather than in the gate,
because a root nobody delivers is a fleet-delivery question, not a routing one.
─── scripts/tests/reverify-gates.sh ───
A fourth planted defect for this gate: the notification policy dropped from the
kustomization's resources. Every earlier check still passes on that tree — the
rules are labelled, the routes match, the contact points are declared — which is
what makes it worth planting. MIN_CHECKS follows to 41 against 44 `run` lines.
Member
Author
|
Merged Then the question of whether the two gates agree on what their corpus is. They did not, and it was a live false green on this branch.
27 rules labelled This branch closed the case of a routing tree that is wrong. What stayed open was the tree not being there, by a different door: present in the directory the gate reads, absent from the list the cluster is built from.
Proof — four defects planted in the kustomization, each caught and each naming the file; then the tests by mutation with
The third is the one worth keeping: a gate globbing the same directory twice passes every routing test written before this round. Local run at |
CI Results
All checks passed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #222. A rule labelled
severity: pageis a promise that a human is woken. A route to no contact point means the promise is kept by nobody, and the failure is silent in exactly the moment it matters — nobody is watching a page that never arrives to notice it did not.Every label claiming a destination must resolve to one, derived from the routing tree rather than from a list of the rules someone checked.