Derive the burn-rate budget figure from the expression that spends it - #244
Conversation
A ticket-tier rule's summary claimed the whole error budget where its own
expression spends a tenth of it.
─── The defect ───
dashboards/base/alerting/agent-operator.yaml, the factor-1 / 3d tier:
summary: operator reconcile latency budget burning (100% over 3d)
Burn factor 1 spends the budget at exactly the rate that exhausts it over the
SLO window, so 3d of it consumes 3/30 = 10%. The expression is correct; only
the sentence is wrong, which is why every rule-level validation passed it. A
summary is a free-text annotation and nothing parsed it.
The tier carries severity: ticket, the least urgent the file ships, and its
title described total budget exhaustion — the most urgent condition an SLO has.
A summary is the alert TITLE: the sentence that reaches a human first and often
the only one they read. Escalated, it is an outage that is not happening.
Recognised as wrong, it teaches on-call that these titles are not to be trusted,
which spends the credibility of the three tiers that are right.
Fixed to `budget burning slowest (10% in 3d)`. The 1d and 3d tiers both consume
10% — 3 x 1d and 1 x 3d are the same spend — so the adverb separates them, not
the figure.
─── scripts/check-burn-rate-budgets.py ───
Fixing the string closes the instance. A figure that agrees with the standard
today and is compared to nothing is the same defect waiting, so the gate derives
it rather than listing it:
budget consumed = burn factor x long window / SLO window
Every term is read from a different place in the tree. The factor and the long
window come from the rule's own expression. The SLO window comes from the
dashboard panel measuring the same metric selector over its longest explicit
range — dashboards/base/platform/agent-operator.yaml computes the objective over
30d on the same controller_runtime_reconcile_time_seconds_bucket the rules burn
against, and portal and fleet-vend pair the same way. That panel is edited by
different work than an alert summary, so it cannot agree with a wrong figure by
construction.
Arithmetic is exact (fractions.Fraction), so the comparison is an equality
rather than a tolerance. A tolerance loose enough for the factors this catalog
uses is loose enough to miss a one-step drift.
Rules enter the corpus on their EXPRESSION, never on their summary. Selecting on
the sentence would let a rule leave the corpus by having its claim deleted, and
that is the edit most likely to accompany a wrong one — so a burn rule stating
no figure is a finding rather than a skip.
The burn factor is the comparison against the budget-normalised ratio,
`/ <budget> > bool <factor>`. Matching `> bool` alone also matches the traffic
guard four of these rules carry — `sum(rate(...)) > bool 0.0167`, roughly one
request a minute, there to stop an idle service alerting on a ratio computed
from no traffic. Read as a factor it makes four correct rules look like they
each state two contradictory rates of spend.
The dashboard walk is imported from check-athena-panel-columns.py rather than
copied, so the two gates cannot come to disagree about which dashboards exist.
─── Registration ───
CI runs it in the dashboards job beside the alert-coverage gate. controls.py
plants the original figure and requires a rejection naming the file;
reverify-gates.sh runs it green and plants the same defect, floor 37 -> 39
against 42 run lines. empty-corpus.py picks it up by discovery and classifies
its refusal as one about a derivation input rather than its corpus.
─── scripts/tests/run.py ───
MAX_UNCOVERED_GATES 11 -> 10, not 9. The run reports 9 of 25 against a baseline
of 10 of 24, but two things moved: this gate arrived with tests, and
check-athena-panel-columns.py flipped to covered because this gate imports it.
Only the first is coverage. The comment names the third import-covered file
beside the two it already named, and says the number moves by what a change
covered rather than by what the run reports.
─── .github/workflows/ci.yml ───
The alert-coverage step's comment carried its last sentence twice.
Brings in the severity-routing gate. Three collisions, all additive: each side added a block at the same insertion point and neither edited the other's. ─── .github/workflows/ci.yml, scripts/tests/controls.py ─── Both steps and both mutators kept, the routing ones first because they landed first. The two mutators plant into the same file — dashboards/base/alerting/ agent-operator.yaml — and do not interact: one relabels a rule's severity, the other rewrites a summary, and each control runs against its own tree. ─── scripts/tests/reverify-gates.sh ─── The four planted defects kept alongside each other, and the floor recomputed rather than picked. Both sides raised MIN_CHECKS off the same base of 37, each by the number of `run` lines it added: the routing work to 41 for four, the burn-rate work to 39 for two. The floor tracks the harness's own size and sits three under the `run` count on both parents and on the base, which is the margin that lets a check be retired without turning the harness red while still catching one that stopped executing most of itself. The merged file carries 46 `run` lines, so the floor is 43. Taking either side verbatim resolves it downward: at 41, two of the checks this merge adds could stop running and the floor would still pass. ─── scripts/tests/run.py ─── Auto-merged, and the two constants each side moved are disjoint. COMBINED_FLOOR is 42 from the routing branch and MAX_UNCOVERED_GATES is 10 from this one; both hold on the merged tree.
The burn-rate figure has no independent existence. It is not a number stored
anywhere and corrected when wrong: it is the product of three terms, each read
from a different place, and the summary is checked against what they produce.
A reader looking for the value will not find one to edit.
budget consumed = burn factor x long window / SLO window
That is the point of the gate, and the reason a drift now fails: there is no
longer a figure to drift FROM. A wrong summary disagrees with the expression
that produces it; a moved SLO window disagrees with the panel that measures it;
a panel that stops shipping anchors nothing and says so.
─── What the merge exposed ───
The routing gate that landed on main asks what a cluster RECEIVES: it renders
dashboards/base and refuses an alerting object present in the directory and
absent from the render. This gate's third term still asked what files EXIST —
the SLO window came from a walk over dashboards/**.yaml on disk.
Those are different questions, and the gap was reachable with one deleted line:
$ sed -i '/platform\/agent-operator.yaml/d' dashboards/base/kustomization.yaml
$ kustomize build dashboards/base | grep -c 'name: agent-operator$' -> 0
$ kustomize build dashboards/base | grep -c 'name: agent-operator-slo' -> 1
The rules ship and burn. The panel measuring the objective they burn against
does not. Every figure in the file is then anchored to a document no cluster
receives, and both gates report green.
─── scripts/check-burn-rate-budgets.py ───
`delivered_dashboards()` renders the kustomization and reads the GrafanaDashboard
JSON out of the render. KUSTOMIZE_ROOT is ALERT_DIR.parent — the same derivation
the routing gate uses, so the rules and the objective cannot be read out of
different trees. A dashboard on disk and not in `resources` anchors nothing, and
the existing "compared to nothing" verdict fires, reworded to say delivered.
A render that fails and a kustomize that is not installed are both exit 2: a
kustomization that does not build says nothing about which panels a cluster
receives, and the dashboards on disk say nothing about it either.
This drops the check-athena-panel-columns.py import, which existed to avoid a
second walk of the dashboards directory. There is no walk of that directory now.
─── The agreement is asserted, not assumed ───
The corpus here is still the alerting directory on disk. This gate does not take
the render reading itself, because the routing gate already refuses a rule group
the same root does not render — so the corpus is the delivered one by way of
that gate. test_the_rules_this_reads_are_rules_the_catalog_delivers asserts both
halves of what that leans on: the two gates render the same root, and
ALERTING_KINDS still contains GrafanaAlertRuleGroup. Narrow either and this gate
returns to reading files no cluster has, with nothing to say so.
─── scripts/tests/reverify-gates.sh ───
A second planted defect for this gate: the panel dropped from the kustomization.
The dashboard is unedited and still renders; it is simply not delivered. 47 `run`
lines, floor 44.
─── scripts/tests/run.py ───
MAX_UNCOVERED_GATES stays 10, and it now reads 10 of 26 rather than 9. Dropping
the import returned check-athena-panel-columns.py to uncovered, which is the
honest reading: it never had tests, only a module body executed by somebody
else's import. The comment no longer names it as import-covered.
|
Merged The mergeThree collisions, all additive — each side added a block at the same insertion point and neither edited the other's, so both are kept in The floor was recomputed rather than picked. Both sides raised What the merge did not settleThe routing gate asks what a cluster receives — it renders Different questions, and the gap was one deleted line wide: The rules ship and burn. The panel measuring the objective they burn against does not. Every figure in the file is then anchored to a document no cluster receives, and both gates certify it.
That drops the The agreement is asserted, not assumedThe corpus here is still the alerting directory on disk. This gate does not take the render reading itself, because the routing gate already refuses a rule group the same root does not render — so the corpus is the delivered one by way of that gate.
Against the tree: the panel dropped from The ceiling reads 10 of 26, not 9Dropping the import returned Verification at
|
CI Results
All checks passed. |
1 participant
Closes #199.
The figure has no independent existence
That is the change, and everything below is how it is arrived at. After this
there is no number in the tree holding the budget percentage — nothing to look
up, nothing to correct, and nothing in the summary that anything trusts. The
figure is the product of three terms:
each read from a different place, and the sentence in the alert title is checked
against what they produce. A reader looking for the value will not find one to
edit.
This is what makes a drift fail: there is no longer a figure to drift from.
A wrong summary disagrees with the expression that produces it. A moved SLO
window disagrees with the panel that measures it. A panel that stops being
delivered anchors nothing and says so, rather than falling back to whatever the
summary claims.
Correcting
100%to10%would have left the next wrong figure exactly assurvivable as this one was.
Where the 100 came from
dashboards/base/alerting/agent-operator.yaml, the factor-1 / 3d ticket tier. Burn factor 1 spends the error budget at exactly the rate that exhausts it over the SLO window, so 3d of it consumes 3/30 = 10%. The expression is right; only the sentence is wrong, which is why every rule-level validation passed it — a summary is a free-text annotation and nothing parsed it.The tier is
severity: ticket, the least urgent the file ships, and its title described total budget exhaustion. Fixed tobudget burning slowest (10% in 3d). The 1d and 3d tiers both consume 10% — 3×1d and 1×3d are the same spend — so the adverb separates them, not the figure.The second question
A constant that agrees with the standard today and is compared to nothing is the same defect waiting, so the figure is derived rather than listed:
/ <budget> > bool <factor>N% in|over Win the summaryThe SLO window is the term that decides, and it had to come from outside the alerting directory.
dashboards/base/platform/agent-operator.yamlcomputes the objective over[30d]on the samecontroller_runtime_reconcile_time_seconds_bucketthe rules burn against; portal and fleet-vend pair the same way. That panel is edited by different work than an alert summary, so it cannot agree with a wrong figure by construction.Verified across the ladder:
Exact rational arithmetic (
fractions.Fraction), so the comparison is an equality rather than a tolerance — a tolerance loose enough for these factors is loose enough to miss a one-step drift.Rules enter the corpus on their expression, never on their summary: selecting on the sentence would let a rule leave the corpus by having its claim deleted, and that is the edit most likely to accompany a wrong one. So a burn rule stating no figure is a finding, not a skip.
A regression the gate survived while being written
The first factor pattern matched
> boolanywhere, which also matched the traffic guard four of the eight rules carry —sum(rate(...)) > bool 0.0167, roughly one request a minute, there to stop an idle service alerting on a ratio computed from no traffic. Read as a burn factor it made four correct rules look like they each stated two contradictory rates of spend. The pattern now requires the budget division that defines a burn comparison, and a test plants the guard specifically.Proof
Seven defects planted, each caught and each naming the file:
100% over 3dThen the tests by mutation,
__pycache__cleared between each:2592000instead of readtest_the_objective_is_read_from_the_dashboards_not_declared_heretest_a_traffic_guard_is_not_a_burn_factor+ the catalog testtest_a_burn_rule_stating_no_figure_is_reportedtest_a_claim_compared_to_nothing_is_reportedtest_a_summary_naming_another_window_is_reportedThe first is the one that answers the property. A gate declaring
SLO_WINDOW = 30dwould pass every arithmetic case above and be the same defect one level up. The test asserts the source contains neither2592000nor"30d", and that it callspanels.dashboards().The uncovered-gate ratchet went to 10, not 9
The run reports 9 of 25 uncovered against a baseline of 10 of 24. Two things moved: this gate arrived with its own tests, and
check-athena-panel-columns.pyflipped from uncovered to covered because this gate imports it for the dashboard walk. Only the first is coverage, soMAX_UNCOVERED_GATESis 10 and the comment now names the third import-covered file beside the two it already named.Scope
The gate states what it does not cover: whether the objective is the right one, whether the factors implement the intended policy, whether any rule fires, the
descriptionprose under each summary — and agreement with the org SLO standard. The standard is the authority for these figures and is not in this repository, so what is asserted here is that the catalog agrees with itself. A window that moved in the standard and nowhere else would leave this green.Verification
ruff clean, mypy clean over 48 files, yamllint clean, zizmor unchanged (one pre-existing low at
ci.yml:428), 498 tests across 19 modules, coverage 43.5% against the 40% floor with 13 per-gate floors held,controls.py18 controls,empty-corpus.py31 probed gates,reverify-gates.sh42/42 against the floor of 39,check-named-things.py197 references resolve,task validateclean.