From bf1fdd2974063fa19bb17b0e6113a3f38fb5f526 Mon Sep 17 00:00:00 2001 From: Erik Darling <2136037+erikdarlingdata@users.noreply.github.com> Date: Fri, 11 Sep 2026 19:45:19 -0400 Subject: [PATCH] Add the #3282 CHANGELOG entry for the High CPU persistence gate --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 66e84d9d0..7adfe6b03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **High CPU now requires the condition to persist** ([#3282]) - the built-in High CPU alerts on both engines fired on a single sample over the threshold and resolved on the next sample under it, so a momentary spike was indistinguishable from sustained saturation; measured fire/resolve pairs on a 42-server fleet ran 42-147 seconds apart. Both evaluators now go through the shared `AlertPersistenceGate`: three consecutive breaching CPU samples to fire, two consecutive clearing samples to resolve, counted per collected SAMPLE rather than per alert sweep so a re-read of one reading cannot fill the streak. The per-server state is persisted on both SKUs, so a restart neither re-announces an already-open incident nor loses a partly-built streak, and a CPU reading that stops arriving now freezes the gate instead of announcing a recovery nobody measured. - **A mute rule created or deleted outside the Viewer did not take effect, for an unbounded time, and every surface said it had** ([#3315]) - `config.config_mute_rules` carried no `config_version` bump trigger, and the alert engine consults `MuteRuleService`'s in-memory cache, which is reloaded only when that beacon changes. So `create_mute_rule` returned `created`, the row really landed, and matching alerts kept being delivered until an unrelated config write or a service restart happened to force a reload - with `get_mute_rules` listing the rule the whole time, because it reads the table and not the cache. `delete_mute_rule` had the same gap in the more dangerous direction: an operator un-mutes, believes alerting is restored, and the stale cache keeps suppressing. Storage V117 adds `trg_bump_mute_rules`, the fifth instance of V17's statement-level beacon trigger and sharing its function verbatim, so ANY write to the table makes the service reload its mute cache on the next sweep - `AFTER INSERT OR UPDATE OR DELETE`, so disabling a rule counts as well as removing it. The `mcp` role needs no new grant: the column-level `UPDATE (config_version, updated_at)` on `config_service` provisioned for the `config_alert_settings` trigger is exactly what this SECURITY INVOKER trigger uses, verified against a live store in both directions. The viewer's connect-time probe gains a `pg_trigger` sentinel for the rung rather than an `information_schema.triggers` one, which lists only triggers on relations the current role owns or holds a non-SELECT privilege on and so would report the rung absent to the read-only `viewer` seat on a store that is fully migrated. - **Webhook and email alerts no longer re-render incidents that are still inside their own cooldown window** ([#3313]) - a batch posted when any one of its incident fingerprints was outside its #1154 cooldown window and then rendered every incident it carried, so a single new fingerprint re-delivered every co-resident fingerprint that had already gone out minutes earlier. Teams, Slack, PagerDuty, the generic webhook and email now render only the incidents actually outside their own window, with a one-line footer stating how many others are still open. The alert history row continues to record every incident, so nothing the MCP reader, the triage page, the Viewer's detail pane or the mute pre-fill reads has changed. Per-event delivery already sent one incident per message and is unchanged. PagerDuty's `dedup_key` now anchors on the first DELIVERABLE incident, which corrects a correlation defect: a batch of one in-window and one fresh incident previously triggered under the in-window incident's key and folded into its existing alert, so the genuinely new incident never surfaced there at all. - **A Collector Cost Regression self-alert now has to be worth reporting, not merely real** ([#3316]) - the predicate floored the average daily TOTAL cost and then compared cost PER RUN, and a total-cost floor is cleared by volume, so it never constrained the quantity the ratio tested: a collector averaging 3 ms per run cleared the 1,000 ms/day floor on run count alone and was then judged by a 2x ratio on that 3 ms. Those firings were truthful rather than noisy - both sides of the ratio are means over many runs, so rounding averages out - and still unactionable, the measured case doubling 3.0 to 6.1 ms per run across 50 runs to add 0.16 s of collection time a day. A third gate now asks what the regression COSTS, the per-run rise times the volume it is paid on, which is unit-consistent with the ratio and unlike a minimum per-run baseline still reports a 3 ms collector that runs often enough for the rise to matter. The floor is derived from a measured fleet sample where real regressions added 21.8-22.6 s a day and unactionable ones 0.16-3.6 s, and the alert now states the added cost that justified it. @@ -3658,3 +3659,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 [#3316]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/3316 [#3315]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/3315 [#3313]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/3313 +[#3282]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/3282