Skip to content

email_cooldown_minutes is the only throttle on Slack/Teams/PagerDuty delivery, but it is named for email, absent from get_alert_settings/update_alert_settings, and capped at 120 minutes #3314

Description

@erikdarlingdata

Summary

The only throttle between a fired alert and a Slack / Teams / PagerDuty / generic-webhook post is email_cooldown_minutes:

WebhookAlertService.cs:167   TimeSpan.FromMinutes(_settings.EmailCooldownMinutes)
EmailSendCore.cs:121         TimeSpan.FromMinutes(_settings.EmailCooldownMinutes)

Three problems, all operability rather than correctness:

1. It is named for the one channel an operator may not be using. A headless deployment with Slack configured and no SMTP at all is still governed entirely by a setting called email_cooldown_minutes, living in the SMTP config block (DarlingConfig.Smtp). There is no reason to look there when tuning Slack volume.

2. It is not reachable from the control plane. get_alert_settings does not return it and update_alert_settings does not accept it — the accepted-key switch in DarlingMcpAlertTools carries cooldown_minutes (the engine's fire cooldown) and analysis_notify_cooldown_minutes, but not this one. The column exists in config.config_notification and is written only by ViewerDataService.Notification, i.e. the WPF Settings window. So on a headless box the sole documented path to the setting that governs channel volume is a desktop app, or a hand-written UPDATE.

This also makes it invisible: an operator reading get_alert_settings sees cooldown_minutes: 5 and reasonably concludes that is the delivery cadence. It is not — it gates AlertEngine's fire decision. The delivery cadence is a different number in a different table that the tool will not show them.

3. The 120-minute clamp is low for a fleet. DarlingAlertSettings.cs:230 clamps to 1..120. On a fleet where one recurring signature is legitimately open for hours, 2 hours is the longest an operator can ask for, per fingerprint.

Observed

Measured on a 42-server store with Slack configured and no SMTP: 18 Deadlocks Detected cards in 74 minutes. The setting was at its default 15, never explicitly configured, and the delivered/suppressed boundary in config_alert_log lands on it exactly:

fingerprint gap since its own last delivery notification_type
11.5 min undelivered
29.5 min webhook
49.2 min webhook

That is the 15-minute per-fingerprint window behaving exactly as #1154 specified. The defect is not the mechanism; it is that the number driving it cannot be found or changed through the interface the deployment actually uses.

Suggested

  • Accept and return it in update_alert_settings / get_alert_settings under a channel-neutral name (delivery.cooldown_minutes), keeping email_cooldown_minutes as the stored column and an accepted alias so existing configs and the Lite Settings window keep working.
  • Raise the clamp ceiling, or document why 120 is the ceiling.
  • get_alert_settings should return it next to delivery.mode / delivery.per_event_max, where someone tuning volume will actually look.

Notes

cooldown_minutes and email_cooldown_minutes being two different numbers with near-identical names, governing two different stages of the same pipeline, is the root of the confusion — worth a sentence in each one's description regardless of whether the rename happens.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions