Add starter custom-alert templates + drift-guard + list tool (#3285)

[erikdarlingdata]

What & why

Plan Component 7 and #3282's Q2 answer: ship starter custom-alert templates rather than new hardcoded evaluators, so the collected-but-unalerted signals become starting points an operator customizes — and the set generalizes to signals nobody has named yet.

Changes

• CustomAlertTemplates — a code-defined set of nine starter templates, each {key, name, description, definition-JSON} where the definition is a genuine CustomAlertRuleDefinition (metric + predicate + hysteresis; scope defaults to all servers). Thresholds are deliberately conservative starters (documented per template) that fire only after 3 consecutive breaching evaluations (2 for the more urgent blocking signal).
• CustomAlertTemplatesTests — the drift-guard, the alert twin of ViewTemplatesTests: every template's definition TryParses clean against the live rule-validator + MeasureCatalog on every build, so a measure that later drifts out of the catalog fails the build rather than shipping a rule that can never fire. Plus unique-key / shape / list-tool coverage.
• list_custom_alert_templates MCP tool (read-only, touches no store) — returns the set with each definition embedded as a JSON object, ready to hand to create_custom_alert_rule or test_custom_alert_rule.

The template set (signal → catalog measure → starter threshold)

PostgreSQL (the #3282 signals):

Template	Measure (source / measure or ratio, agg)	Starter (warn / critical)
dead tuples piling up	pg_autovacuum_stats / pg_av_dead_tuples, max	≥ 1,000,000 / 10,000,000
replica replay lag high	pg_replication_stats / pg_repl_replay_lag_ms, max	≥ 30s / 5min
connection count high	pg_session_states / pg_sess_total_sessions, max	≥ 200 / 500
table bloat high	pg_table_bloat_stats / pg_tbl_bloat_pct, max	≥ 40% / 70%
replication slot retaining WAL	pg_replication_slot_stats / pg_slot_retained_wal_bytes, max	≥ ~10 GB / ~50 GB

SQL Server (the obvious ones):

Template	Measure	Starter
high signal-wait %	wait_stats / signal_wait_pct (ratio)	≥ 25% / 40%
sustained blocking	blocked_process_reports / bpr_wait_time_ms, max	≥ 30s / 2min
long-running query	query_stats / query_avg_elapsed_us (ratio)	≥ 10s / 60s avg
tempdb space	tempdb_stats / tempdb_total_reserved_mb, max	≥ ~50 GB / ~100 GB

Deliberately skipped: a PostgreSQL "storage growth" template — the catalog has no PostgreSQL database-size gauge to threshold on (pg_database_stats is transaction/block/temp counters, not size). The related disk-fill risk is covered by the slot-WAL-retention template (retained WAL is the PostgreSQL storage a stuck slot actually grows unbounded); SQL Server storage is already covered by the built-in low-disk / file-growth alerts. Every unit is set explicitly (ms/mb/percent/count) so the threshold scale is unambiguous, and the drift-guard confirms each unit is catalog-valid.

Inventory pins (all five)

The new MCP tool trips the cross-project pins; all updated and verified by the passing CrossAppMcpToolInventoryPinTests, not hardcoded:

1. DarlingMcpInstructions census: 146 → 147 total, 86 shared, 60 → 61 unique, "seven → eight custom-alert-rule tools".
2. KnownLiteMissingMcpTools ratchet += list_custom_alert_templates.
3. DarlingWebEndpoints.ExcludedToolNames += it.
4. DarlingWebEndpointsTests pinned Ordinal-sorted array += it (between list_custom_alert_rules and list_custom_views).
5. DarlingMcpCustomAlertToolsSurfaceTests surface (7 → 8 names + Assert.Equal(8, …)) + a required-params row.

The census numbers are the live values with the tool registered: the pin derives total from the actual Darling tool count and unique = total − shared, and asserts the sentence matches — it passes at 147/86/61.

Tests

CustomAlertTemplatesTests (4): the drift-guard (every template validates against the live catalog), unique keys + says-what-it's-for, Scalar-with-predicate-and-both-tiers, and the list_custom_alert_templates tool returns every template with a parseable definition object.

Full local suites (from a worktree): Darling.Tests 8648 total / 3 failed, Lite.Tests 3699 total / 4 failed — all 7 are the same pre-existing environmental artifacts as the prior slices (source-tree-scan guards that resolve 0 files from a .claude/worktrees/ checkout, plus one TLS-cert platform test); every new + pinned test passes. Build: 0 Warning(s) / 0 Error(s).

Part of #3285.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WejwpgWF5Xfm3fAmoEbFz4

[claude]

Nit: the set currently has 9 templates, but this asserts >= 8. That lets someone silently drop a template (down to 8) without the "full starter set" guard catching it. Either assert == 9 (matching the PR description's "nine starter templates") or use CustomAlertTemplates.All.Count compared against a named constant so this stays a real regression guard rather than a loose floor.

[claude]

Review summary

Reviewed the diff (Darling-only: CustomAlertTemplates.cs, the new list_custom_alert_templates MCP tool, and the associated tests/doc updates). No SQL files are touched, so the T-SQL style section doesn't apply here.

Correctness — verified against the live catalog
Cross-checked all 9 template definitions against MeasureCatalog.cs by hand:

• Every source/measure/ratio key (pg_av_dead_tuples, pg_repl_replay_lag_ms, pg_sess_total_sessions, pg_tbl_bloat_pct, pg_slot_retained_wal_bytes, signal_wait_pct, bpr_wait_time_ms, query_avg_elapsed_us, tempdb_total_reserved_mb) exists and matches its Kind (plain measure vs. Ratio) — the two ratio measures correctly use "ratio" instead of "measure" and omit aggregate, matching ValidAggs = NoAggs.
• Units line up with UnitFamily/DefaultUnit, and the mb-family conversion factor (1024*1024, confirmed in MeasureCatalog.cs:309) makes the "~10 GB / ~50 GB" WAL-retention thresholds (10240/51200 MB) and the ms-based duration thresholds arithmetically correct.
• All hours values fall within [MinWindowHours, MaxWindowHours], and the drift-guard test (CustomAlertTemplatesTests.EveryTemplate_ValidatesAgainstTheLiveRuleValidatorAndCatalog) exercises the exact same CustomAlertRuleDefinition.TryParse authority the evaluator uses, so a future catalog drift will fail the build rather than ship a dead rule — good design.

Parity
Custom alert rules are Darling-only by architecture (central-store feature; no Lite twin), which is already established and re-documented at every place test_custom_alert_rule appears. This PR correctly extends list_custom_alert_templates to the exact same set of files (DarlingWebEndpoints.ExcludedToolNames, DarlingMcpInstructions tool-count prose, CrossAppMcpToolInventoryPinTests, DarlingMcpCustomAlertToolsTests), and the tool-count arithmetic (86 shared + 61 Darling-only = 147; 8 custom-alert tools) checks out. No parity drift found.

Security / Performance
ListCustomAlertTemplates is read-only, touches no store, takes no parameters, and only serializes a small static in-memory list — no injection surface, no perf concern.

One minor nit posted inline on the test file (a loose >= 8 count assertion where the set is actually 9). Otherwise this looks solid and well-guarded.