Add realm-server availability alert rules

[lukemelia]

What

A new Grafana alert rule group (realm-server-availability-group) with four rules covering how the realm-server fails under search load, ordered from "we're down" to early warning:

Rule	Signal	Threshold
Realm Server ALB 5xx Surge	CloudWatch HTTPCode_ELB_5XX_Count on the realm-server ALB	>50/min for 1m
Realm Server OOM Crash	log filter for Reached heap limit / JavaScript heap out of memory	any occurrence
Realm Server Heap Near Limit	heapMB= health-line value	>1600MB for 2m
Realm Server Search Saturation	inFlightSearch= health-line value	>15 for 3m

Why

During the 2026-09-09 production incident, both realm-server replicas hit the ~2GB V8 heap limit within a minute of each other under sustained concurrent _federated-search load, so the ALB had no healthy targets for ~2 minutes. Nothing alerted ahead of user reports: the processes died without app-level 5xx (ELB 502s never reach the app), and the heap / in-flight-search telemetry was only visible in logs. Replaying these rules against the incident, search saturation would have fired ~6 minutes before the crash and the other three at onset.

How

Follows the existing worker-status-group.json pattern: file-provisioned alert JSON pushed by apply-alerting.sh, with per-environment values substituted from the apply workflows' env vars. Two log-based rules read the realm-server's periodic health line via CloudWatch Logs Insights. The ALB CloudWatch dimension embeds a generated hash that can't be derived from a naming convention, so it's a per-env workflow var (REALM_SERVER_ALB_FULL_NAME), with a comment documenting how to look it up — same rationale as the worker log-group vars.

🤖 Generated with Claude Code

[habdelra]

[Claude Code 🤖] Reviewed each rule against the realm-server code that emits the signal it reads, plus the apply path and the two workflows. I did not exercise the rules against a live Grafana or CloudWatch, so the Logs Insights results and the ALB dimension are unverified against real data.

The rule shapes and the apply plumbing are right — resolve_placeholders discovers ${VAR} refs by grep, so the two new vars need no allowlist edit, and neither observability-diff.yml nor observability-preview.yml invokes apply-alerting.sh. One blocking issue: the heap rule's input isn't a periodic line, so that rule can't fire in the case it's named for.

Recommendations:

1. Emit heap unconditionally (and with its limit) before shipping rs-heap-near-limit — see the comment on the heap rule's expression.
2. Decide whether each for should mean sustained or single-spike, and align the window or the summary — comment on rs-search-saturation's for.
3. Say _search / _federated-search in the two annotations that name only the federated route — comment on the rs-search-saturation summary.
4. Answer how a stale REALM_SERVER_ALB_FULL_NAME gets caught, and weigh HealthyHostCount as the primary availability rule — comment on the 5xx rule's dimensions.

One question not tied to a line: the rules carry no labels and notification_settings: null, and this package provisions no contact points or notification policies (provisioning/alerting/ holds rule groups only), so routing rests on whatever each Grafana's default policy happens to be. Given the PR's premise is that nothing alerted, worth confirming the default route reaches a human in staging and production. If the worker-status-group rules are already firing into a channel today, that settles it.

Adjacent, out of scope: the README's tree listing describes alerting/ as "alert rule groups, contact points, notification policies" — only rule groups live there, in either tree.

---

Generated by Claude Code

[habdelra]

[Claude Code 🤖] heapMB= is not on a periodic line, so this rule can't see the case it's named for. startHealthSampler returns before formatting anything when the loop is calm and nothing is in flight — the maxLagMs < lagThresholdMs && inFlightSearch === 0 early return in packages/realm-server/health-sampler.ts, ahead of the heapMB computation. Heap is therefore reported only inside saturation windows, and noDataState: OK makes that blind spot indistinguishable from a healthy process.

The reachable case: once a search storm subsides, the process can sit at 1700MB retained with inFlightSearch=0 and a calm event loop. No health line, empty query, rule resolves to OK — while the process is one storm from death. That is the surviving-replica state the OOM rule's own summary tells the responder to worry about.

Second half of the same problem, at the 1600 threshold below: "the V8 limit is ~2048MB" is pinned to an observed value that nothing holds. packages/realm-server/scripts/start-production.sh sets no --max-old-space-size, so the realm-server runs on V8's default, and packages/realm-server/prerender/heap-telemetry.ts already documents why that isn't derivable — "V8 sizes its default old-space limit from visible memory rather than from the task's allocation". Resize the task or move Node versions and the threshold silently becomes always-hot or unreachable.

One emitter-side change fixes both: log heap unconditionally and include the limit. heapTelemetry() / formatHeapTelemetry() in prerender/heap-telemetry.ts already emit heapUsedMB=… heapLimitMB=…; reusing them for the realm-server health line gives an always-present number, lets this rule threshold on the ratio rather than a hardcoded MB figure, and drops the second spelling of the same quantity (heapMB= here vs heapUsedMB= there).

Regression, and blocking for this rule specifically — as written it can only fire in windows the search-saturation rule already covers.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Fixed in 33c5662. health-sampler.ts now emits the realm:health line every interval (the calm-loop early return is gone), and the heap fields come from the shared heapTelemetry()/formatHeapTelemetry() helpers, so the line carries heapUsedMB=… heapLimitMB=…. rs-heap-near-limit parses both and thresholds on used / limit * 100 > 80 instead of a hardcoded 1600MB, so a surviving replica sitting at a high retained heap with an idle loop now emits a line and is alertable, and a task resize or Node bump can't make the threshold always-hot or unreachable.

[habdelra]

[Claude Code 🤖] inFlightSearch counts plain _search as well as _federated-search — SEARCH_PATH_PATTERN in packages/realm-server/middleware/index.ts is /(^|\/)_(federated-)?search$/, and that's the only gate on the increment. So this summary can send a responder hunting a federated-search caller when the load is per-realm _search, including the in-render _search round-trips from prerender that the sampler was written to diagnose (see the header comment in health-sampler.ts). The OOM summary carries the same wording ("usually a _federated-search storm").

Worth spelling both in the two annotations. The module comment at the top of search-inflight.ts says the same wrong thing — pre-existing, but this PR turns it into incident-time instructions, so it's worth correcting in the same pass.

Class: pre-existing, now load-bearing. Non-blocking.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Done in 33c5662. The search-saturation and OOM annotations now name both _search and _federated-search, and the search-inflight.ts module comment says the counter tracks both routes (including prerender's in-render _search), since SEARCH_PATH_PATTERN is the only gate on the increment.

[habdelra]

[Claude Code 🤖] for doesn't buy sustained-ness here, because refId C reduces the entire query window with max. The window is 300s binned at 1m, so a single minute above 15 keeps C above 15 for the next ~5 evaluations at the group's 60s interval — which satisfies for: 3m by itself. The rule fires on one spike while its summary says "has been handling 15+ … for several minutes", which is the wording an on-call reads before deciding whether to act.

Same shape in the other two: heap is a 300s window against for: 2m, and the 5xx rule is a 600s window against for: 1m (which also keeps it Alerting for ~10 minutes after the surge ends, since the max stays in the window).

To make for mean what it says, shrink each query window to roughly one bin and let for do the sustaining — with the caveat that a 60s Logs Insights window can miss lines still inside the ingestion lag, so ~120s is the safer floor. Otherwise the window is the real condition and the summaries should say so ("exceeded 15 at least once in the last 5 minutes").

Regression (new rules). Non-blocking, but it decides how noisy these are on day one.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Addressed in 33c5662. Took the make-the-summary-honest option rather than shrinking windows to ~1 bin, to stay robust against Logs Insights ingestion lag: all rules are now for: 0s and the summaries read "at least once in the last N minutes". Also cut the 5xx window 600s→300s to halve its post-surge alerting tail.

[habdelra]

[Claude Code 🤖] A wrong or rotted REALM_SERVER_ALB_FULL_NAME fails silently rather than loudly. A bad LoadBalancer dimension isn't an error from GetMetricData — it's an empty result — so it lands in noDataState: OK, not execErrState: Alerting. The WORKER_LOG_GROUP_* vars are covered by that asymmetry (a wrong log group raises ResourceNotFoundException, which is exactly how their original mis-targeting was caught); this value isn't. It's also the one value in the PR that's hand-copied and can't be derived, and the dashboards deliberately avoid pinning it — overview.json resolves the ALB through a dimension_values(default,AWS/ApplicationELB,RequestCount,LoadBalancer) template variable, which alert rules can't use. How do you want to catch a stale value here — a post-apply preview of the rule in each env, or something standing?

Related, and worth weighing as the primary rule rather than 5xx: HealthyHostCount on the target group. The incident's signature was no healthy targets for ~2 minutes, and aws-elb.json already charts that metric with the note "Healthy = 0 means clients get 503s". Unlike HTTPCode_ELB_5XX_Count it's published continuously, so its noDataState can be Alerting — a misconfigured dimension then goes loud instead of silent, which answers the paragraph above at the same time. As written, the 5xx rule only fires if clients happen to be hitting the ALB during the outage window.

Question plus a follow-up. Non-blocking, but I'd want the stale-value answer either way.

---

Generated by Claude Code

[lukemelia]

[Claude Code 🤖] Both addressed in 33c5662. Added rs-no-healthy-targets (HealthyHostCount, min-reduced, < 1) as the primary availability rule with noDataState: Alerting — since the metric is published continuously, a stale or mistyped REALM_SERVER_TARGET_GROUP_FULL_NAME yields an empty result that trips this rule loudly instead of resolving OK. REALM_SERVER_TARGET_GROUP_FULL_NAME is wired into both apply workflows (staging + prod values read from each target group ARN). The 5xx rule stays as a secondary signal.

[lukemelia]

[Claude Code 🤖] Pushed 33c5662 addressing the four inline threads. Two review-level items:

• Notification routing: the rules still carry notification_settings: null and this package provisions no contact points/policies, so routing rests on each Grafana's default policy. Worth confirming the default route reaches a human in staging and production — if worker-status-group already fires into a channel today, that settles it. (Left as-is pending that confirmation.)
• README: fixed the alerting/ tree comment to say only rule groups live there.

[github-actions]

Host Test Results

    1 files      1 suites   2h 8m 39s ⏱️
4 734 tests 4 720 ✅ 14 💤 0 ❌
4 749 runs  4 735 ✅ 14 💤 0 ❌

Results for commit 0a0f4c9.

Realm Server Test Results

    1 files    210 suites   1h 15m 12s ⏱️
2 764 tests 2 764 ✅ 0 💤 0 ❌
2 803 runs  2 803 ✅ 0 💤 0 ❌

Results for commit 0a0f4c9.