[Teeeyanaa]

Summary

Fix DB alerts, add runbooks, and configure real Alertmanager receivers

What was solved

#88 — DatabaseDown alert measures the exporter, not Postgres

Update monitoring/alert-rules.yml so the DatabaseDown alert fires on pg_up == 0 (actual Postgres availability) instead of only up{job="database"} == 0 (exporter availability), and add/keep a separate alert for the postgres-exporter being down.

Addressed:

• Changed: monitoring/alert-rules.yml
• Alert on pg_up == 0 to detect Postgres being down even when the exporter is up
• Keep a separate alert for the exporter itself (up{job="database"} == 0)
• Scope limited to monitoring/alert-rules.yml; do not change unrelated monitoring config or other files

#89 — Replace placeholder runbook URLs with real runbooks

Replace placeholder docs.example.com runbook URLs in monitoring/alert-rules.yml with real runbook files authored under docs/runbooks/, one per alert, and update the alert rules to point runbook_url at those docs.

Addressed:

• Changed: monitoring/alert-rules.yml, docs/MONITORING.md
• Create runbook markdown files in docs/runbooks/ covering each alert defined in monitoring/alert-rules.yml
• Update every runbook_url in monitoring/alert-rules.yml to reference the new real runbook docs instead of https://docs.example.com/runbooks/...
• Keep changes scoped to alert-rules.yml and docs/runbooks/ (plus minimal docs/ references if needed); do not refactor unrelated monitoring config or code

#90 — Configure real Alertmanager receivers

Replace the dead localhost webhook receiver in monitoring/alertmanager.yml with real Slack, PagerDuty, and email receivers sourced from env/secret files, and add routing so critical alerts go to high-urgency channels (PagerDuty + Slack critical + email) while warnings go to lower-urgency channels (Slack warnings + email).

Addressed:

• Changed: monitoring/alertmanager.yml, monitoring/alertmanager.yml, .env.example, monitoring/alertmanager.yml
• Add Slack, PagerDuty, and email receivers configured from env/secret files (no hardcoded secrets)
• Route critical vs warning alerts differently via route tree
• Remove/replace the non-existent http://localhost:9095/alerts receiver

Changes

• monitoring/alert-rules.yml (modify)
• docs/MONITORING.md (modify)
• monitoring/alertmanager.yml (modify)
• .env.example (modify)

Approach

1. DatabaseDown alert measures the exporter, not Postgres #88 — DatabaseDown alert measures the exporter, not Postgres (Changed: monitoring/alert-rules.yml)
2. Replace placeholder runbook URLs with real runbooks #89 — Replace placeholder runbook URLs with real runbooks (Changed: monitoring/alert-rules.yml, docs/MONITORING.md)
3. Configure real Alertmanager receivers #90 — Configure real Alertmanager receivers (Changed: monitoring/alertmanager.yml, monitoring/alertmanager.yml, .env.example, monitoring/alertmanager.yml)

Issues

Closes #88
Closes #89
Closes #90

[drips-wave]

@Teeeyanaa Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits