feat(backend): add Prometheus alert metrics and health telemetry to Database Row-Level Security Service

[Abidoyesimze]

Summary

Closes #1468

Depends on #1609 (payload sanitization for the RLS service). This branch is stacked on #1609's branch, so until #1609 merges the diff below also shows that commit — after #1609 merges this PR auto-retargets to main and shrinks to just the telemetry work. The instrumentation wraps the validated withMerchantContext() path from #1609.

Every merchant-scoped transaction runs through withMerchantContext(), but the RLS path had no metrics, no health signal and no alerting: connection exhaustion, failing SET LOCAL transactions and rollback storms were invisible until callers started seeing query errors.

New src/lib/db-rls-telemetry.js (own prom-client registry, merged into /metrics by src/routes/prometheus.js):

Metric	Type	Labels
database_rls_transactions_total	Counter	outcome
database_rls_transaction_duration_seconds	Histogram	outcome
database_rls_active_transactions	Gauge	—
database_rls_health_state (0/1/2)	Gauge	—
database_rls_error_ratio / database_rls_rollback_ratio	Gauge	—
database_rls_last_transaction_timestamp_seconds	Gauge	—

outcome is a fixed set: committed, rolled_back (callback threw), invalid_input (failed validation, pool never touched), connection_error (pool.connect() failed), transaction_error (BEGIN/SET LOCAL/COMMIT failed, or ROLLBACK itself failed). withMerchantContext() records exactly one outcome per attempt via a timing wrapper; the active gauge is incremented on connect and decremented in finally.

Rolling-window health monitor (fixed ring of time buckets, constant memory): unhealthy when connection+transaction errors ≥ 5% of ≥ 20 samples; degraded when callback rollbacks ≥ 50%, or ≥ 10 invalid_input outcomes (a caller is misbehaving). Gauges refresh at scrape time so they decay when idle. Thresholds are env-overridable (DATABASE_RLS_*).

Endpoints:

• GET /health/database-row-level-security — 503 only when unhealthy; degraded still 200
• services.database_row_level_security added to GET /health (informational, does not affect ok)

Alert rules docs/alerts/database-row-level-security.rules.yml (promtool-valid): DatabaseRlsUnhealthy (critical), DatabaseRlsHighErrorRatio (critical), DatabaseRlsHighRollbackRatio (warning), DatabaseRlsInvalidInputSpike (warning), DatabaseRlsNoTraffic (info). Reference doc: docs/DATABASE_ROW_LEVEL_SECURITY.md.

Test plan

npx vitest run src/lib/db-rls-telemetry.test.js src/lib/db-rls.test.js
 Test Files  2 passed (2)
      Tests  37 passed (37)

18 new telemetry tests: health-monitor windows (healthy / unhealthy error ratio / degraded rollback ratio / invalid-input spike / min-sample floor / expiry of stale buckets / last-transaction timestamp), counter + histogram + gauge recording, unknown-outcome fallback, and outcome attribution through withMerchantContext with a mocked pool (committed, rolled_back, invalid_input ×2, connection_error, BEGIN failure → transaction_error, active gauge returning to zero). The 19 existing #1609 RLS tests still pass — no behavioural change to the transaction contract.

Full backend suite: no new failures vs. origin/main (identical pre-existing failure count).

Security notes

• No cardinality explosion: all label values are fixed server-side sets. Merchant ids, SQL text and error strings are never label values — request traffic cannot inflate the series count, and an attacker cannot spray unique labels through the RLS path to blow up Prometheus memory or cost.
• No data leakage into scrapes/health: the health snapshot returns only aggregate counts, ratios and derived reasons. Merchant ids and error messages stay in structured logs, never in /metrics or /health/* output, so the monitoring stack sees no customer data.
• invalid_input as a security signal: a spike in sanitized-away merchant ids usually means a caller (or an attacker probing the API) is sending malformed IDs; the dedicated alert turns that into a detectable event rather than silent log noise.
• Active-transaction gauge is decremented in finally, so a throwing callback cannot leak a permanently "in flight" count that would mask real leaks in the alert rules.

[vercel]

@Abidoyesimze is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@Abidoyesimze Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits