feat(backend): WebSocket relay retry and telemetry, payment processor telemetry suites, log retention validation

backend/docs/LOG_RETENTION.md

@@ -0,0 +1,45 @@
+# Log Retention & Archival Service
+
+`src/lib/log-retention.js` purges operational webhook delivery logs
+(`webhook_delivery_logs`) older than the retention window, in batches. Run it
+with `npm run purge:webhook-logs` (see `scripts/purge-webhook-logs.js` and
+`.github/workflows/webhook-log-retention.yml`).
+
+> Compliance: confirm the table holds operational delivery telemetry only.
+> Payment audit records may need longer retention by policy or regulation.
+
+## Configuration
+
+| Option | Environment variable | Default | Allowed |
+|---|---|---|---|
+| `retentionDays` | `LOG_RETENTION_DAYS` | 30 | 1-3650 |
+| `batchSize` | `LOG_PURGE_BATCH_SIZE` | 1000 | 1-50000 |
+| `maxDurationMs` | `LOG_PURGE_MAX_DURATION_MS` | 300000 | 1000-86400000 |
+| `maxBatches` | `LOG_PURGE_MAX_BATCHES` | 100000 | 1-10000000 |
+
+Explicit options win over environment variables. `logger` (an object with
+`log` and `error`) may be passed instead of `console`.
+
+## Strict validation and sanitization (#1472)
+
+- Numbers must be plain base-10 integers in range. Values such as `30days`,
+  `1e3`, `-5`, `0x10` or `1.5` are rejected with a
+  `LogRetentionValidationError` naming the option or environment variable.
+  Before this change `parseInt` silently turned `30days` into `30`.
+- Unknown option keys (for example a typo like `retentiondays`) are rejected
+  rather than silently falling back to the default retention.
+- `pool` must expose `query()`. Everything is validated before the first
+  query runs.
+- Only allow-listed table/column pairs can be purged, and identifiers are
+  checked against `^[a-z_][a-z0-9_]{0,62}$` before they reach SQL. The
+  retention period and batch size are always bind parameters.
+- Each batch's `rowCount` must be an integer between 0 and `batchSize`; a
+  bad value stops the purge. `maxBatches` caps the loop so a misbehaving
+  driver cannot spin forever.
+- The returned summary is a frozen object with `tableName`, `retentionDays`,
+  `batchSize`, `totalDeleted`, `durationMs` and `batches` only.
+- `sanitizeLogMessage()` masks credentials (`scheme://user:pass@`,
+  `password=`, `token=`, `Bearer ...`), strips control characters (log
+  injection) and caps length. The purge script logs only the scrubbed message
+  and the error's name/code, never the raw error object, which can echo the
+  database connection string.

backend/docs/PAYMENT_PROCESSOR.md

@@ -110,11 +110,40 @@ Include the printed percentile tables in PRs touching the processor.
 
 ---
 
-## 5. Security notes
+## 5. Telemetry integration and stress suite (#1481)
+
+| Suite | File | Run |
+|---|---|---|
+| Integration | `tests/integration/payment-processor-telemetry.test.js` | `npx vitest run tests/integration/payment-processor-telemetry.test.js` |
+| Stress | `load-tests/payment-processor-telemetry-stress.test.js` | `npm run test:load -- payment-processor-telemetry` |
+
+The integration suite scrapes the real `/metrics` route and parses the
+exposition text a Prometheus server would receive. It checks that every
+`payment_processor_*` series is merged into the scrape, that counts are
+exact per asset and outcome, that histograms are consistent (cumulative
+buckets, `+Inf` equals `_count`, `_sum` matches), that concurrent scrapes
+during writes never fail, and that the output always parses.
+
+The stress suite checks there are no lost increments (100 concurrent workers
+x 1,000 events), that cardinality stays bounded under 50,000 adversarial
+asset values, that histogram count and sum stay exact over 100,000
+observations, and that scrape p95 latency stays under 100 ms while writes
+continue.
+
+**Fix found by the suite:** session creation used `body.asset` straight from
+the request as the `asset` label, so a client could create one time series
+per arbitrary string (including on `validation_failed`). Labels now go
+through `toAssetLabel()`: well-formed asset codes are upper-cased, anything
+else becomes `invalid`, and a missing value becomes `unknown`.
+
+---
+
+## 6. Security notes
 
 - API keys are hashed for rate-limit bucketing (`getCreatePaymentRateLimitKey`)
 - Metadata keys must match `^[a-zA-Z0-9_-]{1,64}$`; unsafe keys never reach SQL
 - Refunds require a confirmed payment and are single-use (`refund_status`)
 - Verification uses a DB-level conditional update so a `tx_hash` can only be
   claimed once even under concurrent confirmation attempts
 - Signature verification failure keeps payments pending (fail-closed)
+- Metric labels never carry raw client input; `asset` is bounded by `toAssetLabel()`

backend/docs/WEBSOCKET_RELAY.md

@@ -0,0 +1,101 @@
+# WebSocket Relay Service
+
+Covers issues **#1454** (automated retry with exponential backoff) and
+**#1453** (Prometheus alert metrics and health telemetry).
+
+## 1. Layout
+
+| File | Role |
+|---|---|
+| `src/lib/websocket-relay-security.js` | Origin, size, JWT and message-shape checks |
+| `src/lib/websocket-relay-recovery.js` | Circuit breaker, dead letter queue, `reconnectWithBackoff` |
+| `src/lib/websocket-relay-retry.js` | Retry policy, `deliverRelayMessage`, `redriveDeadLetters` (#1454) |
+| `src/lib/websocket-relay-metrics.js` | Granular relay counters (#1083) |
+| `src/lib/websocket-relay-telemetry.js` | Alert metrics and rolling-window health (#1453) |
+| `docs/alerts/websocket-relay.rules.yml` | Prometheus alert rules |
+
+## 2. Retry with exponential backoff (#1454)
+
+```js
+import { deliverRelayMessage, redriveDeadLetters, withRelayRetry } from "./lib/websocket-relay-retry.js";
+import { CircuitBreaker, DeadLetterQueue } from "./lib/websocket-relay-recovery.js";
+import { attachRelayHealthSource } from "./lib/websocket-relay-telemetry.js";
+
+const circuitBreaker = new CircuitBreaker();
+const dlq = new DeadLetterQueue();
+attachRelayHealthSource({ circuitBreaker, dlq });
+
+// Deliver, retrying transient failures, dead-lettering what still fails.
+await deliverRelayMessage(message, (msg) => socket.sendAsync(msg), { circuitBreaker, dlq });
+
+// Reconnect with the same policy.
+await withRelayRetry(() => connect(), { operation: "reconnect", signal });
+
+// Later, once the downstream recovers:
+await redriveDeadLetters(dlq, (msg) => socket.sendAsync(msg), { limit: 100 });
+```
+
+- **Delay:** full jitter, `random(0, min(maxDelayMs, baseDelayMs * 2^n))`.
+- **Retried:** network error codes (`ECONNRESET`, `ETIMEDOUT`, ...), timeouts,
+  HTTP 408/429/5xx (not 501), WebSocket close codes 1006/1011/1012/1013, and
+  errors flagged `retryable: true`.
+- **Not retried:** an OPEN circuit breaker (`circuitOpen`), other 4xx
+  (validation, signature, auth), policy close codes, `retryable: false`,
+  and programming errors. These fail or dead-letter immediately.
+- **Bounds:** `maxAttempts`, a total `maxElapsedMs` budget including waits,
+  and an optional `AbortSignal`.
+- **Failures** carry `retryAttempts`. With a `dlq`, `deliverRelayMessage`
+  dead-letters the message (with the total retry count) instead of throwing.
+- **Logs** contain the operation label, attempt counts, delay and the
+  error's code/status/close code, with the message truncated to 200
+  characters. Payloads are never logged.
+
+| Variable | Default | Range |
+|---|---|---|
+| `RELAY_RETRY_MAX_ATTEMPTS` | `5` | 1-10 |
+| `RELAY_RETRY_BASE_DELAY_MS` | `250` | 0-60000 |
+| `RELAY_RETRY_MAX_DELAY_MS` | `10000` | base-60000 |
+| `RELAY_RETRY_MAX_ELAPSED_MS` | `30000` | 0-300000 |
+
+`reconnectWithBackoff` in `websocket-relay-recovery.js` is unchanged.
+
+## 3. Alert metrics and health (#1453)
+
+Exposed on `GET /metrics` (merged from their own registry, together with the
+existing `relay_*` series, which were not previously scraped).
+
+| Metric | Type | Labels |
+|---|---|---|
+| `websocket_relay_deliveries_total` | counter | `outcome`: delivered, recovered, dead_lettered, failed |
+| `websocket_relay_retries_total` | counter | `operation`: deliver, reconnect, redrive, other; `result`: scheduled, recovered, exhausted, aborted |
+| `websocket_relay_retry_delay_seconds` | histogram | `operation` |
+| `websocket_relay_health_state` | gauge | 0 healthy, 1 degraded, 2 unhealthy |
+| `websocket_relay_delivery_failure_ratio` | gauge | rolling window |
+| `websocket_relay_dlq_depth` | gauge | live DLQ size |
+| `websocket_relay_circuit_open` | gauge | 1 OPEN, 0.5 HALF_OPEN, 0 CLOSED |
+| `websocket_relay_last_delivery_timestamp_seconds` | gauge | |
+
+Gauges are computed at scrape time from a fixed ring of 5-second buckets, so
+they decay when traffic stops and memory stays constant.
+
+`GET /health/websocket-relay` returns the snapshot (no payloads or payment
+ids). Status rules:
+
+| Status | When |
+|---|---|
+| unhealthy (HTTP 503) | circuit OPEN, failure ratio >= `RELAY_FAILURE_RATIO_THRESHOLD` (default 0.1) with at least `RELAY_HEALTH_MIN_SAMPLES` (20) deliveries, or DLQ >= `RELAY_DLQ_UNHEALTHY_THRESHOLD` (100) |
+| degraded (HTTP 200) | circuit HALF_OPEN, DLQ >= `RELAY_DLQ_DEGRADED_THRESHOLD` (1), or retries exhausted in the window |
+| healthy | otherwise |
+
+The window is `RELAY_HEALTH_WINDOW_MS` (default 5 minutes). `GET /health`
+also reports `services.websocket_relay` (informational; it does not change
+the overall status code).
+
+## 4. Security notes
+
+- Label values come from fixed server-side sets; message fields, origins,
+  payment ids and error text never become labels.
+- The health snapshot and logs contain counts, states and error codes only.
+- An OPEN breaker is never retried, so retries cannot defeat it.
+- Retry configuration is clamped, so a misconfigured environment cannot
+  disable backoff or make a delivery wait for minutes.

backend/docs/alerts/websocket-relay.rules.yml

@@ -0,0 +1,83 @@
+# Prometheus alerting rules for the WebSocket Relay Service (issue #1453).
+#
+# Load with:   rule_files: ["docs/alerts/websocket-relay.rules.yml"]
+# Validate:    promtool check rules docs/alerts/websocket-relay.rules.yml
+#
+# Metric reference: docs/WEBSOCKET_RELAY.md
+
+groups:
+  - name: websocket-relay
+    rules:
+      - alert: WebSocketRelayUnhealthy
+        expr: max(websocket_relay_health_state) >= 2
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: WebSocket relay reports unhealthy
+          description: >
+            The circuit breaker is open, the delivery failure ratio is above
+            threshold, or the dead letter queue has a large backlog. See
+            GET /health/websocket-relay for the reasons.
+
+      - alert: WebSocketRelayCircuitOpen
+        expr: max(websocket_relay_circuit_open) >= 1
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: WebSocket relay circuit breaker is open
+          description: >
+            Deliveries are being rejected without being attempted and go
+            straight to the dead letter queue.
+
+      - alert: WebSocketRelayHighFailureRatio
+        expr: |
+          sum(rate(websocket_relay_deliveries_total{outcome=~"failed|dead_lettered"}[10m]))
+            /
+          clamp_min(sum(rate(websocket_relay_deliveries_total[10m])), 1e-9)
+            > 0.1
+          and sum(increase(websocket_relay_deliveries_total[10m])) >= 20
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          summary: More than 10% of WebSocket relay deliveries are failing
+          description: >
+            Check logs for "WebSocket relay operation failed after exhausting
+            retries".
+
+      - alert: WebSocketRelayRetriesExhausted
+        expr: sum(increase(websocket_relay_retries_total{result="exhausted"}[10m])) > 5
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: WebSocket relay retries are being exhausted
+          description: >
+            Transient failures are lasting longer than the retry budget
+            (RELAY_RETRY_MAX_ATTEMPTS / RELAY_RETRY_MAX_ELAPSED_MS).
+
+      - alert: WebSocketRelayDeadLetterBacklog
+        expr: max(websocket_relay_dlq_depth) > 0
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: WebSocket relay dead letter queue is not draining
+          description: >
+            Messages have been waiting in the DLQ for 15 minutes. Run
+            redriveDeadLetters() once the downstream recovers.
+
+      - alert: WebSocketRelayNoDeliveries
+        expr: |
+          time() - max(websocket_relay_last_delivery_timestamp_seconds) > 1800
+          and sum(increase(websocket_relay_deliveries_total[30m])) > 0
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: WebSocket relay has attempted deliveries but none succeeded for 30 minutes
+          description: >
+            Deliveries are being attempted but none succeed. Check the
+            relay_errors_total breakdown and the circuit breaker state.