feat(backend): WebSocket relay retry and telemetry, payment processor telemetry suites, log retention validation

[noevidence1017]

Summary

WebSocket Relay Service: automated retry with exponential backoff (#1454)

• New src/lib/websocket-relay-retry.js:
  • withRelayRetry(operation, options): full-jitter backoff random(0, min(maxDelayMs, baseDelayMs * 2^n)), with a maxAttempts cap, a total maxElapsedMs budget and AbortSignal support.
  • deliverRelayMessage(message, send, { circuitBreaker, dlq }): retries a delivery, then moves the message to the dead letter queue with its total retry count when every attempt fails.
  • redriveDeadLetters(dlq, send, { limit }): replays queued messages oldest first; messages that fail again go back on the queue and are not retried twice in the same run.
• Only transient failures are retried: network error codes, timeouts, HTTP 408/429/5xx (except 501), and WebSocket close codes 1006/1011/1012/1013.
• Validation, signature and auth failures (other 4xx), policy close codes, retryable: false and an open circuit breaker fail at once, so retries never defeat the breaker.
• Options can be set with RELAY_RETRY_MAX_ATTEMPTS, RELAY_RETRY_BASE_DELAY_MS, RELAY_RETRY_MAX_DELAY_MS and RELAY_RETRY_MAX_ELAPSED_MS, and every value is clamped to a safe range.
• Logs carry the operation, attempt count, delay and the error's code, status and close code (messages truncated to 200 characters), never payloads.

WebSocket Relay Service: Prometheus alert metrics and health telemetry (#1453)

• New src/lib/websocket-relay-telemetry.js with its own registry:
  • Counters: websocket_relay_deliveries_total{outcome} and websocket_relay_retries_total{operation,result}.
  • Histogram: websocket_relay_retry_delay_seconds.
  • Gauges, computed at scrape time: websocket_relay_health_state, websocket_relay_delivery_failure_ratio, websocket_relay_dlq_depth, websocket_relay_circuit_open and websocket_relay_last_delivery_timestamp_seconds.
• A rolling-window health monitor (a fixed ring of buckets, so memory stays constant) combines the failure ratio and retry exhaustion with the live circuit breaker state and DLQ depth, attached through attachRelayHealthSource().
• New GET /health/websocket-relay endpoint: 503 only when unhealthy, 200 when degraded. GET /health now also reports services.websocket_relay for information only.
• /metrics now merges the relay telemetry registry and the existing relay_* registry ([Backend] Implement granular metrics tracking for Websocket Relay #1083), which was defined but never scraped before.
• Alert rules in docs/alerts/websocket-relay.rules.yml: unhealthy, circuit open, high failure ratio, retries exhausted, DLQ backlog, and no successful deliveries for 30 minutes. A test checks that every metric the rules reference is actually registered.

Payment Processor Telemetry: integration and stress suite (#1481)

• tests/integration/payment-processor-telemetry.test.js scrapes the real /metrics route and parses the Prometheus text. It checks:
  • every payment_processor_* series is in the merged output;
  • counts are exact per asset and outcome;
  • histograms are consistent (cumulative buckets, +Inf equals _count, correct _sum);
  • concurrent scrapes during writes never fail;
  • hostile asset values cannot add series or inject lines.
• load-tests/payment-processor-telemetry-stress.test.js checks:
  • no lost increments with 100 concurrent workers x 1,000 events;
  • cardinality stays bounded under 50,000 adversarial asset values;
  • histogram count and sum stay exact over 100,000 observations;
  • scrape p95 latency stays under 100 ms while writes continue.
• Fix found by the suite: createPaymentSession used the request's body.asset as a label, so any client could create one time series per arbitrary string, including on validation_failed. The new toAssetLabel() keeps well-formed asset codes (upper-cased) and maps anything else to invalid, or unknown when missing. It is applied at both paymentService call sites that used a raw asset.

Log Retention & Archival Service: payload sanitization and strict validation (#1472)

• parsePositiveInteger now only accepts plain base-10 integers in range. Before, parseInt silently turned values like 30days into 30; now 30days, 1e3, -5, 0x10 and 1.5 are rejected with a LogRetentionValidationError that names the option or environment variable. The old two-argument call still works.
• Allowed ranges are retentionDays 1-3650, batchSize 1-50000, maxDurationMs 1s-24h, plus a new maxBatches / LOG_PURGE_MAX_BATCHES cap.
• Unknown option keys are rejected, the pool must have query(), and everything is validated before the first query runs.
• SQL identifiers must be on an allow-list and match ^[a-z_][a-z0-9_]{0,62}$; values are always bind parameters.
• Each batch's driver row count must be between 0 and batchSize, and the maxBatches cap stops a misbehaving driver from looping forever.
• The return value is a frozen summary containing only known fields.
• sanitizeLogMessage() masks credentials (scheme://user:pass@, password=, token=, Bearer ...), strips control characters to prevent log injection, and caps length. The purge script now logs only the scrubbed message and the error's name and code, not the raw error object, which can echo the database connection string.

Tests

• backend/src/lib/websocket-relay-retry.test.js
• backend/src/lib/websocket-relay-telemetry.test.js
• backend/src/lib/log-retention-validation.test.js
• backend/tests/integration/payment-processor-telemetry.test.js
• backend/load-tests/payment-processor-telemetry-stress.test.js

Docs

• backend/docs/WEBSOCKET_RELAY.md (new)
• backend/docs/LOG_RETENTION.md (new)
• backend/docs/PAYMENT_PROCESSOR.md: new telemetry suite section (the following sections are renumbered)
• backend/docs/alerts/websocket-relay.rules.yml (new)

Notes

• No existing tests or fixtures were changed; the existing log-retention.test.js cases still apply.
• The WebSocket relay modules are libraries with no production caller in src yet. The retry and telemetry helpers are ready to be wired in where a relay connection is created; usage is in docs/WEBSOCKET_RELAY.md. reconnectWithBackoff is unchanged.
• Behaviour change: malformed log-retention environment values that used to be coerced now fail the purge run with a clear error, and unknown purgeWebhookLogs options are rejected.
• Behaviour change: the asset label on payment_processor_sessions_total, payment_processor_session_duration_seconds and confirmed verifications is now upper-cased, with invalid input bucketed as invalid. Dashboards filtering on a lower-case asset value would need updating.
• /metrics output grows by the relay_* and websocket_relay_* series.

closes #1454
closes #1453
closes #1481
closes #1472

[vercel]

@noevidence1017 is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@noevidence1017 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-	-	Generic High Entropy Secret	a5a3939	backend/src/lib/websocket-relay-retry.test.js	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}