feat(backend): SEP-0001 retry with backoff; Payment Processor Telemetry sanitization, health and retries

[Elizabethxxx]

Summary

#1459 SEP-0001 Stellar Info Generator: retry with exponential backoff

• The only I/O behind GET /.well-known/stellar.toml is the merchant lookup. It now goes through fetchMerchantForStellarToml() (src/lib/sep0001-retry.js):
  • transient failures (network, 408/429/5xx, connection-class Postgres errors) are retried with full jitter, random(0, min(max, base * 2^n));
  • each attempt builds a fresh PostgREST query.
• Settings: SEP0001_RETRY_MAX_ATTEMPTS / _BASE_DELAY_MS / _MAX_DELAY_MS, defaults 3 / 100 / 1000, clamped to at most 5 attempts and 5 s.
• Route behaviour:
  • a lookup still failing transiently after the last attempt returns 503 with Retry-After: 5 instead of 500, so wallets retry rather than treating the domain as having no stellar.toml;
  • deterministic errors still return 500, an unknown merchant still 404, a missing id still 400;
  • every error response is Cache-Control: no-store;
  • errors are logged through the structured logger with code, attempt count and message only.
• New counter sep0001_merchant_lookup_retries_total{result} on /metrics.
• Tests: src/lib/sep0001-retry.test.js, new src/routes/sep0001.test.js (recover, 503, 500, 404, 400, no error-text leak), plus the app-level cases in the integration suite.
• Docs: "Retry with Exponential Backoff" section in SEP0001_GENERATOR_GUIDE.md.

#1477 Payment Processor Telemetry: payload sanitization and strict validation

• routes/payments.js and services/paymentService.js previously put req.body.asset straight into Prometheus labels, so a client could create a new series per request (unbounded memory here and in Prometheus) or inject text into dashboards.
• All processor telemetry now goes through recorders in src/lib/payment-processor-telemetry.js:
  • the payload must be a plain object;
  • outcome / stage / operation are checked against fixed allow-lists;
  • asset is normalised to ^[A-Z0-9]{1,12}$ (native becomes XLM; malformed codes become invalid; missing becomes unknown);
  • distinct asset labels are capped by PAYMENT_PROCESSOR_TELEMETRY_MAX_ASSET_LABELS (default 50), with overflow recorded as other;
  • durations must be finite and between 0 and 3600 s.
• Rejected payloads never touch the business series. They increment payment_processor_telemetry_rejected_total{metric,reason} and log one warning. Recorders never throw.
• Tests: src/lib/payment-processor-telemetry.test.js (sanitizer table, cardinality cap under a 500-asset flood, every rejection path, a recorder that must not throw).

#1478 Payment Processor Telemetry: Prometheus alert metrics and health

• A rolling-window health monitor (fixed ring of buckets) produces:
  • session failure ratio (validation failures are excluded as client errors);
  • verification error ratio;
  • refund error ratio;
  • telemetry rejection ratio;
  • exhausted read retries.
• Status: healthy, degraded or unhealthy, with configurable thresholds and minimum samples.
• New gauges refreshed at scrape time: payment_processor_health_state and the four ratio gauges.
• GET /health/payment-processor (503 only when unhealthy). GET /health lists services.payment_processor for information only. The body holds counts and ratios only.
• Alert rules: docs/alerts/payment-processor.rules.yml.
• Tests: health monitor transitions and window expiry, scrape-time gauges and series registration, plus the HTTP integration cases.

#1479 Payment Processor Telemetry: retry with exponential backoff

• The first Supabase read of the status poll and of verification (status_lookup, verify_lookup, in both the route and the service) is retried through retryProcessorRead() (src/lib/payment-processor-retry.js):
  • transient errors only, a fresh query per attempt;
  • settings: PAYMENT_PROCESSOR_RETRY_*, defaults 3 / 50 / 500, clamped to at most 5 attempts and 3 s.
• After the last attempt the final { data, error } comes back unchanged, so response codes are unchanged.
• Writes (status transitions, tx claims, refunds) are not wrapped.
• Outcomes go to payment_processor_retries_total{operation,result}; exhausted retries degrade health.
• The shared engine src/lib/backoff-retry.js (retryWithBackoff, retrySupabaseRead) is also used by SEP-0001. It reuses the existing isRetryableSessionError classifier and computeBackoffDelay.
• Tests: src/lib/payment-processor-retry.test.js, src/lib/backoff-retry.test.js.

Test output

Security notes

• Telemetry labels are bounded and allow-listed, so a client cannot inflate cardinality or inject label text. A unit test floods 500 distinct assets and checks the label count stays within the cap.
• Retries cover idempotent reads only, with hard ceilings on attempts and delay so request latency stays bounded. Deterministic errors (invalid id, not found, permission, missing column) are never retried, so retries cannot change an outcome.
• Health bodies and logs carry no payment ids, amounts, addresses, asset codes or merchant records. The SEP-0001 error body never includes database error text.
• stellar.toml error responses are no-store, so a CDN cannot cache a failure for the hour that successful responses are cached.

Notes

• Reading of [Backend] Implement automated retry with exponential backoff in Payment Processor Telemetry #1479: retry in "Payment Processor Telemetry" means retrying the processor's idempotent reads and recording every retry outcome through processor telemetry. Retrying in-memory metric writes would be meaningless.
• Behaviour changes:
  • stellar.toml returns 503 + Retry-After (was 500) when a transient lookup outage outlasts the retries.
  • Processor metric labels are normalised: asset codes are upper-cased; malformed codes become invalid, overflow becomes other; payloads with unknown outcomes are dropped and counted as rejected.
  • Status and verify reads can take slightly longer during a transient outage (at most about 1 s with the defaults).
• The existing payment-processor-metrics.js exports and series names are unchanged, and the existing metrics, e2e and SEP-0001 tests keep their expectations. No existing tests or fixtures were changed.

closes #1459
closes #1477
closes #1478
closes #1479

[vercel]

@Elizabethxxx is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@Elizabethxxx Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits