feat(backend): harden Horizon retry manager and sanitize exchange rate

[arandomogg]

Automated retry with exponential backoff (#1439)

• New horizon-client/backoff.js: jittered exponential backoff by default (full, equal or none). An explicit retryDelays array keeps the old fixed schedule.
• Honours Retry-After, and fails fast when the hint is longer than maxRetryAfterMs. A total time budget (maxElapsedMs) covers every wait.
• Errors with retryable: false are never retried. Every failure carries attempts and retryStopReason.
• Bug fix: the Stellar SDK rethrows connection failures as plain Errors with no code (for example connect ECONNREFUSED), so they were never retried. isRetryableError now also checks the message and err.cause.code.

Distributed concurrency control and locking (#1440)

• ConcurrencyLimiter: caps in-flight Horizon calls per process and holds the rest in a bounded FIFO queue. When the queue is full or a wait times out, the caller gets a 503 (HORIZON_CLIENT_QUEUE_FULL / HORIZON_CLIENT_QUEUE_TIMEOUT).
• HorizonRetryCoordinator: retry slots shared across the cluster, stored as Redis keys taken with SET NX PX and released with compare-and-delete. It also publishes a shared rate-limit cooldown after a 429 or 503, so many instances cannot turn one Horizon incident into a retry storm.
• Falls back to an in-process implementation when Redis is missing or erroring. Keys are built from a hashed host and a sanitized operation name.
• Turned on at startup with configureHorizonRetryCoordination() in app.js.

Integration and stress test suite (#1441)

• tests/integration/horizon-retry-manager.test.js: runs the real HorizonClient and Stellar SDK against a local HTTP Horizon stub. Covers retries, 404s, running out of retries, the time budget, connection refused, the concurrency cap, load shedding, and the shared cooldown and retry slots across instances.
• load-tests/horizon-retry-manager-stress.test.js: 2,000 operations with 30% transient failures, 8 instances retrying against a Horizon that is down, everyone holding through the shared 429 cooldown, queue saturation, Redis flapping, and 10,000 successful operations. Each test also checks that nothing leaks.
• Unit tests for backoff, the limiter, the coordinator, the retry manager and the error handler.

Payload sanitization and strict validation for the Exchange Rate Oracle Cache (#1442)

• New exchange-rate-oracle-sanitizer.js:
  • sanitizeQuoteRequest() validates and normalizes asset codes, issuers and the source account (full StrKey checksum), amounts (positive, at most 7 decimals, within int64 stroops) and slippage. Bad input gets a 400 ExchangeRateValidationError that is never retried, and the route returns { error, field }.
  • sanitizeQuote() rebuilds every quote from a fixed list of allowed fields, both for Horizon results and for entries read from the shared Redis store. Unknown fields and prototype keys are dropped, and a poisoned entry is treated as a cache miss.
• Invalidation uses the same normalized key and never throws.

Notes

• The default retry timing is now jittered exponential (still the first attempt plus 2 retries). Anyone passing retryDelays keeps the exact old schedule.
• Strict validation now rejects credit assets without an issuer and issuers or accounts with bad checksums. These used to fail later inside resolveAsset or at Horizon. Some existing test fixtures used made-up keys with bad checksums; I replaced them with valid generated keys and added issuers where they were missing.
• The SDK does not expose response headers on its errors, so when there is no Retry-After header the cooldown falls back to the backoff ceiling.
• These suites already fail on main and are not touched by this PR: src/lib/stellar/horizon-client/metrics.test.js, src/lib/stellar.test.js, src/lib/stellar-metrics.test.js, tests/integration/exchange-rate-cache.test.js (401 from auth setup) and load-tests/path-payment-load.test.js.

Docs

• backend/docs/HORIZON_CLIENT_RETRY_MANAGER.md (new)
• backend/docs/EXCHANGE_RATE_ORACLE_CACHE.md (new section)

closes #1439
closes #1440
closes #1441
closes #1442

[vercel]

@arandomogg is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@arandomogg Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-	-	Generic High Entropy Secret	f2ae1d5	backend/src/lib/exchange-rate-oracle-sanitizer.test.js	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}