Skip to content

feat(backend): add relay distributed locking, harden and instrument Horizon retry manager, add French dashboard locale - #1601

Open
Andreschuks101 wants to merge 1 commit into
emdevelopa:mainfrom
Andreschuks101:feature/be-relay-locking-horizon-retry-hardening-dashboard-i18n
Open

Andreschuks101 wants to merge 1 commit into
emdevelopa:mainfrom
Andreschuks101:feature/be-relay-locking-horizon-retry-hardening-dashboard-i18n

Conversation

@Andreschuks101

Copy link
Copy Markdown

Summary

#1455 - Distributed concurrency control and locking for the WebSocket Relay Service

  • New backend/src/lib/websocket-relay-lock.js:
    • Takes a Redis lease with SET NX PX.
    • Only the holder can release or renew the lease. This uses compare-and-delete / compare-and-pexpire Lua scripts keyed on a random per-lease token.
    • withRelayLock(scope, resource, fn) renews the lease every ttl/3. If a renewal fails, it aborts the AbortSignal passed to fn with RELAY_LOCK_LOST instead of letting the work continue without exclusion.
    • Bounded waiting with +/-25% jitter (waitMs, max 30s).
    • When Redis is down, or is the no-op client from redis.js, it falls back to an in-process lock table.
    • createSingleFlight() gives a single in-flight promise per key inside one process, so reconnects don't stampede.
  • Scopes are allow-listed (event, dlq-drain, reconnect, connection). Resources are SHA-256 hashed into lock:ws-relay:<scope>:<sha256>, with a 512-char cap and no key-separator/CRLF injection.
  • CircuitBreaker (websocket-relay-recovery.js) now admits exactly one probe in HALF_OPEN, as its docs already said. Before, every concurrent caller was let through to the recovering upstream. getStatus() gains probeInFlight.
  • New relay metrics: relay_lock_acquisitions_total{scope,result}, relay_lock_wait_duration_seconds, relay_lock_hold_duration_seconds, relay_lock_lost_total, relay_circuit_breaker_probe_rejected_total.
  • Docs: backend/docs/WEBSOCKET_RELAY_CONCURRENCY.md. Alert rules: backend/docs/alerts/websocket-relay-locking.rules.yml.

#1437 - Payload sanitization and strict validation for the Horizon Client Retry Manager

  • New backend/src/lib/stellar/horizon-client/sanitizer.js.
  • Strict validation, failing with TypeError, code: HORIZON_CLIENT_VALIDATION_ERROR:
    • retryDelays: array, at most 10 entries, integers 0-60000 ms. Stored as a frozen copy.
    • operationName: must match /^[A-Za-z][A-Za-z0-9_]{0,63}$/, because it becomes a Prometheus label.
    • operation: must be a function. It is never called if validation fails.
  • Each rejection increments horizon_client_data_validation_errors_total{operation,validation_type}.
  • Context sanitization:
    • Plain objects only, max 16 keys, safe key names, prototype-pollution keys dropped.
    • Values stringified, control characters stripped, length capped.
    • Stellar secret seeds and Bearer/Basic credentials redacted.
    • Sensitive keys (secret, seed, token, api_key, authorization, ...) become [REDACTED].
  • Horizon URL: credentials, query string and fragment are stripped before it appears in errors or logs.
  • Horizon error detail and network error text are sanitized and capped at 512 chars.
  • error-handler.js (handleError, formatContext) and the retry log line now use the sanitized values.

#1438 - Prometheus alert metrics and health telemetry for the Horizon Client Retry Manager

  • New metrics:
    • horizon_client_retry_exhausted_total{operation,error_type}
    • horizon_client_consecutive_failures{operation}
    • horizon_client_last_success_timestamp_seconds{operation}
    • horizon_client_health_state (0 healthy, 1 degraded, 2 unhealthy)
  • RetryManager#getHealth() gives per-operation status, success/failure totals, last success/failure time and last error type. The worst operation decides the overall status. Unhealthy means unhealthyAfterFailures consecutive failures (default 5, configurable through HorizonClient options). The table is capped at 64 operations.
  • Non-retryable Horizon answers (404/400) mean Horizon is reachable, so they reset the failure streak instead of counting as an outage.
  • HorizonClient#getHealth() adds the last isReachable() probe. A failed probe downgrades a healthy client to degraded.
  • A one-time error log fires when an operation crosses the unhealthy threshold.
  • Alert rules in backend/docs/alerts/horizon-client-retry-manager.rules.yml: unhealthy, degraded, retries exhausted, high error ratio, no recent success, validation rejections.
  • Docs: backend/docs/HORIZON_CLIENT_RETRY_MANAGER.md.

#428 - Dashboard multi-language switcher (i18n)

  • The dashboard header already had a LocaleSwitcher and ES/PT catalogues.
  • This PR adds French: fr in src/i18n/config.ts (mapped to fr-FR) and a full messages/fr.json with all 705 keys.
  • Each catalogue now has a switcher label for fr.
  • Filled in the 18 keys ES and PT were missing (network.*, portfolioChart.*), so core pages no longer show raw keys in those locales.
  • Moved the remaining hard-coded string in the dashboard shell (the mobile nav aria-label) into dashboardShell.openNavigationMenu.
  • src/i18n/messages.test.ts checks that every locale has the same keys as en, keeps the same ICU arguments, has no empty strings, and that the switcher lists every locale.
  • LocaleSwitcher.test.tsx covers the options, the cookie write plus route replace, a no-op when re-selecting, and the fallback for an unsupported locale.
  • Docs: frontend/I18N.md.

Notes

  • Behaviour change: RetryManager / HorizonClient now throw at construction if retryDelays is invalid (negative, non-integer, over 60s, more than 10 entries). The defaults and every existing caller are valid.
  • Behaviour change: Horizon error messages no longer include URL credentials, query strings, secret seeds or raw control characters. Object context values are now JSON-encoded instead of showing [object Object].
  • Behaviour change: a HALF_OPEN circuit breaker rejects concurrent calls while its single probe is in flight.
  • Test fixture changes:
    • backend/tests/helpers/fake-redis.js gains support for the compare-and-pexpire EVAL script, which lease renewal needs. This is purely additive; existing behaviour is unchanged.
    • The ../metrics.js mock in horizon-client.test.js gains the five new metric exports, because Vitest throws on access to undefined mock exports.
    • websocket-relay-recovery.test.js and horizon-client.test.js gain new test cases only.
  • Assumption: the relay modules are not wired into a running relay process yet (the existing relay files are standalone libraries). The locking primitives are ready for that wiring and are covered by tests on their own.
  • Assumption ([FE] Dashboard Multi-Language Switcher (i18n) #428): "core pages" means the authenticated dashboard namespaces already in messages/*.json. Locale names in the switcher follow the existing unaccented style (Espanol, Portugues, Francais).
  • Security notes:
    • Locks are owner-checked, so a stale holder cannot release or revive a lease someone else now holds.
    • Lock keys are hashed and scope-allow-listed.
    • The in-process fallback keeps one instance correct when Redis is down; it is logged and alertable.
    • Retry-manager input is validated before any work or metrics happen, and every value echoed to logs or API errors is redacted and bounded.
  • Risk: lease renewal depends on the event loop. A stall longer than the TTL loses the lease, and the work is told to stop through the abort signal (counted in relay_lock_lost_total).
  • Test output: to be attached after running the suites below.

closes #1455
closes #428
closes #1437
closes #1438

…orizon retry manager, add French dashboard locale
@vercel

vercel Bot commented Oct 16, 2026

Copy link
Copy Markdown

@Andreschuks101 is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

@drips-wave

drips-wave Bot commented Oct 16, 2026

Copy link
Copy Markdown

@Andreschuks101 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment