diff --git a/backend/docs/HORIZON_CLIENT_RETRY_MANAGER.md b/backend/docs/HORIZON_CLIENT_RETRY_MANAGER.md new file mode 100644 index 00000000..aaa6def4 --- /dev/null +++ b/backend/docs/HORIZON_CLIENT_RETRY_MANAGER.md @@ -0,0 +1,68 @@ +# Horizon Client Retry Manager + +`src/lib/stellar/horizon-client/retry-manager.js` wraps every Horizon call made +by `HorizonClient` (`loadAccount`, `fetchPayments`, `fetchTransaction`, +`findStrictReceivePaths`, `getFeeStats`) with retries, error mapping, metrics +and health tracking. + +## Payload sanitization and strict validation (issue #1437) + +Implemented in `src/lib/stellar/horizon-client/sanitizer.js`. + +| Input | Rule | On failure | +|-------|------|------------| +| `retryDelays` (constructor) | Array of at most 10 integers, each 0–60000 ms. Stored as a frozen copy. | `TypeError`, `code: HORIZON_CLIENT_VALIDATION_ERROR`, `validationType: "retry_delays"` | +| `operationName` | `/^[A-Za-z][A-Za-z0-9_]{0,63}$/` (it becomes a Prometheus label) | Same error, `validationType: "operation_name"`. The operation is never called. | +| `operation` | Must be a function | Same error, `validationType: "operation"` | +| `context` | Plain objects only. Keys must match `/^[A-Za-z0-9_.-]{1,64}$/`. `__proto__`/`constructor`/`prototype` are dropped. Max 16 keys. | Invalid keys are dropped silently. | +| context values | Stringified (objects as JSON). Control characters stripped. Stellar secret seeds (`S…`, 56 chars) and `Bearer`/`Basic` credentials redacted. Truncated to 256 chars. Values under keys matching `secret|seed|password|passphrase|token|private|signature|authorization|api_key|mnemonic|cookie` are replaced with `[REDACTED]`. | — | +| Horizon URL in errors and logs | Only `scheme://host[:port]/path` is kept. Credentials, query string and fragment are removed. | `[invalid-horizon-url]` | +| Horizon error `detail` / network error text | Same string rules, capped at 512 chars | — | + +Validation failures increment +`horizon_client_data_validation_errors_total{operation, validation_type}`. +The `operation` label is `invalid` when the name itself was rejected. + +## Alert metrics and health telemetry (issue #1438) + +| Metric | Type | Meaning | +|--------|------|---------| +| `horizon_client_retry_exhausted_total{operation,error_type}` | Counter | Calls that failed with a retryable error after every retry | +| `horizon_client_consecutive_failures{operation}` | Gauge | Consecutive exhausted failures. Reset by a success or by a non-retryable Horizon response (404/400 means Horizon is reachable). | +| `horizon_client_last_success_timestamp_seconds{operation}` | Gauge | Unix time of the last success | +| `horizon_client_health_state` | Gauge | `0` healthy, `1` degraded (any operation has ≥1 consecutive failure), `2` unhealthy (≥ `unhealthyAfterFailures`, default 5) | + +`HorizonClient#getHealth()` returns the same data as JSON, plus the result of +the last `isReachable()` probe. A failed probe downgrades an otherwise healthy +client to `degraded`: + +```json +{ + "status": "degraded", + "unhealthyAfterFailures": 5, + "concurrentOperations": 0, + "operations": { + "loadAccount": { + "status": "degraded", + "consecutiveFailures": 2, + "totalSuccesses": 120, + "totalFailures": 2, + "lastSuccessAt": "2026-10-16T10:00:00.000Z", + "lastFailureAt": "2026-10-16T10:05:00.000Z", + "lastErrorType": "network_error" + } + }, + "lastHealthCheck": { "reachable": false, "durationMs": 2001, "errorType": "timeout", "checkedAt": "2026-10-16T10:05:01.000Z" } +} +``` + +Configure the threshold with `new HorizonClient(url, passphrase, { unhealthyAfterFailures: 3 })`. +The per-operation table is capped at 64 entries. + +Alerting rules: `docs/alerts/horizon-client-retry-manager.rules.yml`. + +## Tests + +```bash +npx vitest run src/lib/stellar/horizon-client src/lib/stellar/horizon-client.test.js +``` diff --git a/backend/docs/WEBSOCKET_RELAY_CONCURRENCY.md b/backend/docs/WEBSOCKET_RELAY_CONCURRENCY.md new file mode 100644 index 00000000..48d5b9c5 --- /dev/null +++ b/backend/docs/WEBSOCKET_RELAY_CONCURRENCY.md @@ -0,0 +1,73 @@ +# WebSocket Relay: distributed concurrency control + +Issue #1455. Module: `src/lib/websocket-relay-lock.js`. + +## What it protects + +| Scope (`RelayLockScope`) | Resource | Prevents | +|--------------------------|----------|----------| +| `event` | relay event / payment id | Two instances delivering the same event | +| `dlq-drain` | queue name | Concurrent DLQ replays re-sending every entry once per instance | +| `reconnect` | upstream URL | Reconnect stampedes across instances | +| `connection` | connection id | Concurrent mutation of one connection's state | + +Inside one process, `createSingleFlight()` makes concurrent callers share a +single in-flight promise per key. Use it for reconnect loops so that many +sockets seeing the same upstream drop start only one reconnect. + +`CircuitBreaker` in `websocket-relay-recovery.js` now admits exactly **one** +probe while `HALF_OPEN`. Concurrent callers are rejected with +`err.circuitOpen = true` and counted in +`relay_circuit_breaker_probe_rejected_total`. Before this change every +concurrent caller was let through to the recovering upstream. + +## Usage + +```js +import { withRelayLock, RelayLockScope } from "./websocket-relay-lock.js"; + +await withRelayLock(RelayLockScope.DLQ_DRAIN, "default", async ({ signal }) => { + while (!signal.aborted && dlq.size > 0) { + await deliver(dlq.shift()); + } +}, { ttlMs: 15_000, waitMs: 0 }); +``` + +- `withRelayLock` throws a `409` error with `code: "RELAY_LOCK_BUSY"` (`retryable: true`) when another holder owns the lock after `waitMs` (default `0`, max 30 s, polled with ±25% jitter). +- The lease is renewed every `ttl/3`. If a renewal fails, `signal` is aborted with `code: "RELAY_LOCK_LOST"`. Long-running work must check `signal.aborted` between units of work. +- The lock is always released in `finally`, including when the work throws. + +## Safety properties + +- **Atomic acquire:** `SET key token NX PX ttl`. +- **Owner-only release and renew:** compare-and-delete and compare-and-pexpire Lua scripts check the random per-lease token. A worker whose lease expired can neither delete nor revive a lock that someone else now holds. +- **Key hygiene:** the scope must be in an allow-list. The resource is SHA-256 hashed (`lock:ws-relay::`), so it cannot inject key separators or CRLF, and its length is capped at 512 characters. +- **TTL bounds:** `WS_RELAY_LOCK_TTL_MS` (default 15000) is clamped to 1 s–120 s. +- **Degraded mode:** if Redis is down, or is the no-op client from `redis.js`, locks fall back to an in-process table. A single instance stays correct, and a warning is logged. Cross-instance exclusion returns when Redis does. + +## Metrics + +All on the relay registry (`relayMetricsRegister`): + +| Metric | Labels | +|--------|--------| +| `relay_lock_acquisitions_total` | `scope`, `result` = `acquired` / `contended` / `error` | +| `relay_lock_wait_duration_seconds` | `scope` | +| `relay_lock_hold_duration_seconds` | `scope` | +| `relay_lock_lost_total` | `scope` | +| `relay_circuit_breaker_probe_rejected_total` | — | + +Alerting rules: `docs/alerts/websocket-relay-locking.rules.yml`. + +## Configuration + +| Env var | Default | Notes | +|---------|---------|-------| +| `WS_RELAY_LOCK_TTL_MS` | `15000` | Clamped to 1000–120000 | +| `REDIS_URL` | — | Shared with the rest of the API | + +## Tests + +```bash +npx vitest run src/lib/websocket-relay-lock.test.js src/lib/websocket-relay-recovery.test.js +``` diff --git a/backend/docs/alerts/horizon-client-retry-manager.rules.yml b/backend/docs/alerts/horizon-client-retry-manager.rules.yml new file mode 100644 index 00000000..b1571190 --- /dev/null +++ b/backend/docs/alerts/horizon-client-retry-manager.rules.yml @@ -0,0 +1,82 @@ +# Prometheus alerting rules for the Horizon Client Retry Manager (issue #1438). +# +# Load with: rule_files: ["docs/alerts/horizon-client-retry-manager.rules.yml"] +# Validate: promtool check rules docs/alerts/horizon-client-retry-manager.rules.yml +# +# Metric reference: docs/HORIZON_CLIENT_RETRY_MANAGER.md + +groups: + - name: horizon-client-retry-manager + rules: + - alert: HorizonClientUnhealthy + expr: max(horizon_client_health_state) >= 2 + for: 2m + labels: + severity: critical + annotations: + summary: Horizon client reports unhealthy + description: > + At least one Horizon operation has failed after exhausting its + retries on consecutive calls. Payments cannot be verified against + the Stellar network. Check horizon_client_consecutive_failures by + operation and HorizonClient.getHealth(). + + - alert: HorizonClientDegraded + expr: max(horizon_client_health_state) == 1 + for: 10m + labels: + severity: warning + annotations: + summary: Horizon client has been degraded for 10 minutes + description: > + Some Horizon operations keep failing after retries but are below + the unhealthy threshold. + + - alert: HorizonClientRetriesExhausted + expr: sum by (operation, error_type) (increase(horizon_client_retry_exhausted_total[5m])) > 5 + for: 0m + labels: + severity: warning + annotations: + summary: "Horizon {{ $labels.operation }} is exhausting retries ({{ $labels.error_type }})" + description: > + {{ $value | humanize }} {{ $labels.operation }} calls failed after + every retry in the last 5m with {{ $labels.error_type }}. + + - alert: HorizonClientHighErrorRatio + expr: | + sum(rate(horizon_client_operations_total{result="error"}[10m])) + / + clamp_min(sum(rate(horizon_client_operations_total[10m])), 1e-9) + > 0.25 + and sum(increase(horizon_client_operations_total[10m])) >= 20 + for: 5m + labels: + severity: warning + annotations: + summary: More than 25% of Horizon client operations are failing + description: > + Error ratio over the last 10m is {{ $value | humanizePercentage }}. + + - alert: HorizonClientNoRecentSuccess + expr: (time() - max by (operation) (horizon_client_last_success_timestamp_seconds)) > 900 + for: 5m + labels: + severity: warning + annotations: + summary: "No successful Horizon {{ $labels.operation }} for 15 minutes" + description: > + The operation has succeeded before but has not succeeded for over + 15 minutes. It may be failing or no longer being called. + + - alert: HorizonClientValidationRejections + expr: sum(increase(horizon_client_data_validation_errors_total[10m])) > 0 + for: 0m + labels: + severity: warning + annotations: + summary: Horizon client rejected invalid retry requests + description: > + Callers passed an invalid operation name or callback to the retry + manager. This is a code defect; check logs for + "Horizon client: rejected invalid retry request". diff --git a/backend/docs/alerts/websocket-relay-locking.rules.yml b/backend/docs/alerts/websocket-relay-locking.rules.yml new file mode 100644 index 00000000..eb0d977b --- /dev/null +++ b/backend/docs/alerts/websocket-relay-locking.rules.yml @@ -0,0 +1,47 @@ +# Prometheus alerting rules for WebSocket Relay distributed locking (issue #1455). +# +# Load with: rule_files: ["docs/alerts/websocket-relay-locking.rules.yml"] +# Validate: promtool check rules docs/alerts/websocket-relay-locking.rules.yml +# +# Metric reference: docs/WEBSOCKET_RELAY_CONCURRENCY.md + +groups: + - name: websocket-relay-locking + rules: + - alert: WebSocketRelayLockLost + expr: sum by (scope) (increase(relay_lock_lost_total[10m])) > 0 + for: 0m + labels: + severity: warning + annotations: + summary: "WebSocket relay lost a {{ $labels.scope }} lock mid-work" + description: > + A lease could not be renewed while work was running, so the work was + aborted. Usually Redis latency, a Redis failover or an event loop + stall longer than the lock TTL. + + - alert: WebSocketRelayLockBackendErrors + expr: sum(increase(relay_lock_acquisitions_total{result="error"}[5m])) > 0 + for: 5m + labels: + severity: warning + annotations: + summary: WebSocket relay is falling back to in-process locks + description: > + Redis SET NX is failing, so locks are only enforced per instance. + Cross-instance duplicate delivery is possible until Redis recovers. + + - alert: WebSocketRelayHighLockContention + expr: | + sum by (scope) (rate(relay_lock_acquisitions_total{result="contended"}[10m])) + / + clamp_min(sum by (scope) (rate(relay_lock_acquisitions_total[10m])), 1e-9) + > 0.5 + for: 15m + labels: + severity: info + annotations: + summary: "Over half of {{ $labels.scope }} lock attempts are contended" + description: > + Workers are mostly competing for the same resources. Consider + partitioning work or lowering the number of relay consumers. diff --git a/backend/src/lib/metrics.js b/backend/src/lib/metrics.js index e7090b0f..89cf5ac9 100644 --- a/backend/src/lib/metrics.js +++ b/backend/src/lib/metrics.js @@ -890,6 +890,30 @@ export const horizonClientOperationSuccessRate = new client.Gauge({ labelNames: ["operation"], }); +// Retry Manager alert metrics and health telemetry (Issue #1438) +export const horizonClientRetryExhausted = new client.Counter({ + name: "horizon_client_retry_exhausted_total", + help: "Horizon client operations that failed after exhausting every retry", + labelNames: ["operation", "error_type"], +}); + +export const horizonClientConsecutiveFailures = new client.Gauge({ + name: "horizon_client_consecutive_failures", + help: "Consecutive infrastructure failures per Horizon client operation (reset on success)", + labelNames: ["operation"], +}); + +export const horizonClientLastSuccessTimestamp = new client.Gauge({ + name: "horizon_client_last_success_timestamp_seconds", + help: "Unix timestamp of the last successful Horizon client operation", + labelNames: ["operation"], +}); + +export const horizonClientHealthState = new client.Gauge({ + name: "horizon_client_health_state", + help: "Horizon client retry manager health (0=healthy, 1=degraded, 2=unhealthy)", +}); + export const paymentMatchingOperations = new client.Counter({ name: "payment_matching_operations_total", help: "Total number of payment matching operations", @@ -1111,6 +1135,10 @@ register.registerMetric(horizonClientSerializationErrors); register.registerMetric(horizonClientRateLimitWaitTime); register.registerMetric(horizonClientThroughput); register.registerMetric(horizonClientOperationSuccessRate); +register.registerMetric(horizonClientRetryExhausted); +register.registerMetric(horizonClientConsecutiveFailures); +register.registerMetric(horizonClientLastSuccessTimestamp); +register.registerMetric(horizonClientHealthState); register.registerMetric(paymentMatchingOperations); register.registerMetric(paymentMatchingErrors); register.registerMetric(signatureVerificationOperations); diff --git a/backend/src/lib/stellar/horizon-client.js b/backend/src/lib/stellar/horizon-client.js index 31f1f2a7..1fbba836 100644 --- a/backend/src/lib/stellar/horizon-client.js +++ b/backend/src/lib/stellar/horizon-client.js @@ -31,7 +31,10 @@ export class HorizonClient { this.healthTimeout = options.healthTimeout || DEFAULT_HEALTH_TIMEOUT_MS; this.server = new StellarSdk.Horizon.Server(this.horizonUrl); - this.retryManager = createRetryManager(this.retryDelays); + this.retryManager = createRetryManager(this.retryDelays, { + unhealthyAfterFailures: options.unhealthyAfterFailures, + }); + this.lastHealthCheck = null; // Track active connections horizonClientConnections.inc(); @@ -71,6 +74,7 @@ export class HorizonClient { operation: "health_check", result: isReachable ? "success" : "failure" }, duration); + this._recordHealthCheck(isReachable, duration, isReachable ? null : `http_${response.status}`); return isReachable; } catch (err) { @@ -83,6 +87,7 @@ export class HorizonClient { operation: "health_check", error_type: classifyError(err) }); + this._recordHealthCheck(false, duration, classifyError(err)); return false; } finally { @@ -172,6 +177,37 @@ export class HorizonClient { }; } + /** + * Remember the outcome of the last reachability probe (#1438). + * @private + */ + _recordHealthCheck(reachable, durationSeconds, errorType) { + this.lastHealthCheck = { + reachable, + durationMs: Math.round(durationSeconds * 1000), + errorType, + checkedAt: new Date().toISOString(), + }; + } + + /** + * Health telemetry (#1438): retry-manager per-operation health plus the + * last reachability probe. A failed probe marks an otherwise healthy client + * as degraded. + */ + getHealth() { + const health = this.retryManager.getHealth(); + const status = + health.status === "healthy" && this.lastHealthCheck?.reachable === false + ? "degraded" + : health.status; + return { + ...health, + status, + lastHealthCheck: this.lastHealthCheck, + }; + } + /** * Get the retry manager instance (for advanced use cases) */ diff --git a/backend/src/lib/stellar/horizon-client.test.js b/backend/src/lib/stellar/horizon-client.test.js index 85217b49..c53682ac 100644 --- a/backend/src/lib/stellar/horizon-client.test.js +++ b/backend/src/lib/stellar/horizon-client.test.js @@ -47,6 +47,12 @@ vi.mock('../metrics.js', () => ({ horizonClientOperationDuration: { observe: vi.fn() }, horizonClientRetryAttemptDuration: { observe: vi.fn() }, horizonClientErrorRecovery: { inc: vi.fn() }, + // Added for issues #1437 / #1438 (validation errors and health telemetry) + horizonClientDataValidationErrors: { inc: vi.fn() }, + horizonClientRetryExhausted: { inc: vi.fn() }, + horizonClientConsecutiveFailures: { set: vi.fn() }, + horizonClientLastSuccessTimestamp: { set: vi.fn() }, + horizonClientHealthState: { set: vi.fn() }, })); describe('HorizonClient Unit Tests', () => { @@ -344,4 +350,53 @@ describe('HorizonClient Unit Tests', () => { }); }); }); + describe('getHealth (issue #1438)', () => { + it('is healthy before any operation or probe', () => { + expect(horizonClient.getHealth()).toMatchObject({ + status: 'healthy', + operations: {}, + lastHealthCheck: null, + }); + }); + + it('records a successful reachability probe', async () => { + global.fetch = vi.fn().mockResolvedValue({ ok: true, status: 200 }); + await horizonClient.isReachable(); + + expect(horizonClient.getHealth()).toMatchObject({ + status: 'healthy', + lastHealthCheck: { reachable: true, errorType: null }, + }); + }); + + it('reports degraded when the last probe failed', async () => { + global.fetch = vi.fn().mockRejectedValue(new Error('ECONNREFUSED')); + await horizonClient.isReachable(); + + const health = horizonClient.getHealth(); + expect(health.status).toBe('degraded'); + expect(health.lastHealthCheck).toMatchObject({ reachable: false }); + expect(typeof health.lastHealthCheck.checkedAt).toBe('string'); + }); + + it('reports a non-2xx probe with its HTTP status', async () => { + global.fetch = vi.fn().mockResolvedValue({ ok: false, status: 503 }); + await horizonClient.isReachable(); + + expect(horizonClient.getHealth().lastHealthCheck).toMatchObject({ + reachable: false, + errorType: 'http_503', + }); + }); + + it('surfaces per-operation retry-manager health', async () => { + mockServer.feeStats.mockResolvedValue({ last_ledger_base_fee: '100' }); + await horizonClient.getFeeStats(); + + expect(horizonClient.getHealth().operations.getFeeStats).toMatchObject({ + status: 'healthy', + totalSuccesses: 1, + }); + }); + }); }); diff --git a/backend/src/lib/stellar/horizon-client/error-handler.js b/backend/src/lib/stellar/horizon-client/error-handler.js index 65322d22..998ce11e 100644 --- a/backend/src/lib/stellar/horizon-client/error-handler.js +++ b/backend/src/lib/stellar/horizon-client/error-handler.js @@ -11,6 +11,7 @@ import { RETRYABLE_ERROR_STATUS_CODES, RETRYABLE_MESSAGE_PATTERNS, } from "./constants.js"; +import { sanitizeContext, sanitizeErrorDetail, sanitizeHorizonUrl } from "./sanitizer.js"; /** * Extract HTTP status from error @@ -71,13 +72,14 @@ export function isRetryableError(err) { } /** - * Wrap Horizon errors into descriptive error objects + * Wrap Horizon errors into descriptive error objects. + * + * Context, Horizon error detail and the Horizon URL are sanitized (#1437) so + * secrets, URL credentials and control characters never reach API clients. */ export function handleError(err, operation, context, horizonUrl) { const status = getErrorStatus(err); - const contextStr = Object.entries(context) - .map(([k, v]) => `${k}=${v}`) - .join(", "); + const contextStr = formatContext(context); if (status === 429) { const error = new Error( @@ -98,7 +100,7 @@ export function handleError(err, operation, context, horizonUrl) { } if (status && status >= 400 && status < 500) { - const detail = err?.response?.data?.detail || err.message; + const detail = sanitizeErrorDetail(err?.response?.data?.detail || err?.message); const error = new Error(`Horizon request error (${status}): ${detail}`); error.status = status; error.operation = operation; @@ -116,7 +118,7 @@ export function handleError(err, operation, context, horizonUrl) { // Network / connection errors const error = new Error( - `Unable to connect to Horizon (${horizonUrl}): ${err.message}`, + `Unable to connect to Horizon (${sanitizeHorizonUrl(horizonUrl)}): ${sanitizeErrorDetail(err?.message)}`, ); error.status = 502; error.operation = operation; @@ -127,7 +129,7 @@ export function handleError(err, operation, context, horizonUrl) { * Format context object for logging */ export function formatContext(context) { - return Object.entries(context) + return Object.entries(sanitizeContext(context)) .map(([k, v]) => `${k}=${v}`) .join(", "); } diff --git a/backend/src/lib/stellar/horizon-client/retry-manager.js b/backend/src/lib/stellar/horizon-client/retry-manager.js index 68ff14da..1098b355 100644 --- a/backend/src/lib/stellar/horizon-client/retry-manager.js +++ b/backend/src/lib/stellar/horizon-client/retry-manager.js @@ -1,7 +1,14 @@ /** * Horizon Client Retry Manager * - * Handles retry logic with exponential backoff and configurable delays + * Handles retry logic with exponential backoff and configurable delays. + * + * - Payload sanitization and strict validation (#1437): the retry schedule, + * operation name and callback are validated up front; context, Horizon URL + * and error text are sanitized before they reach logs or error messages. + * - Alert metrics and health telemetry (#1438): retry exhaustion, consecutive + * failures, last-success timestamps and an aggregate health state are + * exported to Prometheus and available through getHealth(). */ import { logger } from "../../logger.js"; @@ -14,23 +21,73 @@ import { horizonClientRetryAttemptDuration, horizonClientErrorRecovery, horizonClientConcurrentOperations, + horizonClientDataValidationErrors, + horizonClientRetryExhausted, + horizonClientConsecutiveFailures, + horizonClientLastSuccessTimestamp, + horizonClientHealthState, } from "../../metrics.js"; import { classifyError, handleError, isRetryableError } from "./error-handler.js"; +import { + sanitizeContext, + sanitizeErrorDetail, + sanitizeHorizonUrl, + validateOperation, + validateOperationName, + validateRetryDelays, +} from "./sanitizer.js"; + +export const HealthStatus = Object.freeze({ + HEALTHY: "healthy", + DEGRADED: "degraded", + UNHEALTHY: "unhealthy", +}); + +const HEALTH_STATE_VALUE = Object.freeze({ + [HealthStatus.HEALTHY]: 0, + [HealthStatus.DEGRADED]: 1, + [HealthStatus.UNHEALTHY]: 2, +}); + +export const DEFAULT_UNHEALTHY_AFTER_FAILURES = 5; +// Bound the per-operation health table; operation names are validated, but a +// caller generating names dynamically must not grow it without limit. +const MAX_TRACKED_OPERATIONS = 64; + +function resolveUnhealthyThreshold(value) { + const raw = Number(value ?? DEFAULT_UNHEALTHY_AFTER_FAILURES); + if (!Number.isInteger(raw) || raw < 1) return DEFAULT_UNHEALTHY_AFTER_FAILURES; + return Math.min(raw, 1_000); +} /** * Retry manager class with configurable retry strategy */ export class RetryManager { - constructor(retryDelays, metricsPrefix = "") { - this.retryDelays = retryDelays; + /** + * @param {number[]} retryDelays Delay (ms) before each retry + * @param {string} [metricsPrefix] + * @param {object} [options] + * @param {number} [options.unhealthyAfterFailures=5] Consecutive failures before "unhealthy" + * @param {() => number} [options.now] Injectable clock + */ + constructor(retryDelays, metricsPrefix = "", options = {}) { + this.retryDelays = validateRetryDelays(retryDelays); this.metricsPrefix = metricsPrefix; this.concurrentOperations = 0; + this.unhealthyAfterFailures = resolveUnhealthyThreshold(options.unhealthyAfterFailures); + this._now = typeof options.now === "function" ? options.now : () => Date.now(); + this.operationHealth = new Map(); } /** * Execute operation with automatic retry logic */ async executeWithRetry(operation, operationName, context = {}, horizonUrl) { + this._validateRequest(operation, operationName); + + const safeContext = sanitizeContext(context); + const safeUrl = sanitizeHorizonUrl(horizonUrl); const totalStartTime = Date.now(); this.concurrentOperations++; horizonClientConcurrentOperations.inc(); @@ -52,6 +109,7 @@ export class RetryManager { ); this._recordSuccess(operationName, totalStartTime); + this._markReachable(operationName, null); // If this was a retry (attempt > 0), record successful recovery if (attempt > 0) { @@ -73,22 +131,34 @@ export class RetryManager { attemptDuration ); - if (!isRetryableError(err) || attempt === this.retryDelays.length) { + const retryable = isRetryableError(err); + if (!retryable || attempt === this.retryDelays.length) { this._recordError(operationName, err); - throw handleError(err, operationName, context, horizonUrl); + if (retryable) { + // Every retry was spent on an infrastructure-level failure. + horizonClientRetryExhausted.inc({ + operation: operationName, + error_type: classifyError(err), + }); + this._markFailure(operationName, err); + } else { + // Horizon answered (e.g. 404 / 400): the dependency is reachable. + this._markReachable(operationName, err); + } + throw handleError(err, operationName, safeContext, safeUrl); } this._recordRetry(operationName); const delay = this.retryDelays[attempt]; - this._logRetry(operationName, attempt + 1, delay, err, context); + this._logRetry(operationName, attempt + 1, delay, err, safeContext); await this._sleep(delay); } } // Should not reach here, but for completeness - throw handleError(lastError, operationName, context, horizonUrl); + throw handleError(lastError, operationName, safeContext, safeUrl); } finally { this.concurrentOperations--; horizonClientConcurrentOperations.dec(); @@ -102,6 +172,30 @@ export class RetryManager { } } + /** + * Strictly validate the request before any work or metrics happen. + * @private + */ + _validateRequest(operation, operationName) { + try { + validateOperationName(operationName); + validateOperation(operation); + } catch (err) { + const label = typeof operationName === "string" && /^[A-Za-z][A-Za-z0-9_]{0,63}$/.test(operationName) + ? operationName + : "invalid"; + horizonClientDataValidationErrors.inc({ + operation: label, + validation_type: err.validationType ?? "unknown", + }); + logger.error( + { operation: label, validationType: err.validationType }, + "Horizon client: rejected invalid retry request" + ); + throw err; + } + } + /** * Record successful operation metrics * @private @@ -140,6 +234,116 @@ export class RetryManager { horizonClientRetries.inc({ operation: operationName }); } + /** + * Get (or create) the health entry for an operation. + * @private + */ + _healthEntry(operationName) { + let entry = this.operationHealth.get(operationName); + if (!entry) { + if (this.operationHealth.size >= MAX_TRACKED_OPERATIONS) { + return null; + } + entry = { + consecutiveFailures: 0, + totalSuccesses: 0, + totalFailures: 0, + lastSuccessAt: null, + lastFailureAt: null, + lastErrorType: null, + }; + this.operationHealth.set(operationName, entry); + } + return entry; + } + + /** + * Horizon responded (success or a non-retryable client error). + * @private + */ + _markReachable(operationName, err) { + const entry = this._healthEntry(operationName); + if (!entry) return; + const now = this._now(); + entry.consecutiveFailures = 0; + if (err) { + entry.lastErrorType = classifyError(err); + } else { + entry.totalSuccesses += 1; + entry.lastSuccessAt = now; + horizonClientLastSuccessTimestamp.set({ operation: operationName }, Math.floor(now / 1000)); + } + horizonClientConsecutiveFailures.set({ operation: operationName }, 0); + this._publishHealthState(); + } + + /** + * Horizon was unreachable / failing after all retries. + * @private + */ + _markFailure(operationName, err) { + const entry = this._healthEntry(operationName); + if (!entry) return; + entry.consecutiveFailures += 1; + entry.totalFailures += 1; + entry.lastFailureAt = this._now(); + entry.lastErrorType = classifyError(err); + horizonClientConsecutiveFailures.set({ operation: operationName }, entry.consecutiveFailures); + this._publishHealthState(); + if (entry.consecutiveFailures === this.unhealthyAfterFailures) { + logger.error( + { + operation: operationName, + consecutiveFailures: entry.consecutiveFailures, + errorType: entry.lastErrorType, + }, + "Horizon client: operation is unhealthy after consecutive failures" + ); + } + } + + /** @private */ + _statusFor(entry) { + if (entry.consecutiveFailures === 0) return HealthStatus.HEALTHY; + if (entry.consecutiveFailures >= this.unhealthyAfterFailures) return HealthStatus.UNHEALTHY; + return HealthStatus.DEGRADED; + } + + /** @private */ + _publishHealthState() { + horizonClientHealthState.set(HEALTH_STATE_VALUE[this.getHealth().status]); + } + + /** + * Health telemetry snapshot for health endpoints and alert context. + * + * @returns {{ status: string, unhealthyAfterFailures: number, concurrentOperations: number, + * operations: Record }} + */ + getHealth() { + const operations = {}; + let worst = HealthStatus.HEALTHY; + for (const [name, entry] of this.operationHealth) { + const status = this._statusFor(entry); + if (HEALTH_STATE_VALUE[status] > HEALTH_STATE_VALUE[worst]) worst = status; + operations[name] = { + status, + consecutiveFailures: entry.consecutiveFailures, + totalSuccesses: entry.totalSuccesses, + totalFailures: entry.totalFailures, + lastSuccessAt: entry.lastSuccessAt ? new Date(entry.lastSuccessAt).toISOString() : null, + lastFailureAt: entry.lastFailureAt ? new Date(entry.lastFailureAt).toISOString() : null, + lastErrorType: entry.lastErrorType, + }; + } + return { + status: worst, + unhealthyAfterFailures: this.unhealthyAfterFailures, + concurrentOperations: this.concurrentOperations, + operations, + }; + } + /** * Log retry attempt * @private @@ -150,8 +354,8 @@ export class RetryManager { operation: operationName, attempt, delayMs, - error: error.message, - ...context + error: sanitizeErrorDetail(error?.message), + ...sanitizeContext(context), }, "Horizon client: retrying operation after error" ); @@ -180,6 +384,6 @@ export class RetryManager { /** * Create a retry manager with default configuration */ -export function createRetryManager(retryDelays) { - return new RetryManager(retryDelays); +export function createRetryManager(retryDelays, options) { + return new RetryManager(retryDelays, "", options); } diff --git a/backend/src/lib/stellar/horizon-client/retry-manager.test.js b/backend/src/lib/stellar/horizon-client/retry-manager.test.js new file mode 100644 index 00000000..0d78e829 --- /dev/null +++ b/backend/src/lib/stellar/horizon-client/retry-manager.test.js @@ -0,0 +1,232 @@ +/** + * Retry Manager tests + * + * - Strict validation and payload sanitization (#1437) + * - Prometheus alert metrics and health telemetry (#1438) + */ + +import { describe, it, expect, beforeEach, vi } from 'vitest'; + +vi.mock('../../logger.js', () => ({ + logger: { warn: vi.fn(), error: vi.fn(), info: vi.fn(), debug: vi.fn() }, +})); + +import { logger } from '../../logger.js'; +import { + horizonClientDataValidationErrors, + horizonClientRetryExhausted, + horizonClientConsecutiveFailures, + horizonClientLastSuccessTimestamp, + horizonClientHealthState, + horizonClientConcurrentOperations, +} from '../../metrics.js'; +import { DEFAULT_UNHEALTHY_AFTER_FAILURES, HealthStatus, RetryManager, createRetryManager } from './retry-manager.js'; +import { HORIZON_VALIDATION_ERROR, REDACTED } from './sanitizer.js'; + +const SECRET_SEED = `S${'B'.repeat(55)}`; +const URL = 'https://horizon-testnet.stellar.org'; + +async function metricValue(metric, labels = {}) { + const { values } = await metric.get(); + const match = values.find((v) => Object.entries(labels).every(([k, val]) => v.labels[k] === val)); + return match ? match.value : undefined; +} + +function networkError(message = 'socket hang up') { + const err = new Error(message); + err.code = 'ECONNRESET'; + return err; +} + +function httpError(status) { + const err = new Error(`HTTP ${status}`); + err.response = { status }; + return err; +} + +beforeEach(() => { + vi.clearAllMocks(); + horizonClientDataValidationErrors.reset(); + horizonClientRetryExhausted.reset(); + horizonClientConsecutiveFailures.reset(); + horizonClientLastSuccessTimestamp.reset(); + horizonClientHealthState.reset(); + horizonClientConcurrentOperations.reset(); +}); + +describe('RetryManager validation (#1437)', () => { + it('rejects an invalid retry schedule at construction', () => { + expect(() => new RetryManager([-5])).toThrow(expect.objectContaining({ code: HORIZON_VALIDATION_ERROR })); + expect(() => new RetryManager(undefined)).toThrow(expect.objectContaining({ validationType: 'retry_delays' })); + expect(() => createRetryManager(Array(20).fill(1))).toThrow(TypeError); + }); + + it('freezes the schedule so callers cannot mutate it later', () => { + const delays = [0, 0]; + const manager = new RetryManager(delays); + delays.push(99_999); + expect(manager.getConfig()).toMatchObject({ retryDelays: [0, 0], maxAttempts: 3 }); + }); + + it('rejects an invalid operation name before calling Horizon', async () => { + const manager = new RetryManager([0]); + const op = vi.fn(); + + await expect(manager.executeWithRetry(op, 'load account\n', {}, URL)).rejects.toMatchObject({ + code: HORIZON_VALIDATION_ERROR, + validationType: 'operation_name', + }); + expect(op).not.toHaveBeenCalled(); + expect(await metricValue(horizonClientDataValidationErrors, { operation: 'invalid', validation_type: 'operation_name' })).toBe(1); + expect(manager.getConfig().currentConcurrentOperations).toBe(0); + }); + + it('rejects a non-function operation', async () => { + const manager = new RetryManager([0]); + await expect(manager.executeWithRetry('nope', 'loadAccount', {}, URL)).rejects.toMatchObject({ + validationType: 'operation', + }); + expect(await metricValue(horizonClientDataValidationErrors, { operation: 'loadAccount', validation_type: 'operation' })).toBe(1); + expect(logger.error).toHaveBeenCalled(); + }); + + it('sanitizes context and error text in retry logs', async () => { + const manager = new RetryManager([0]); + const op = vi.fn() + .mockRejectedValueOnce(networkError(`boom\r\nforged ${SECRET_SEED}`)) + .mockResolvedValue('ok'); + + await manager.executeWithRetry(op, 'loadAccount', { accountId: 'GABC', secret: SECRET_SEED, 'x\ny': 1 }, URL); + + const [fields] = logger.warn.mock.calls[0]; + expect(fields).toMatchObject({ operation: 'loadAccount', attempt: 1, accountId: 'GABC', secret: REDACTED }); + expect(fields.error).toBe(`boom forged ${REDACTED}`); + expect(fields).not.toHaveProperty('x\ny'); + expect(JSON.stringify(logger.warn.mock.calls)).not.toContain(SECRET_SEED); + }); + + it('never leaks URL credentials or secrets in the thrown error', async () => { + const manager = new RetryManager([]); + const op = vi.fn().mockRejectedValue(networkError(`refused ${SECRET_SEED}`)); + + const err = await manager + .executeWithRetry(op, 'loadAccount', { seed: SECRET_SEED }, 'https://u:p@horizon.example.com/?key=1') + .catch((e) => e); + + expect(err.message).toBe(`Unable to connect to Horizon (https://horizon.example.com): refused ${REDACTED}`); + expect(err.status).toBe(502); + }); +}); + +describe('RetryManager alert metrics and health telemetry (#1438)', () => { + it('starts healthy with no tracked operations', () => { + const manager = new RetryManager([0]); + expect(manager.getHealth()).toEqual({ + status: HealthStatus.HEALTHY, + unhealthyAfterFailures: DEFAULT_UNHEALTHY_AFTER_FAILURES, + concurrentOperations: 0, + operations: {}, + }); + }); + + it('records last success and keeps the operation healthy', async () => { + const manager = new RetryManager([0], '', { now: () => 1_700_000_000_000 }); + await manager.executeWithRetry(() => Promise.resolve('ok'), 'getFeeStats', {}, URL); + + const health = manager.getHealth(); + expect(health.status).toBe(HealthStatus.HEALTHY); + expect(health.operations.getFeeStats).toMatchObject({ + status: 'healthy', + consecutiveFailures: 0, + totalSuccesses: 1, + lastSuccessAt: new Date(1_700_000_000_000).toISOString(), + }); + expect(await metricValue(horizonClientLastSuccessTimestamp, { operation: 'getFeeStats' })).toBe(1_700_000_000); + expect(await metricValue(horizonClientHealthState)).toBe(0); + }); + + it('counts retry exhaustion and degrades, then turns unhealthy at the threshold', async () => { + const manager = new RetryManager([0], '', { unhealthyAfterFailures: 2 }); + const failing = () => Promise.reject(networkError()); + + await manager.executeWithRetry(failing, 'loadAccount', {}, URL).catch(() => {}); + expect(manager.getHealth().status).toBe(HealthStatus.DEGRADED); + expect(await metricValue(horizonClientHealthState)).toBe(1); + expect(await metricValue(horizonClientConsecutiveFailures, { operation: 'loadAccount' })).toBe(1); + + await manager.executeWithRetry(failing, 'loadAccount', {}, URL).catch(() => {}); + expect(manager.getHealth().status).toBe(HealthStatus.UNHEALTHY); + expect(manager.getHealth().operations.loadAccount).toMatchObject({ + consecutiveFailures: 2, + totalFailures: 2, + lastErrorType: 'network_error', + }); + expect(await metricValue(horizonClientHealthState)).toBe(2); + expect(await metricValue(horizonClientRetryExhausted, { operation: 'loadAccount', error_type: 'network_error' })).toBe(2); + expect(logger.error).toHaveBeenCalledWith( + expect.objectContaining({ operation: 'loadAccount', consecutiveFailures: 2 }), + 'Horizon client: operation is unhealthy after consecutive failures', + ); + }); + + it('recovers to healthy after a success', async () => { + const manager = new RetryManager([], '', { unhealthyAfterFailures: 1 }); + await manager.executeWithRetry(() => Promise.reject(httpError(503)), 'fetchTransaction', {}, URL).catch(() => {}); + expect(manager.getHealth().status).toBe(HealthStatus.UNHEALTHY); + + await manager.executeWithRetry(() => Promise.resolve({}), 'fetchTransaction', {}, URL); + expect(manager.getHealth().status).toBe(HealthStatus.HEALTHY); + expect(await metricValue(horizonClientConsecutiveFailures, { operation: 'fetchTransaction' })).toBe(0); + expect(await metricValue(horizonClientHealthState)).toBe(0); + }); + + it('does not treat non-retryable client errors as an outage', async () => { + const manager = new RetryManager([0, 0]); + const op = vi.fn().mockRejectedValue(httpError(404)); + + await expect(manager.executeWithRetry(op, 'loadAccount', { accountId: 'G1' }, URL)).rejects.toThrow('not found'); + + expect(op).toHaveBeenCalledTimes(1); + expect(manager.getHealth().operations.loadAccount).toMatchObject({ + status: 'healthy', + consecutiveFailures: 0, + lastErrorType: 'not_found', + }); + expect(await metricValue(horizonClientRetryExhausted, { operation: 'loadAccount', error_type: 'not_found' })).toBeUndefined(); + }); + + it('does not count exhaustion when a retry succeeds', async () => { + const manager = new RetryManager([0, 0]); + const op = vi.fn().mockRejectedValueOnce(httpError(429)).mockResolvedValue('ok'); + + await expect(manager.executeWithRetry(op, 'fetchPayments', {}, URL)).resolves.toBe('ok'); + expect(op).toHaveBeenCalledTimes(2); + expect((await horizonClientRetryExhausted.get()).values).toHaveLength(0); + expect(manager.getHealth().operations.fetchPayments.status).toBe('healthy'); + }); + + it('reports the worst operation as the overall status', async () => { + const manager = new RetryManager([], '', { unhealthyAfterFailures: 3 }); + await manager.executeWithRetry(() => Promise.resolve(1), 'getFeeStats', {}, URL); + await manager.executeWithRetry(() => Promise.reject(networkError()), 'loadAccount', {}, URL).catch(() => {}); + + const health = manager.getHealth(); + expect(health.status).toBe(HealthStatus.DEGRADED); + expect(health.operations.getFeeStats.status).toBe('healthy'); + expect(health.operations.loadAccount.status).toBe('degraded'); + }); + + it('falls back to the default threshold for invalid options', () => { + expect(new RetryManager([], '', { unhealthyAfterFailures: 0 }).unhealthyAfterFailures).toBe(DEFAULT_UNHEALTHY_AFTER_FAILURES); + expect(new RetryManager([], '', { unhealthyAfterFailures: 'x' }).unhealthyAfterFailures).toBe(DEFAULT_UNHEALTHY_AFTER_FAILURES); + expect(new RetryManager([], '', { unhealthyAfterFailures: 2.5 }).unhealthyAfterFailures).toBe(DEFAULT_UNHEALTHY_AFTER_FAILURES); + }); + + it('bounds the number of tracked operations', async () => { + const manager = new RetryManager([]); + for (let i = 0; i < 70; i += 1) { + await manager.executeWithRetry(() => Promise.resolve(i), `op${i}`, {}, URL); + } + expect(Object.keys(manager.getHealth().operations)).toHaveLength(64); + }); +}); diff --git a/backend/src/lib/stellar/horizon-client/sanitizer.js b/backend/src/lib/stellar/horizon-client/sanitizer.js new file mode 100644 index 00000000..746aad61 --- /dev/null +++ b/backend/src/lib/stellar/horizon-client/sanitizer.js @@ -0,0 +1,191 @@ +/** + * Horizon Client Sanitizer + * + * Payload sanitization and strict validation for the Retry Manager + * (issue #1437). + * + * Everything the retry manager receives from callers (operation name, retry + * schedule, log context) and everything it echoes back (Horizon URL, error + * detail returned by Horizon) can end up in logs, Prometheus labels and API + * error messages. This module makes sure that data is: + * - well-formed (strict validation, fail fast with a typed error), + * - bounded (key counts and string lengths are capped), + * - safe (secrets redacted, control characters stripped, URL + * credentials and query strings removed). + */ + +/** Error code for invalid input passed to the retry manager. */ +export const HORIZON_VALIDATION_ERROR = "HORIZON_CLIENT_VALIDATION_ERROR"; + +export const MAX_RETRY_ATTEMPTS = 10; +export const MAX_RETRY_DELAY_MS = 60_000; +export const MAX_CONTEXT_KEYS = 16; +export const MAX_CONTEXT_VALUE_LENGTH = 256; +export const MAX_ERROR_DETAIL_LENGTH = 512; +export const REDACTED = "[REDACTED]"; + +// Operation names become Prometheus label values and log fields. +const OPERATION_NAME_PATTERN = /^[A-Za-z][A-Za-z0-9_]{0,63}$/; +const CONTEXT_KEY_PATTERN = /^[A-Za-z0-9_.-]{1,64}$/; +const SENSITIVE_KEY_PATTERN = + /(secret|seed|password|passphrase|token|private|signature|authorization|api[_-]?key|mnemonic|cookie)/i; +// Stellar secret seeds: "S" + 55 base32 characters. +const STELLAR_SECRET_PATTERN = /\bS[A-Z2-7]{55}\b/g; +// Bearer / basic credentials that sometimes leak into error text. +const AUTH_HEADER_PATTERN = /\b(Bearer|Basic)\s+[A-Za-z0-9._~+/=-]+/gi; +// eslint-disable-next-line no-control-regex +const CONTROL_CHARS_PATTERN = /[\u0000-\u001f\u007f-\u009f

]+/g; + +/** + * Build a typed validation error. + */ +export function createValidationError(message, validationType) { + const error = new TypeError(message); + error.code = HORIZON_VALIDATION_ERROR; + error.validationType = validationType; + error.status = 500; + return error; +} + +/** + * Remove control characters, redact secrets and cap the length of a string. + */ +export function sanitizeString(value, maxLength = MAX_CONTEXT_VALUE_LENGTH) { + const text = String(value ?? "") + .replace(CONTROL_CHARS_PATTERN, " ") + .replace(STELLAR_SECRET_PATTERN, REDACTED) + .replace(AUTH_HEADER_PATTERN, `$1 ${REDACTED}`) + .trim(); + if (text.length <= maxLength) return text; + return `${text.slice(0, Math.max(0, maxLength - 1))}…`; +} + +function stringifyValue(value) { + if (value === null) return "null"; + if (value === undefined) return "undefined"; + const type = typeof value; + if (type === "string") return value; + if (type === "number" || type === "boolean" || type === "bigint") return String(value); + if (type === "function" || type === "symbol") return `[${type}]`; + if (value instanceof Date) return Number.isNaN(value.getTime()) ? "Invalid Date" : value.toISOString(); + try { + return JSON.stringify(value, (_key, nested) => (typeof nested === "bigint" ? String(nested) : nested)); + } catch { + return "[unserializable]"; + } +} + +function isPlainObject(value) { + if (value === null || typeof value !== "object" || Array.isArray(value)) return false; + const proto = Object.getPrototypeOf(value); + return proto === Object.prototype || proto === null; +} + +/** + * Turn arbitrary caller context into a small, flat, log-safe object. + * + * - Non plain-object input becomes {}. + * - Keys that are not simple identifiers (or "__proto__"-style keys) are dropped. + * - At most MAX_CONTEXT_KEYS keys are kept. + * - Values under sensitive keys are replaced with "[REDACTED]". + * - Every other value is stringified, stripped of control characters, + * scrubbed of secret seeds and truncated. + * + * @param {unknown} context + * @returns {Record} + */ +export function sanitizeContext(context) { + if (!isPlainObject(context)) return {}; + const sanitized = Object.create(null); + let kept = 0; + for (const key of Object.keys(context)) { + if (kept >= MAX_CONTEXT_KEYS) break; + if (!CONTEXT_KEY_PATTERN.test(key) || key === "__proto__" || key === "constructor" || key === "prototype") { + continue; + } + sanitized[key] = SENSITIVE_KEY_PATTERN.test(key) + ? REDACTED + : sanitizeString(stringifyValue(context[key])); + kept += 1; + } + return { ...sanitized }; +} + +/** + * Strip credentials, query string and fragment from the Horizon URL before it + * is used in a log line or error message. + */ +export function sanitizeHorizonUrl(url) { + if (typeof url !== "string" || url.trim() === "") return "[unknown-horizon-url]"; + try { + const parsed = new URL(url); + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") return "[invalid-horizon-url]"; + const path = parsed.pathname === "/" ? "" : parsed.pathname; + return sanitizeString(`${parsed.protocol}//${parsed.host}${path}`); + } catch { + return "[invalid-horizon-url]"; + } +} + +/** + * Sanitize an error detail / message coming back from Horizon or the network + * layer before it is surfaced to API clients or logs. + */ +export function sanitizeErrorDetail(detail) { + return sanitizeString(stringifyValue(detail), MAX_ERROR_DETAIL_LENGTH); +} + +/** + * Validate an operation name used for metrics labels and logs. + * + * @throws {TypeError} code HORIZON_CLIENT_VALIDATION_ERROR + */ +export function validateOperationName(operationName) { + if (typeof operationName !== "string" || !OPERATION_NAME_PATTERN.test(operationName)) { + throw createValidationError( + "Horizon client operation name must match /^[A-Za-z][A-Za-z0-9_]{0,63}$/", + "operation_name", + ); + } + return operationName; +} + +/** + * Validate the operation callback. + * + * @throws {TypeError} code HORIZON_CLIENT_VALIDATION_ERROR + */ +export function validateOperation(operation) { + if (typeof operation !== "function") { + throw createValidationError("Horizon client operation must be a function", "operation"); + } + return operation; +} + +/** + * Validate a retry schedule and return a frozen copy of it. + * + * @param {unknown} retryDelays + * @returns {ReadonlyArray} + * @throws {TypeError} code HORIZON_CLIENT_VALIDATION_ERROR + */ +export function validateRetryDelays(retryDelays) { + if (!Array.isArray(retryDelays)) { + throw createValidationError("retryDelays must be an array of milliseconds", "retry_delays"); + } + if (retryDelays.length > MAX_RETRY_ATTEMPTS) { + throw createValidationError( + `retryDelays allows at most ${MAX_RETRY_ATTEMPTS} retries`, + "retry_delays", + ); + } + for (const delay of retryDelays) { + if (!Number.isInteger(delay) || delay < 0 || delay > MAX_RETRY_DELAY_MS) { + throw createValidationError( + `retryDelays entries must be integers between 0 and ${MAX_RETRY_DELAY_MS}`, + "retry_delays", + ); + } + } + return Object.freeze([...retryDelays]); +} diff --git a/backend/src/lib/stellar/horizon-client/sanitizer.test.js b/backend/src/lib/stellar/horizon-client/sanitizer.test.js new file mode 100644 index 00000000..82b3ab9f --- /dev/null +++ b/backend/src/lib/stellar/horizon-client/sanitizer.test.js @@ -0,0 +1,236 @@ +/** + * Tests for Horizon Client payload sanitization and strict validation (#1437) + */ + +import { describe, it, expect } from 'vitest'; +import { + HORIZON_VALIDATION_ERROR, + MAX_CONTEXT_KEYS, + MAX_CONTEXT_VALUE_LENGTH, + MAX_ERROR_DETAIL_LENGTH, + REDACTED, + sanitizeContext, + sanitizeErrorDetail, + sanitizeHorizonUrl, + sanitizeString, + validateOperation, + validateOperationName, + validateRetryDelays, +} from './sanitizer.js'; +import { formatContext, handleError } from './error-handler.js'; + +const SECRET_SEED = `S${'A'.repeat(55)}`; + +describe('sanitizeString', () => { + it('leaves ordinary text untouched', () => { + expect(sanitizeString('loadAccount failed')).toBe('loadAccount failed'); + }); + + it('strips control characters used for log injection', () => { + expect(sanitizeString('ok\r\n[ERROR] forged line\u0000')).toBe('ok [ERROR] forged line'); + expect(sanitizeString('a
b')).toBe('a b'); + }); + + it('redacts Stellar secret seeds and auth headers', () => { + expect(sanitizeString(`seed=${SECRET_SEED}`)).toBe(`seed=${REDACTED}`); + expect(sanitizeString('Authorization: Bearer abc.def-123')).toBe(`Authorization: Bearer ${REDACTED}`); + }); + + it('does not redact public account ids', () => { + const account = `G${'A'.repeat(55)}`; + expect(sanitizeString(account)).toBe(account); + }); + + it('truncates long values with an ellipsis', () => { + const out = sanitizeString('x'.repeat(1000), 10); + expect(out).toHaveLength(10); + expect(out.endsWith('…')).toBe(true); + }); + + it('handles null and undefined', () => { + expect(sanitizeString(null)).toBe(''); + expect(sanitizeString(undefined)).toBe(''); + }); +}); + +describe('sanitizeContext', () => { + it('returns {} for non plain objects', () => { + expect(sanitizeContext(null)).toEqual({}); + expect(sanitizeContext('accountId=1')).toEqual({}); + expect(sanitizeContext(['a'])).toEqual({}); + expect(sanitizeContext(new Map([['a', 1]]))).toEqual({}); + }); + + it('stringifies values, including nested objects and bigint', () => { + expect( + sanitizeContext({ + accountId: 'GABC', + limit: 10, + options: { order: 'desc', limit: 5 }, + amount: 10n, + when: new Date('2026-01-01T00:00:00Z'), + missing: undefined, + }), + ).toEqual({ + accountId: 'GABC', + limit: '10', + options: '{"order":"desc","limit":5}', + amount: '10', + when: '2026-01-01T00:00:00.000Z', + missing: 'undefined', + }); + }); + + it('redacts values under sensitive keys', () => { + const out = sanitizeContext({ + secretKey: SECRET_SEED, + api_key: 'k', + authorization: 'Bearer x', + seedPhrase: 'words', + txHash: 'abc', + }); + expect(out).toEqual({ + secretKey: REDACTED, + api_key: REDACTED, + authorization: REDACTED, + seedPhrase: REDACTED, + txHash: 'abc', + }); + }); + + it('redacts secrets hidden inside non-sensitive keys', () => { + expect(sanitizeContext({ memo: `pay ${SECRET_SEED}` })).toEqual({ memo: `pay ${REDACTED}` }); + }); + + it('drops malformed and prototype-polluting keys', () => { + const input = JSON.parse('{"__proto__":{"polluted":true},"bad key":1,"a\\nb":2,"ok":3}'); + const out = sanitizeContext(input); + expect(out).toEqual({ ok: '3' }); + expect({}.polluted).toBeUndefined(); + }); + + it('caps the number of keys and value length', () => { + const big = Object.fromEntries( + Array.from({ length: MAX_CONTEXT_KEYS + 10 }, (_, i) => [`k${i}`, 'v'.repeat(1000)]), + ); + const out = sanitizeContext(big); + expect(Object.keys(out)).toHaveLength(MAX_CONTEXT_KEYS); + expect(out.k0).toHaveLength(MAX_CONTEXT_VALUE_LENGTH); + }); + + it('survives circular structures', () => { + const circular = { name: 'x' }; + circular.self = circular; + expect(sanitizeContext({ circular })).toEqual({ circular: '[unserializable]' }); + }); +}); + +describe('sanitizeHorizonUrl', () => { + it('keeps scheme, host and path', () => { + expect(sanitizeHorizonUrl('https://horizon-testnet.stellar.org')).toBe('https://horizon-testnet.stellar.org'); + expect(sanitizeHorizonUrl('https://example.com/horizon')).toBe('https://example.com/horizon'); + }); + + it('removes credentials, query string and fragment', () => { + expect(sanitizeHorizonUrl('https://user:pass@example.com:8000/h?api_key=abc#x')).toBe( + 'https://example.com:8000/h', + ); + }); + + it('rejects non-http schemes and garbage', () => { + expect(sanitizeHorizonUrl('javascript:alert(1)')).toBe('[invalid-horizon-url]'); + expect(sanitizeHorizonUrl('not a url')).toBe('[invalid-horizon-url]'); + expect(sanitizeHorizonUrl(undefined)).toBe('[unknown-horizon-url]'); + }); +}); + +describe('sanitizeErrorDetail', () => { + it('caps detail length and stringifies objects', () => { + expect(sanitizeErrorDetail('x'.repeat(5000))).toHaveLength(MAX_ERROR_DETAIL_LENGTH); + expect(sanitizeErrorDetail({ extras: { result_codes: 'tx_failed' } })).toBe( + '{"extras":{"result_codes":"tx_failed"}}', + ); + }); +}); + +describe('strict validation', () => { + it('accepts valid operation names', () => { + expect(validateOperationName('loadAccount')).toBe('loadAccount'); + expect(validateOperationName('fetch_payments_2')).toBe('fetch_payments_2'); + }); + + it.each(['', '1load', 'load account', 'load\nAccount', 'x'.repeat(65), null, 42])( + 'rejects invalid operation name %p', + (name) => { + expect(() => validateOperationName(name)).toThrow( + expect.objectContaining({ code: HORIZON_VALIDATION_ERROR, validationType: 'operation_name' }), + ); + }, + ); + + it('requires the operation to be a function', () => { + const fn = () => 1; + expect(validateOperation(fn)).toBe(fn); + expect(() => validateOperation('call')).toThrow(expect.objectContaining({ validationType: 'operation' })); + }); + + it('returns a frozen copy of a valid retry schedule', () => { + const input = [150, 500]; + const out = validateRetryDelays(input); + expect(out).toEqual([150, 500]); + expect(Object.isFrozen(out)).toBe(true); + input.push(1); + expect(out).toHaveLength(2); + expect(validateRetryDelays([])).toEqual([]); + }); + + it.each([ + ['not an array', '150'], + ['negative', [-1]], + ['fractional', [1.5]], + ['NaN', [Number.NaN]], + ['Infinity', [Infinity]], + ['too large', [60_001]], + ['string entry', ['100']], + ['too many retries', Array(11).fill(1)], + ])('rejects a retry schedule that is %s', (_label, delays) => { + expect(() => validateRetryDelays(delays)).toThrow( + expect.objectContaining({ code: HORIZON_VALIDATION_ERROR, validationType: 'retry_delays' }), + ); + }); +}); + +describe('error-handler sanitization', () => { + it('formatContext sanitizes keys and values', () => { + expect(formatContext({ accountId: 'G1', secret: SECRET_SEED, 'bad key': 1 })).toBe( + `accountId=G1, secret=${REDACTED}`, + ); + }); + + it('does not leak secrets in 404 messages', () => { + const err = handleError({ response: { status: 404 } }, 'loadAccount', { seed: SECRET_SEED }, 'https://h'); + expect(err.message).not.toContain(SECRET_SEED); + expect(err.status).toBe(404); + }); + + it('sanitizes Horizon error detail on 4xx', () => { + const err = handleError( + { response: { status: 400, data: { detail: `bad\r\nX-Injected: 1 ${SECRET_SEED}` } } }, + 'fetchPayments', + {}, + 'https://h', + ); + expect(err.message).toBe(`Horizon request error (400): bad X-Injected: 1 ${REDACTED}`); + }); + + it('strips credentials from the Horizon URL on network errors', () => { + const err = handleError( + { code: 'ECONNREFUSED', message: 'connect ECONNREFUSED' }, + 'loadAccount', + {}, + 'https://admin:hunter2@horizon.example.com/?token=abc', + ); + expect(err.message).toBe('Unable to connect to Horizon (https://horizon.example.com): connect ECONNREFUSED'); + expect(err.status).toBe(502); + }); +}); diff --git a/backend/src/lib/websocket-relay-lock.js b/backend/src/lib/websocket-relay-lock.js new file mode 100644 index 00000000..471d96f4 --- /dev/null +++ b/backend/src/lib/websocket-relay-lock.js @@ -0,0 +1,379 @@ +/** + * websocket-relay-lock.js + * + * Distributed concurrency control for the WebSocket Relay Service + * (issue #1455). + * + * Problem + * ------- + * Several relay instances run side by side. Without coordination they can: + * - deliver the same relay event twice (two instances pick it up after a + * reconnect or a DLQ replay), + * - drain the dead letter queue concurrently and re-send every entry once + * per instance, + * - stampede the upstream on reconnect, because every caller inside one + * process starts its own reconnect loop. + * + * Solution + * -------- + * - Redis lease: SET NX PX — atomic acquire across + * instances. + * - Owner-only release and renewal: compare-and-delete / compare-and-pexpire + * Lua scripts, so an instance can never release or extend a lease that + * expired and was re-acquired by someone else. + * - Heartbeat renewal while the critical section runs. If renewal fails the + * lease is treated as lost and the work is told to stop via AbortSignal, + * instead of silently continuing without mutual exclusion. + * - Bounded waiting with jittered polling (no unbounded spin, no herd). + * - In-process fallback when Redis is unavailable, so a single instance + * stays correct instead of failing open. + * - Single-flight: concurrent callers inside one process share one + * in-flight promise per key (used for reconnects). + * + * Keys are namespaced by an allow-listed scope and the caller-supplied + * resource id is SHA-256 hashed, so it cannot inject Redis key separators or + * create unbounded key sizes. + */ + +import { createHash, randomUUID } from "node:crypto"; +import { connectRedisClient } from "./redis.js"; +import { logger } from "./logger.js"; +import { + relayLockAcquisitionsTotal, + relayLockHoldDuration, + relayLockLostTotal, + relayLockWaitDuration, +} from "./websocket-relay-metrics.js"; + +export const DEFAULT_RELAY_LOCK_TTL_MS = 15_000; +const MIN_RELAY_LOCK_TTL_MS = 1_000; +const MAX_RELAY_LOCK_TTL_MS = 120_000; +const MAX_WAIT_MS = 30_000; +const DEFAULT_RETRY_INTERVAL_MS = 100; +const MAX_RESOURCE_LENGTH = 512; +const LOCK_KEY_PREFIX = "lock:ws-relay"; + +/** Scopes the relay is allowed to lock. Anything else is rejected. */ +export const RelayLockScope = Object.freeze({ + EVENT: "event", + DLQ_DRAIN: "dlq-drain", + RECONNECT: "reconnect", + CONNECTION: "connection", +}); + +const VALID_SCOPES = new Set(Object.values(RelayLockScope)); + +/** Error code for work rejected because another holder owns the lock. */ +export const RELAY_LOCK_BUSY = "RELAY_LOCK_BUSY"; +/** Error code raised to the critical section when its lease was lost. */ +export const RELAY_LOCK_LOST = "RELAY_LOCK_LOST"; + +export const RELEASE_RELAY_LOCK_SCRIPT = + 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) else return 0 end'; + +export const RENEW_RELAY_LOCK_SCRIPT = + 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("pexpire", KEYS[1], ARGV[2]) else return 0 end'; + +/** In-process fallback: key -> { token, expiresAt } */ +const localLocks = new Map(); + +export function resolveRelayLockTtlMs(value, env = process.env) { + const raw = Number(value ?? env.WS_RELAY_LOCK_TTL_MS); + if (!Number.isFinite(raw)) return DEFAULT_RELAY_LOCK_TTL_MS; + return Math.min(Math.max(Math.trunc(raw), MIN_RELAY_LOCK_TTL_MS), MAX_RELAY_LOCK_TTL_MS); +} + +function resolveWaitMs(value) { + const raw = Number(value ?? 0); + if (!Number.isFinite(raw) || raw <= 0) return 0; + return Math.min(Math.trunc(raw), MAX_WAIT_MS); +} + +/** + * Build an injection-safe lock key. + * + * @param {string} scope One of RelayLockScope + * @param {string} resource Event id, connection id, upstream URL, ... + */ +export function buildRelayLockKey(scope, resource) { + if (!VALID_SCOPES.has(scope)) { + throw new TypeError(`Unknown relay lock scope: ${String(scope).slice(0, 32)}`); + } + if (typeof resource !== "string" || resource.trim() === "") { + throw new TypeError("resource is required to build a relay lock key"); + } + if (resource.length > MAX_RESOURCE_LENGTH) { + throw new RangeError(`resource must be at most ${MAX_RESOURCE_LENGTH} characters`); + } + const digest = createHash("sha256").update(resource).digest("hex"); + return `${LOCK_KEY_PREFIX}:${scope}:${digest}`; +} + +function scopeFromKey(key) { + return String(key).split(":")[2] ?? "unknown"; +} + +function acquireLocal(key, token, ttlMs, now = Date.now()) { + const existing = localLocks.get(key); + if (existing && existing.expiresAt > now) return false; + localLocks.set(key, { token, expiresAt: now + ttlMs }); + return true; +} + +function releaseLocal(key, token) { + const existing = localLocks.get(key); + if (existing && existing.token === token) { + localLocks.delete(key); + return true; + } + return false; +} + +function renewLocal(key, token, ttlMs, now = Date.now()) { + const existing = localLocks.get(key); + if (!existing || existing.token !== token || existing.expiresAt <= now) return false; + existing.expiresAt = now + ttlMs; + return true; +} + +async function resolveRedis(getRedis) { + try { + const client = await getRedis(); + // connectRedisClient() returns a no-op client (isOpen:false) when Redis is + // unreachable. Its SET reports success for everyone, so it must not be + // trusted as a distributed lock backend. + if (!client || client.isOpen === false || typeof client.sendCommand !== "function") { + return null; + } + return client; + } catch (err) { + logger.warn({ err: err?.message }, "WebSocket relay lock: Redis unavailable, using local lock"); + return null; + } +} + +function buildLease({ key, token, ttl, backend, client }) { + const scope = scopeFromKey(key); + return { + key, + token, + backend, + ttlMs: ttl, + acquiredAt: Date.now(), + async renew() { + if (backend === "local") return renewLocal(key, token, ttl); + try { + const renewed = await client.sendCommand([ + "EVAL", + RENEW_RELAY_LOCK_SCRIPT, + "1", + key, + token, + String(ttl), + ]); + return Number(renewed) === 1; + } catch (err) { + logger.error({ err: err?.message, scope }, "WebSocket relay lock renewal failed"); + return false; + } + }, + async release() { + if (backend === "local") return releaseLocal(key, token); + try { + const released = await client.sendCommand(["EVAL", RELEASE_RELAY_LOCK_SCRIPT, "1", key, token]); + if (Number(released) !== 1) { + logger.warn({ scope, ttlMs: ttl }, "WebSocket relay lock expired before release"); + return false; + } + return true; + } catch (err) { + // The lease still expires via PX, so a failed release is not fatal. + logger.error({ err: err?.message, scope }, "WebSocket relay lock release failed"); + return false; + } + }, + }; +} + +/** + * Try once to acquire a relay lock. + * + * @param {string} key From buildRelayLockKey() + * @param {object} [options] + * @param {number} [options.ttlMs] + * @param {() => Promise} [options.getRedis] Injected for tests + * @returns {Promise} + * `null` when another holder currently owns the lock. + */ +export async function tryAcquireRelayLock(key, { ttlMs, getRedis = connectRedisClient } = {}) { + const ttl = resolveRelayLockTtlMs(ttlMs); + const token = randomUUID(); + const scope = scopeFromKey(key); + const client = await resolveRedis(getRedis); + + if (client) { + try { + const reply = await client.sendCommand(["SET", key, token, "NX", "PX", String(ttl)]); + if (reply !== "OK") { + relayLockAcquisitionsTotal.inc({ scope, result: "contended" }); + return null; + } + relayLockAcquisitionsTotal.inc({ scope, result: "acquired" }); + return buildLease({ key, token, ttl, backend: "redis", client }); + } catch (err) { + relayLockAcquisitionsTotal.inc({ scope, result: "error" }); + logger.warn( + { err: err?.message, scope }, + "WebSocket relay lock: Redis SET NX failed, falling back to local lock", + ); + } + } + + if (!acquireLocal(key, token, ttl)) { + relayLockAcquisitionsTotal.inc({ scope, result: "contended" }); + return null; + } + relayLockAcquisitionsTotal.inc({ scope, result: "acquired" }); + return buildLease({ key, token, ttl, backend: "local" }); +} + +/** + * Acquire a relay lock, waiting up to `waitMs` with jittered polling. + * + * @returns {Promise} The lease, or null when still busy after waitMs. + */ +export async function acquireRelayLock( + key, + { + waitMs, + retryIntervalMs = DEFAULT_RETRY_INTERVAL_MS, + sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)), + now = () => Date.now(), + ...options + } = {}, +) { + const budget = resolveWaitMs(waitMs); + const interval = Math.max(10, Math.trunc(Number(retryIntervalMs) || DEFAULT_RETRY_INTERVAL_MS)); + const startedAt = now(); + const scope = scopeFromKey(key); + + for (;;) { + const lease = await tryAcquireRelayLock(key, options); + const elapsed = now() - startedAt; + if (lease || elapsed >= budget) { + relayLockWaitDuration.observe({ scope }, elapsed / 1000); + return lease; + } + // ±25% jitter so waiting instances do not retry in lockstep. + const jittered = Math.round(interval * (0.75 + Math.random() * 0.5)); + await sleep(Math.min(jittered, Math.max(0, budget - elapsed))); + } +} + +export function createRelayLockBusyError(scope) { + const error = new Error(`WebSocket relay ${scope} is already being processed by another worker.`); + error.status = 409; + error.code = RELAY_LOCK_BUSY; + error.retryable = true; + return error; +} + +function createRelayLockLostError(scope) { + const error = new Error(`WebSocket relay ${scope} lock was lost before the work completed.`); + error.code = RELAY_LOCK_LOST; + error.retryable = true; + return error; +} + +/** + * Run `fn` while holding the relay lock for (scope, resource). + * + * The lease is renewed every `renewIntervalMs` (default ttl/3). If a renewal + * fails, the AbortSignal passed to `fn` is aborted with a RELAY_LOCK_LOST + * error; long-running work (e.g. a DLQ drain) must check `signal.aborted` + * between units of work. + * + * @template T + * @param {string} scope One of RelayLockScope + * @param {string} resource + * @param {(ctx: { lease: object, signal: AbortSignal }) => Promise} fn + * @param {object} [options] + * @param {number} [options.ttlMs] + * @param {number} [options.waitMs=0] How long to wait for a busy lock + * @param {number} [options.renewIntervalMs] + * @returns {Promise} + * @throws 409 error (code RELAY_LOCK_BUSY) when the lock stays held + */ +export async function withRelayLock(scope, resource, fn, options = {}) { + if (typeof fn !== "function") { + throw new TypeError("withRelayLock requires a function to run"); + } + const key = buildRelayLockKey(scope, resource); + const lease = await acquireRelayLock(key, options); + if (!lease) { + logger.warn({ scope }, "WebSocket relay work rejected: lock held by another worker"); + throw createRelayLockBusyError(scope); + } + + const controller = new AbortController(); + const renewEvery = Math.max( + 250, + Math.trunc(Number(options.renewIntervalMs) || lease.ttlMs / 3), + ); + let renewing = false; + const timer = setInterval(async () => { + if (renewing || controller.signal.aborted) return; + renewing = true; + try { + const ok = await lease.renew(); + if (!ok && !controller.signal.aborted) { + relayLockLostTotal.inc({ scope }); + logger.error({ scope, backend: lease.backend }, "WebSocket relay lock lost during work"); + controller.abort(createRelayLockLostError(scope)); + } + } finally { + renewing = false; + } + }, renewEvery); + timer.unref?.(); + + try { + return await fn({ lease, signal: controller.signal }); + } finally { + clearInterval(timer); + const heldMs = Date.now() - lease.acquiredAt; + relayLockHoldDuration.observe({ scope }, heldMs / 1000); + await lease.release(); + logger.debug?.({ scope, backend: lease.backend, heldMs }, "WebSocket relay lock released"); + } +} + +/** + * In-process single-flight: concurrent calls with the same key share one + * in-flight promise. Used so that many sockets noticing the same upstream + * drop trigger exactly one reconnect loop per process. + */ +export function createSingleFlight() { + const inFlight = new Map(); + return { + run(key, fn) { + if (inFlight.has(key)) return inFlight.get(key); + const promise = Promise.resolve() + .then(fn) + .finally(() => inFlight.delete(key)); + inFlight.set(key, promise); + return promise; + }, + has(key) { + return inFlight.has(key); + }, + get size() { + return inFlight.size; + }, + }; +} + +/** Test helper: clear the in-process fallback lock table. */ +export function resetLocalRelayLocksForTests() { + localLocks.clear(); +} diff --git a/backend/src/lib/websocket-relay-lock.test.js b/backend/src/lib/websocket-relay-lock.test.js new file mode 100644 index 00000000..afe3ecec --- /dev/null +++ b/backend/src/lib/websocket-relay-lock.test.js @@ -0,0 +1,380 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; + +vi.mock("./logger.js", () => ({ + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }, +})); + +vi.mock("./redis.js", () => ({ + connectRedisClient: vi.fn(async () => ({ isOpen: false })), +})); + +import { logger } from "./logger.js"; +import { createFakeRedis } from "../../tests/helpers/fake-redis.js"; +import { + DEFAULT_RELAY_LOCK_TTL_MS, + RELAY_LOCK_BUSY, + RELAY_LOCK_LOST, + RELEASE_RELAY_LOCK_SCRIPT, + RENEW_RELAY_LOCK_SCRIPT, + RelayLockScope, + acquireRelayLock, + buildRelayLockKey, + createSingleFlight, + resetLocalRelayLocksForTests, + resolveRelayLockTtlMs, + tryAcquireRelayLock, + withRelayLock, +} from "./websocket-relay-lock.js"; +import { + relayLockAcquisitionsTotal, + relayLockLostTotal, +} from "./websocket-relay-metrics.js"; + +async function counterValue(counter, labels) { + const { values } = await counter.get(); + const match = values.find((v) => + Object.entries(labels).every(([k, val]) => v.labels[k] === val), + ); + return match ? match.value : 0; +} + +beforeEach(() => { + vi.clearAllMocks(); + resetLocalRelayLocksForTests(); + relayLockAcquisitionsTotal.reset(); + relayLockLostTotal.reset(); +}); + +afterEach(() => { + vi.useRealTimers(); +}); + +describe("buildRelayLockKey (issue #1455)", () => { + it("namespaces by scope and hashes the resource", () => { + const key = buildRelayLockKey(RelayLockScope.EVENT, "evt_123"); + expect(key).toMatch(/^lock:ws-relay:event:[0-9a-f]{64}$/); + expect(key).not.toContain("evt_123"); + }); + + it("is deterministic per resource and distinct across scopes", () => { + expect(buildRelayLockKey("event", "a")).toBe(buildRelayLockKey("event", "a")); + expect(buildRelayLockKey("event", "a")).not.toBe(buildRelayLockKey("dlq-drain", "a")); + }); + + it("neutralises key-injection characters in the resource", () => { + const key = buildRelayLockKey("connection", "conn:1\r\nDEL *"); + expect(key.split(":")).toHaveLength(4); + expect(key).not.toMatch(/[\r\n* ]/); + }); + + it("rejects unknown scopes, empty and oversized resources", () => { + expect(() => buildRelayLockKey("admin", "x")).toThrow(TypeError); + expect(() => buildRelayLockKey("event", "")).toThrow(TypeError); + expect(() => buildRelayLockKey("event", " ")).toThrow(TypeError); + expect(() => buildRelayLockKey("event", 42)).toThrow(TypeError); + expect(() => buildRelayLockKey("event", "x".repeat(513))).toThrow(RangeError); + }); +}); + +describe("resolveRelayLockTtlMs", () => { + it("defaults, clamps and reads the env override", () => { + expect(resolveRelayLockTtlMs(undefined, {})).toBe(DEFAULT_RELAY_LOCK_TTL_MS); + expect(resolveRelayLockTtlMs("abc", {})).toBe(DEFAULT_RELAY_LOCK_TTL_MS); + expect(resolveRelayLockTtlMs(10, {})).toBe(1_000); + expect(resolveRelayLockTtlMs(10_000_000, {})).toBe(120_000); + expect(resolveRelayLockTtlMs(undefined, { WS_RELAY_LOCK_TTL_MS: "5000" })).toBe(5_000); + }); +}); + +describe("tryAcquireRelayLock with Redis", () => { + it("acquires with SET NX PX and blocks a second holder across instances", async () => { + const redis = createFakeRedis(); + const getRedis = async () => redis; + const key = buildRelayLockKey("event", "evt_1"); + + const first = await tryAcquireRelayLock(key, { getRedis, ttlMs: 5_000 }); + const second = await tryAcquireRelayLock(key, { getRedis, ttlMs: 5_000 }); + + expect(first).toMatchObject({ key, backend: "redis", ttlMs: 5_000 }); + expect(second).toBeNull(); + expect(redis.calls[0]).toEqual(["SET", key, first.token, "NX", "PX", "5000"]); + expect(await counterValue(relayLockAcquisitionsTotal, { scope: "event", result: "acquired" })).toBe(1); + expect(await counterValue(relayLockAcquisitionsTotal, { scope: "event", result: "contended" })).toBe(1); + }); + + it("only lets the owner release, via compare-and-delete", async () => { + const redis = createFakeRedis(); + const getRedis = async () => redis; + const key = buildRelayLockKey("event", "evt_2"); + + const lease = await tryAcquireRelayLock(key, { getRedis }); + redis.store.set(key, { value: "someone-else", expiresAt: null }); + + expect(await lease.release()).toBe(false); + expect(redis.keys(key)).toEqual([key]); + expect(redis.calls.at(-1)).toEqual(["EVAL", RELEASE_RELAY_LOCK_SCRIPT, "1", key, lease.token]); + expect(logger.warn).toHaveBeenCalled(); + }); + + it("renews only while the lease is still owned", async () => { + vi.useFakeTimers(); + const redis = createFakeRedis(); + const getRedis = async () => redis; + const key = buildRelayLockKey("dlq-drain", "global"); + + const lease = await tryAcquireRelayLock(key, { getRedis, ttlMs: 2_000 }); + vi.advanceTimersByTime(1_500); + expect(await lease.renew()).toBe(true); + expect(redis.calls.at(-1)).toEqual(["EVAL", RENEW_RELAY_LOCK_SCRIPT, "1", key, lease.token, "2000"]); + + vi.advanceTimersByTime(1_500); + expect(redis.keys(key)).toEqual([key]); // still alive thanks to the renewal + + vi.advanceTimersByTime(1_000); + expect(await lease.renew()).toBe(false); // expired: renewal must not resurrect it + expect(await tryAcquireRelayLock(key, { getRedis })).not.toBeNull(); + }); + + it("falls back to the local lock when SET fails", async () => { + const redis = createFakeRedis(); + redis.setFailure(new Error("READONLY")); + const key = buildRelayLockKey("event", "evt_3"); + + const lease = await tryAcquireRelayLock(key, { getRedis: async () => redis }); + + expect(lease.backend).toBe("local"); + expect(await counterValue(relayLockAcquisitionsTotal, { scope: "event", result: "error" })).toBe(1); + }); + + it("treats a renewal error as a lost lease", async () => { + const redis = createFakeRedis(); + const key = buildRelayLockKey("event", "evt_4"); + const lease = await tryAcquireRelayLock(key, { getRedis: async () => redis }); + + redis.setFailure(new Error("connection reset")); + expect(await lease.renew()).toBe(false); + expect(await lease.release()).toBe(false); + expect(logger.error).toHaveBeenCalledTimes(2); + }); +}); + +describe("tryAcquireRelayLock local fallback", () => { + it("uses the local lock when Redis is a no-op client", async () => { + const key = buildRelayLockKey("connection", "c1"); + const lease = await tryAcquireRelayLock(key); + expect(lease.backend).toBe("local"); + expect(await tryAcquireRelayLock(key)).toBeNull(); + expect(await lease.release()).toBe(true); + expect(await tryAcquireRelayLock(key)).not.toBeNull(); + }); + + it("uses the local lock when connecting to Redis throws", async () => { + const key = buildRelayLockKey("connection", "c2"); + const lease = await tryAcquireRelayLock(key, { + getRedis: async () => { + throw new Error("ECONNREFUSED"); + }, + }); + expect(lease.backend).toBe("local"); + }); + + it("does not release or renew a local lease it no longer owns", async () => { + vi.useFakeTimers(); + const key = buildRelayLockKey("connection", "c3"); + const stale = await tryAcquireRelayLock(key, { ttlMs: 1_000 }); + vi.advanceTimersByTime(1_001); + const fresh = await tryAcquireRelayLock(key, { ttlMs: 1_000 }); + + expect(await stale.renew()).toBe(false); + expect(await stale.release()).toBe(false); + expect(await fresh.renew()).toBe(true); + }); +}); + +describe("acquireRelayLock waiting", () => { + it("returns null immediately when waitMs is 0 and the lock is held", async () => { + const key = buildRelayLockKey("event", "w1"); + await tryAcquireRelayLock(key); + const sleep = vi.fn(); + expect(await acquireRelayLock(key, { waitMs: 0, sleep })).toBeNull(); + expect(sleep).not.toHaveBeenCalled(); + }); + + it("polls until the holder releases within the wait budget", async () => { + const key = buildRelayLockKey("event", "w2"); + const holder = await tryAcquireRelayLock(key); + let clock = 0; + const sleep = vi.fn(async (ms) => { + clock += ms; + if (clock >= 200) await holder.release(); + }); + + const lease = await acquireRelayLock(key, { + waitMs: 1_000, + retryIntervalMs: 100, + sleep, + now: () => clock, + }); + + expect(lease).not.toBeNull(); + expect(sleep.mock.calls.length).toBeGreaterThanOrEqual(2); + for (const [ms] of sleep.mock.calls) { + expect(ms).toBeGreaterThanOrEqual(75); + expect(ms).toBeLessThanOrEqual(125); + } + }); + + it("gives up once the wait budget is spent", async () => { + const key = buildRelayLockKey("event", "w3"); + await tryAcquireRelayLock(key); + let clock = 0; + const lease = await acquireRelayLock(key, { + waitMs: 300, + retryIntervalMs: 100, + sleep: async (ms) => { + clock += ms; + }, + now: () => clock, + }); + expect(lease).toBeNull(); + expect(clock).toBeLessThanOrEqual(300); + }); +}); + +describe("withRelayLock", () => { + it("runs the work once and releases the lock afterwards", async () => { + const redis = createFakeRedis(); + const getRedis = async () => redis; + const work = vi.fn(async ({ lease, signal }) => { + expect(signal.aborted).toBe(false); + expect(redis.keys(lease.key)).toEqual([lease.key]); + return "delivered"; + }); + + await expect(withRelayLock("event", "evt_9", work, { getRedis })).resolves.toBe("delivered"); + expect(work).toHaveBeenCalledTimes(1); + expect(redis.keys("lock:ws-relay:")).toEqual([]); + }); + + it("delivers a contested event exactly once across concurrent workers", async () => { + const redis = createFakeRedis({ latencyMs: 1 }); + const getRedis = async () => redis; + const deliver = vi.fn(async () => new Promise((resolve) => setTimeout(resolve, 20))); + + const results = await Promise.allSettled( + Array.from({ length: 5 }, () => withRelayLock("event", "evt_race", deliver, { getRedis })), + ); + + expect(deliver).toHaveBeenCalledTimes(1); + const rejected = results.filter((r) => r.status === "rejected"); + expect(rejected).toHaveLength(4); + for (const r of rejected) { + expect(r.reason).toMatchObject({ status: 409, code: RELAY_LOCK_BUSY, retryable: true }); + } + }); + + it("releases the lock when the work throws", async () => { + const redis = createFakeRedis(); + const getRedis = async () => redis; + await expect( + withRelayLock("dlq-drain", "global", async () => { + throw new Error("upstream down"); + }, { getRedis }), + ).rejects.toThrow("upstream down"); + expect(redis.keys("lock:ws-relay:")).toEqual([]); + }); + + it("aborts the work with RELAY_LOCK_LOST when renewal fails", async () => { + vi.useFakeTimers(); + const redis = createFakeRedis(); + const getRedis = async () => redis; + let observedReason; + + const run = withRelayLock( + "dlq-drain", + "global", + ({ lease, signal }) => + new Promise((resolve) => { + signal.addEventListener("abort", () => { + observedReason = signal.reason; + resolve("stopped"); + }); + // Another worker steals the key (e.g. after a GC pause). + redis.store.set(lease.key, { value: "thief", expiresAt: null }); + }), + { getRedis, ttlMs: 3_000 }, + ); + + await vi.advanceTimersByTimeAsync(2_500); + await expect(run).resolves.toBe("stopped"); + expect(observedReason).toMatchObject({ code: RELAY_LOCK_LOST }); + expect(await counterValue(relayLockLostTotal, { scope: "dlq-drain" })).toBe(1); + // The thief's lease survives: we never delete a key we do not own. + expect(redis.store.get(buildRelayLockKey("dlq-drain", "global")).value).toBe("thief"); + }); + + it("keeps a long critical section alive by renewing the lease", async () => { + vi.useFakeTimers(); + const redis = createFakeRedis(); + const getRedis = async () => redis; + const key = buildRelayLockKey("dlq-drain", "long"); + + const run = withRelayLock( + "dlq-drain", + "long", + ({ signal }) => + new Promise((resolve) => { + setTimeout(() => resolve(signal.aborted), 5_000); + }), + { getRedis, ttlMs: 1_500 }, + ); + + await vi.advanceTimersByTimeAsync(4_000); + expect(await tryAcquireRelayLock(key, { getRedis })).toBeNull(); + await vi.advanceTimersByTimeAsync(2_000); + await expect(run).resolves.toBe(false); + }); + + it("rejects a non-function before touching the lock", async () => { + await expect(withRelayLock("event", "x", null)).rejects.toThrow(TypeError); + }); +}); + +describe("createSingleFlight", () => { + it("coalesces concurrent calls for the same key", async () => { + const flight = createSingleFlight(); + let resolveConnect; + const connect = vi.fn(() => new Promise((resolve) => { + resolveConnect = resolve; + })); + + const a = flight.run("upstream", connect); + const b = flight.run("upstream", connect); + await Promise.resolve(); + expect(flight.has("upstream")).toBe(true); + resolveConnect("socket"); + + await expect(Promise.all([a, b])).resolves.toEqual(["socket", "socket"]); + expect(connect).toHaveBeenCalledTimes(1); + expect(flight.size).toBe(0); + }); + + it("shares a failure and allows a fresh attempt afterwards", async () => { + const flight = createSingleFlight(); + const failing = vi.fn().mockRejectedValue(new Error("refused")); + await expect(Promise.all([flight.run("k", failing), flight.run("k", failing)])).rejects.toThrow( + "refused", + ); + expect(failing).toHaveBeenCalledTimes(1); + + const ok = vi.fn().mockResolvedValue("ok"); + await expect(flight.run("k", ok)).resolves.toBe("ok"); + }); + + it("keeps different keys independent", async () => { + const flight = createSingleFlight(); + const fn = vi.fn().mockResolvedValue(1); + await Promise.all([flight.run("a", fn), flight.run("b", fn)]); + expect(fn).toHaveBeenCalledTimes(2); + }); +}); diff --git a/backend/src/lib/websocket-relay-metrics.js b/backend/src/lib/websocket-relay-metrics.js index 90677ecc..ca0274c1 100644 --- a/backend/src/lib/websocket-relay-metrics.js +++ b/backend/src/lib/websocket-relay-metrics.js @@ -179,6 +179,39 @@ export const relayLastErrorTimestamp = new client.Gauge({ help: "Unix timestamp of the last relay error", }); +// ─── Distributed Lock Metrics (Issue #1455) ────────────────────────────── + +export const relayLockAcquisitionsTotal = new client.Counter({ + name: "relay_lock_acquisitions_total", + help: "Relay distributed lock acquisition attempts", + labelNames: ["scope", "result"], // result: acquired | contended | error +}); + +export const relayLockWaitDuration = new client.Histogram({ + name: "relay_lock_wait_duration_seconds", + help: "Time spent waiting to acquire a relay distributed lock", + labelNames: ["scope"], + buckets: [0.001, 0.01, 0.05, 0.1, 0.5, 1, 5, 30], +}); + +export const relayLockHoldDuration = new client.Histogram({ + name: "relay_lock_hold_duration_seconds", + help: "Time a relay distributed lock was held", + labelNames: ["scope"], + buckets: [0.01, 0.05, 0.1, 0.5, 1, 5, 15, 60], +}); + +export const relayLockLostTotal = new client.Counter({ + name: "relay_lock_lost_total", + help: "Relay distributed locks lost before the critical section completed", + labelNames: ["scope"], +}); + +export const relayCircuitBreakerProbeRejectedTotal = new client.Counter({ + name: "relay_circuit_breaker_probe_rejected_total", + help: "Calls rejected while a HALF_OPEN probe was already in flight", +}); + // Register all metrics register.registerMetric(relayConnectionsTotal); register.registerMetric(relayConnectionsActive); @@ -205,5 +238,10 @@ register.registerMetric(relayLastHeartbeatSeconds); register.registerMetric(relayUptime); register.registerMetric(relayErrorsTotal); register.registerMetric(relayLastErrorTimestamp); +register.registerMetric(relayLockAcquisitionsTotal); +register.registerMetric(relayLockWaitDuration); +register.registerMetric(relayLockHoldDuration); +register.registerMetric(relayLockLostTotal); +register.registerMetric(relayCircuitBreakerProbeRejectedTotal); export { register as relayMetricsRegister }; diff --git a/backend/src/lib/websocket-relay-recovery.js b/backend/src/lib/websocket-relay-recovery.js index 7a7248dd..57f55dc1 100644 --- a/backend/src/lib/websocket-relay-recovery.js +++ b/backend/src/lib/websocket-relay-recovery.js @@ -6,8 +6,14 @@ * - Circuit breaker (CLOSED → OPEN → HALF_OPEN) * - Dead letter queue (DLQ) for undeliverable messages * - Health snapshot + * + * Concurrency (issue #1455): the circuit breaker admits exactly one probe in + * HALF_OPEN. Cross-instance coordination (event delivery, DLQ drains, + * reconnects) lives in websocket-relay-lock.js. */ +import { relayCircuitBreakerProbeRejectedTotal } from "./websocket-relay-metrics.js"; + // ─── Circuit breaker states ─────────────────────────────────────────────────── const CircuitState = Object.freeze({ @@ -94,6 +100,7 @@ class CircuitBreaker { this._failureCount = 0; this._openedAt = null; this._lastError = null; + this._probeInFlight = false; } get state() { @@ -119,6 +126,19 @@ class CircuitBreaker { } } + // HALF_OPEN admits a single probe. Concurrent callers that arrive while it + // is in flight are rejected instead of all hammering a recovering upstream. + const isProbe = this._state === CircuitState.HALF_OPEN; + if (isProbe) { + if (this._probeInFlight) { + relayCircuitBreakerProbeRejectedTotal.inc(); + const err = new Error("Circuit breaker is HALF_OPEN — probe already in flight"); + err.circuitOpen = true; + throw err; + } + this._probeInFlight = true; + } + try { const result = await fn(); this._onSuccess(); @@ -126,6 +146,8 @@ class CircuitBreaker { } catch (err) { this._onFailure(err); throw err; + } finally { + if (isProbe) this._probeInFlight = false; } } @@ -154,6 +176,7 @@ class CircuitBreaker { state: this._state, failureCount: this._failureCount, lastError: this._lastError ? this._lastError.message : null, + probeInFlight: this._probeInFlight, }; } } diff --git a/backend/src/lib/websocket-relay-recovery.test.js b/backend/src/lib/websocket-relay-recovery.test.js index 760a921c..bf08ce81 100644 --- a/backend/src/lib/websocket-relay-recovery.test.js +++ b/backend/src/lib/websocket-relay-recovery.test.js @@ -276,3 +276,60 @@ describe("getRelayHealth", () => { expect(health.lastError).toBe("down"); }); }); + +// ─── HALF_OPEN single probe (issue #1455) ───────────────────────────────────── + +describe("CircuitBreaker HALF_OPEN concurrency", () => { + async function openBreaker() { + let fakeNow = 0; + const breaker = new CircuitBreaker({ + failureThreshold: 1, + resetTimeoutMs: 1000, + now: () => fakeNow, + }); + await breaker.call(() => Promise.reject(new Error("down"))).catch(() => {}); + fakeNow += 1000; + return breaker; + } + + it("admits exactly one probe while HALF_OPEN", async () => { + const breaker = await openBreaker(); + let finishProbe; + const probe = vi.fn(() => new Promise((resolve) => { + finishProbe = resolve; + })); + + const first = breaker.call(probe); + expect(breaker.state).toBe(CircuitState.HALF_OPEN); + expect(breaker.getStatus().probeInFlight).toBe(true); + + const concurrent = vi.fn().mockResolvedValue("should not run"); + await expect(breaker.call(concurrent)).rejects.toMatchObject({ circuitOpen: true }); + expect(concurrent).not.toHaveBeenCalled(); + + finishProbe("ok"); + await expect(first).resolves.toBe("ok"); + expect(breaker.state).toBe(CircuitState.CLOSED); + expect(breaker.getStatus().probeInFlight).toBe(false); + }); + + it("re-opens and frees the probe slot when the probe fails", async () => { + const breaker = await openBreaker(); + await expect(breaker.call(() => Promise.reject(new Error("still down")))).rejects.toThrow( + "still down", + ); + expect(breaker.state).toBe(CircuitState.OPEN); + expect(breaker.getStatus().probeInFlight).toBe(false); + }); + + it("does not limit concurrency while CLOSED", async () => { + const breaker = new CircuitBreaker(); + const fn = vi.fn().mockResolvedValue("ok"); + await expect(Promise.all([breaker.call(fn), breaker.call(fn), breaker.call(fn)])).resolves.toEqual([ + "ok", + "ok", + "ok", + ]); + expect(fn).toHaveBeenCalledTimes(3); + }); +}); diff --git a/backend/tests/helpers/fake-redis.js b/backend/tests/helpers/fake-redis.js index 6c682755..7e92fd0f 100644 --- a/backend/tests/helpers/fake-redis.js +++ b/backend/tests/helpers/fake-redis.js @@ -2,8 +2,9 @@ * In-memory Redis stand-in shared by the exchange-rate coordinator and the * payment-session lock / idempotency tests. * - * Supports GET, SET (EX/PX/NX), DEL and the compare-and-delete EVAL script, - * both as methods and through sendCommand. Commands run in FIFO order. + * Supports GET, SET (EX/PX/NX), DEL and the compare-and-delete and + * compare-and-pexpire EVAL scripts, both as methods and through sendCommand. + * Commands run in FIFO order. * Expiry follows Date.now(), so it works with vi.useFakeTimers(). * setFailure() makes later commands reject, which the coordinator uses to * exercise its fail-open path. @@ -45,6 +46,12 @@ export function createFakeRedis({ latencyMs = 0 } = {}) { && /redis\.call\(['"]del['"], KEYS\[1\]\)/.test(text); } + function isCompareAndPexpire(script) { + const text = String(script); + return /redis\.call\(['"]get['"], KEYS\[1\]\)/.test(text) + && /redis\.call\(['"]pexpire['"], KEYS\[1\], ARGV\[2\]\)/.test(text); + } + async function run(args, fn) { calls.push(args); await tick(); @@ -102,7 +109,15 @@ export function createFakeRedis({ latencyMs = 0 } = {}) { case 'DEL': return rest.reduce((count, key) => count + (live(key) && store.delete(key) ? 1 : 0), 0); case 'EVAL': { - const [script, numKeys, key, token] = rest; + const [script, numKeys, key, token, ttlMs] = rest; + if (isCompareAndPexpire(script) && String(numKeys) === '1') { + const entry = live(key); + if (entry?.value === token) { + entry.expiresAt = Date.now() + Number(ttlMs); + return 1; + } + return 0; + } if (!isCompareAndDelete(script) || String(numKeys) !== '1') { throw new Error('fake-redis: unsupported EVAL script'); } diff --git a/frontend/I18N.md b/frontend/I18N.md new file mode 100644 index 00000000..7cf7b6c5 --- /dev/null +++ b/frontend/I18N.md @@ -0,0 +1,38 @@ +# Dashboard localization (i18n) + +Issue #428. The merchant dashboard is localized with `next-intl`. It uses +cookie-based locale selection, so URLs carry no locale prefix. + +## Supported locales + +| Code | Language | HTML `lang` | +|------|----------|-------------| +| `en` | English (default, source of truth) | `en-US` | +| `es` | Spanish | `es-ES` | +| `pt` | Portuguese | `pt-BR` | +| `fr` | French | `fr-FR` | + +The list lives in `src/i18n/config.ts` (`locales`). Routing, the request config +and the header switcher all read it. + +## How the locale is chosen + +1. `LocaleSwitcher` in the authenticated dashboard header writes the `NEXT_LOCALE` cookie (1 year, `SameSite=Lax`) and calls `router.replace` on the current route, keeping the query string. +2. `src/i18n/request.ts` reads the cookie on the server, validates it with `resolveAppLocale()` (unknown values fall back to `en`), and loads `messages/.json`. +3. `ThemeLanguageSync` updates `` using `localeToLanguageTag()`. + +## Adding or changing strings + +- Add the key to `messages/en.json` first, then to `es.json`, `pt.json` and `fr.json` under the same path. +- Use `useTranslations("")` in components. Do not hard-code user-facing English in dashboard components; that includes `aria-label`s. +- Keep ICU placeholders identical across locales (`{count}`, `{asset}`, plural blocks). + +`src/i18n/messages.test.ts` enforces this. It fails if any locale is missing a +key, has an extra key, drops an ICU argument or contains an empty string. + +## Adding a locale + +1. Add the code to `locales` in `src/i18n/config.ts` and a case to `localeToLanguageTag`. +2. Create `messages/.json` with every key from `en.json`. +3. Add `localeSwitcher.options.` to every catalogue. +4. Run `npm run test:unit -- src/i18n src/components/LocaleSwitcher.test.tsx`. diff --git a/frontend/messages/en.json b/frontend/messages/en.json index cc0496b4..c2f5b5f6 100644 --- a/frontend/messages/en.json +++ b/frontend/messages/en.json @@ -33,7 +33,8 @@ "options": { "en": "English", "es": "Espanol", - "pt": "Portugues" + "pt": "Portugues", + "fr": "Francais" } }, "nav": { @@ -809,5 +810,8 @@ "fromSyncLabel": "Updating start date…", "toSyncLabel": "Updating end date…" } + }, + "dashboardShell": { + "openNavigationMenu": "Open navigation menu" } } diff --git a/frontend/messages/es.json b/frontend/messages/es.json index 1e424a06..d19b8b2b 100644 --- a/frontend/messages/es.json +++ b/frontend/messages/es.json @@ -33,7 +33,8 @@ "options": { "en": "English", "es": "Espanol", - "pt": "Portugues" + "pt": "Portugues", + "fr": "Frances" } }, "nav": { @@ -205,6 +206,11 @@ "title": "Historial de pagos", "description": "Supervisa todas las transacciones entrantes y su estado de verificacion." }, + "portfolioChart": { + "valueTitle": "Valor del portafolio", + "allocation": "Distribución", + "trend": "Tendencia" + }, "recentPayments": { "missingApiKey": "No se encontro la clave API. Registrate o inicia sesion.", "fetchFailed": "No se pudieron obtener los pagos", @@ -577,6 +583,23 @@ "successDescription": "Tu verificacion KYC ha sido enviada y esta en revision.", "dash": "—" }, + "network": { + "status": "Estado de la red", + "refresh": "Comprobar el estado de la red", + "latency": "Latencia", + "connection": "Conexión", + "error": "Error", + "lastChecked": "Última comprobación", + "online": "En línea", + "offline": "Sin conexión", + "slow": "Lenta", + "checking": "Comprobando...", + "connectionQuality": "Calidad de la conexión", + "excellent": "Excelente", + "good": "Buena", + "fair": "Regular", + "poor": "Deficiente" + }, "darkModeTheme": { "loading": "Cargando configuracion del tema", "errorCleared": "Error eliminado. Intentando cambiar el tema.", @@ -787,5 +810,8 @@ "fromSyncLabel": "Actualizando fecha de inicio…", "toSyncLabel": "Actualizando fecha de fin…" } + }, + "dashboardShell": { + "openNavigationMenu": "Abrir menú de navegación" } } diff --git a/frontend/messages/fr.json b/frontend/messages/fr.json new file mode 100644 index 00000000..2a7791b2 --- /dev/null +++ b/frontend/messages/fr.json @@ -0,0 +1,817 @@ +{ + "twoFactorAuth": { + "ariaLabel": "Configuration de l'authentification à deux facteurs", + "stepsLabel": "Étapes de configuration", + "title": "Activer l'authentification à deux facteurs", + "description": "Protégez votre compte avec une application d'authentification. Vous devrez scanner un code QR pour commencer.", + "enableButton": "Activer la 2FA", + "settingUp": "Configuration…", + "scanTitle": "Scannez avec votre application d'authentification", + "scanDescription": "Utilisez Google Authenticator, Authy ou toute application compatible TOTP pour scanner le code ci-dessous.", + "qrCodeAlt": "Code QR TOTP — scannez-le avec votre application d'authentification", + "qrCodeSkeletonLabel": "Génération du code QR…", + "manualKeyLabel": "Ou saisissez la clé manuellement", + "manualKeyAria": "Clé de configuration manuelle : {key}", + "codeInputLabel": "Saisissez le code à 6 chiffres", + "codeInputPlaceholder": "000000", + "verifyButton": "Vérifier et activer", + "verifying": "Vérification…", + "retryButton": "Réessayer", + "stepAnnouncement": "Étape {current} sur {total}", + "successTitle": "Authentification à deux facteurs activée", + "successDescription": "Votre compte est désormais protégé. Un code vous sera demandé à chaque connexion.", + "error": { + "setupFailed": "Impossible de démarrer la configuration de la 2FA. Veuillez réessayer.", + "invalidCode": "Code invalide. Veuillez réessayer.", + "codeLength": "Veuillez saisir le code à 6 chiffres de votre application d'authentification.", + "networkFailure": "Impossible de joindre le serveur. Vérifiez votre connexion et réessayez — votre code et la configuration QR sont conservés." + } + }, + "localeSwitcher": { + "label": "Langue", + "ariaLabel": "Sélectionner la langue", + "options": { + "en": "English", + "es": "Espanol", + "pt": "Portugues", + "fr": "Francais" + } + }, + "nav": { + "home": "Accueil", + "docs": "Documentation", + "login": "Connexion", + "register": "Inscription", + "toggleMenu": "Afficher/masquer le menu" + }, + "sidebar": { + "overview": "Vue d'ensemble", + "payments": "Paiements", + "webhookLogs": "Journaux des webhooks", + "apiKeys": "Clés API", + "settings": "Paramètres", + "workspace": "Espace de travail authentifié", + "shortcuts": "Raccourcis", + "createPayment": "Créer un paiement", + "network": "Mainnet-testnet", + "collapse": "Réduire la barre latérale", + "expand": "Développer la barre latérale", + "close": "Fermer la navigation" + }, + "breadcrumbs": { + "home": "Accueil", + "segments": { + "dashboard": "Tableau de bord", + "create": "Créer", + "payments": "Paiements", + "settings": "Paramètres", + "webhook-logs": "Journaux des webhooks", + "api-keys": "Clés API" + } + }, + "copyButton": { + "ariaLabel": "Copier dans le presse-papiers", + "copied": "Copié !", + "copiedState": "Copié" + }, + "dashboardPage": { + "title": "Vue d'ensemble marchand", + "description": "Suivez votre volume de transactions, vos paiements en temps réel et gérez votre intégration PLUTO.", + "paymentMetrics": "Indicateurs de paiement", + "recentActivity": "Activité récente", + "viewAllPayments": "Voir tous les paiements", + "quickActions": "Actions rapides", + "createPaymentLink": "Créer un lien de paiement", + "withdrawFunds": "Retirer des fonds (SEP-24)", + "development": "Développement", + "apiKeysTip": "Utilisez la page Clés API pour renouveler régulièrement votre jeton secret.", + "webhookLogsTip": "Consultez les journaux des webhooks si les paiements ne déclenchent pas votre backend." + }, + "realTimeBalanceSync": { + "title": "Soldes en temps réel", + "refreshButton": "Actualiser", + "syncing": "Synchronisation…", + "sectionAriaLabel": "Informations de solde en temps réel", + "balancesListAriaLabel": "Soldes du compte", + "emptyState": "Aucun solde disponible.", + "updatedLabel": "Mis à jour", + "balanceItemAriaLabel": "Solde {asset} : {balance}", + "liveRegion": { + "syncing": "Synchronisation des soldes…", + "error": "Erreur de synchronisation des soldes : {error}", + "updatedAt": "Soldes mis à jour à {time}." + } + }, + "createPaymentPage": { + "eyebrow": "Tableau de bord", + "title": "Créer un lien de paiement", + "description": "Définissez un montant, choisissez un actif et saisissez l'adresse du destinataire pour générer un lien de paiement PLUTO partageable.", + "newHere": "Nouveau ici ?", + "registerMerchant": "Créer un compte marchand" + }, + "createPaymentForm": { + "invalidAmount": "Le montant doit être un nombre positif.", + "invalidRecipient": "Le destinataire doit être une clé publique Stellar valide (56 caractères, commençant par G).", + "invalidHexColor": "{field} doit être une couleur hexadécimale valide.", + "failedCreate": "Impossible de créer le lien de paiement", + "createdToast": "Lien de paiement créé !", + "noApiKeyTitle": "Aucune clé API trouvée", + "noApiKeyDescription": "Créez un compte marchand pour obtenir votre clé API et commencer à créer des liens de paiement.", + "registerAsMerchant": "S'inscrire comme marchand", + "readyEyebrow": "Lien de paiement prêt", + "readyTitle": "Lien créé avec succès", + "readyDescription": "Partagez ce lien avec votre client pour encaisser le paiement.", + "shareLink": "Partager le lien", + "shareTitle": "Lien de paiement prêt", + "shareText": "Ouvrez ce lien de paiement PLUTO.", + "shareFailed": "Le partage natif n'est pas disponible pour le moment.", + "paymentLink": "Lien de paiement", + "paymentId": "ID du paiement", + "status": "Statut", + "createAnother": "Créer un autre lien de paiement", + "amount": "Montant", + "amountPlaceholder": "Saisissez {exampleAmount} {asset}", + "asset": "Actif", + "selectAsset": "Sélectionner un actif", + "issuer": "Émetteur", + "trustedAddresses": "Choisir parmi les adresses de confiance", + "selectSavedAddress": "-- Sélectionner une adresse enregistrée --", + "recipientAddress": "Adresse du destinataire", + "recipientPlaceholder": "Collez le portefeuille de destination {asset}", + "recipientPlaceholderSelected": "{label} sélectionné pour {asset}", + "descriptionLabel": "Description", + "optional": "facultatif", + "descriptionPlaceholder": "À quoi correspond cette demande en {asset} ?", + "brandingTitle": "Personnalisation de la session", + "brandingDescription": "Appliquer des couleurs personnalisées à ce lien de paiement uniquement.", + "enabled": "Activé", + "disabled": "Désactivé", + "primary": "Principale", + "secondary": "Secondaire", + "background": "Arrière-plan", + "checkoutPreview": "Aperçu du paiement", + "payNow": "Payer maintenant", + "generating": "Génération...", + "generate": "Générer le lien de paiement", + "rateLimitError": "Vous créez des liens trop rapidement. Réessayez dans {seconds} secondes.", + "retryWait": "Patientez {seconds} s…", + "integrationCode": "Code d'intégration", + "snippetsHelper": "Les extraits de code se mettent à jour automatiquement avec les valeurs du formulaire." + }, + "paymentMetrics": { + "downloadImage": "Télécharger l'image", + "downloadPng": "Télécharger en PNG", + "downloadSvg": "Télécharger en SVG", + "exporting": "Exportation...", + "exportSuccess": "Graphique téléchargé au format {format}", + "exportFailed": "Impossible d'exporter le graphique.", + "fetchMetricsFailed": "Impossible de récupérer les indicateurs", + "fetchVolumeFailed": "Impossible de récupérer les données de volume", + "retry": "Réessayer", + "sevenDayVolume": "Volume sur 7 jours", + "totalPayments": "Total des paiements", + "paymentsCount": "{count, plural, one {paiement} other {paiements}}", + "chartTitle": "Comparaison du volume multi-actifs", + "chartSubtitle": "Volume quotidien des transactions ventilé par actif", + "heatmapTitle": "Carte de densité des paiements", + "heatmapSubtitle": "Intensité d'activité façon GitHub sur les 365 derniers jours", + "heatmapTotal": "{count, plural, one {# paiement au cours de la dernière année} other {# paiements au cours de la dernière année}}", + "heatmapTooltip": "{count, plural, one {# paiement} other {# paiements}} le {date}", + "heatmapAriaLabel": "Carte de densité des paiements sur 365 jours", + "heatmapEmptyDay": "Aucun paiement enregistré", + "heatmapLegendLess": "Moins", + "heatmapLegendMore": "Plus", + "heatmapWeekdays": { + "sun": "D", + "mon": "L", + "tue": "M", + "wed": "M", + "thu": "J", + "fri": "V", + "sat": "S" + }, + "showRange": "Afficher {range}", + "toggleAssetVisibility": "Afficher/masquer l'actif", + "showAsset": "Afficher {asset}", + "hideAsset": "Masquer {asset}", + "noPayments": "Aucun paiement finalisé sur cette période.", + "weeklyAvgLabel": "Moy. 7 j", + "ranges": { + "7D": "7 jours", + "30D": "30 jours", + "1Y": "1 an" + } + }, + "paymentsPage": { + "title": "Historique des paiements", + "description": "Suivez toutes les transactions entrantes et leur statut de vérification." + }, + "portfolioChart": { + "valueTitle": "Valeur du portefeuille", + "allocation": "Répartition", + "trend": "Tendance" + }, + "recentPayments": { + "missingApiKey": "Clé API introuvable. Veuillez vous inscrire ou vous connecter.", + "fetchFailed": "Impossible de récupérer les paiements", + "loadFailed": "Impossible de charger les paiements", + "connectionError": "Erreur de connexion", + "backendHint": "Vérifiez que le backend est démarré et que le point de terminaison des paiements est disponible.", + "retryConnection": "Réessayer la connexion", + "testWebhook": "Tester le webhook quand même", + "troubleshootingTip": "Conseil de dépannage", + "troubleshootingDescription": "Vous pouvez toujours tester les webhooks pendant le rétablissement des services backend.", + "emptyTitle": "Aucun paiement pour l'instant", + "emptyDescription": "Commencez à accepter des paiements en créant votre premier lien de paiement ou en testant les webhooks pour voir circuler les données de transaction.", + "createPaymentLink": "Créer un lien de paiement", + "sendTestWebhook": "Envoyer un webhook de test", + "gettingStartedTitle": "Guide de démarrage", + "gettingStartedDescription": "Utilisez les outils de webhook pour tester les notifications de paiement et voir les données apparaître en temps réel dans ce tableau de bord.", + "search": "Rechercher", + "searchPlaceholder": "Rechercher par ID de paiement ou description...", + "status": "Statut", + "allStatuses": "Tous les statuts", + "asset": "Actif", + "allAssets": "Tous les actifs", + "fromDate": "Date de début", + "toDate": "Date de fin", + "activeFilters": "Filtres actifs :", + "searchChip": "Recherche : « {value} »", + "statusChip": "Statut : {value}", + "assetChip": "Actif : {value}", + "fromChip": "Du : {value}", + "toChip": "Au : {value}", + "clearSearchFilter": "Effacer le filtre de recherche", + "clearStatusFilter": "Effacer le filtre de statut", + "clearAssetFilter": "Effacer le filtre d'actif", + "clearFromDateFilter": "Effacer le filtre de date de début", + "clearToDateFilter": "Effacer le filtre de date de fin", + "clearAll": "Tout effacer", + "linkCopied": "Lien de paiement copié dans le presse-papiers !", + "showingResults": "Affichage de {shown} paiements sur {total}", + "filteredSuffix": "(filtres appliqués)", + "tableStatus": "Statut", + "tableAmount": "Montant", + "tableRecipient": "Destinataire", + "tableDate": "Date", + "tableLink": "Lien", + "view": "Voir", + "emptyDescriptionValue": "-", + "statuses": { + "all": "Tous", + "pending": "En attente", + "confirmed": "Confirmé", + "failed": "Échoué", + "refunded": "Remboursé" + }, + "pagination": { + "ariaLabel": "Pagination de l'historique des paiements", + "previous": "Page précédente", + "next": "Page suivante", + "goToPage": "Aller à la page {page}", + "currentPage": "Page actuelle, page {page}", + "pageLabel": "Page {page} sur {totalPages}", + "range": "Affichage de {start} à {end} sur {total}" + } + }, + "walletSelector": { + "chooseWallet": "Choisissez un portefeuille", + "description": "Connectez votre portefeuille Stellar pour finaliser ce paiement.", + "connecting": "Connexion...", + "walletConnectWaiting": "En attente du portefeuille...", + "noProjectId": "(aucun ID de projet)", + "notInstalled": "(non installé)", + "tapToConnect": "Cliquez pour vous connecter", + "scanTitle": "Scannez avec votre application de portefeuille compatible", + "scanDescription": "Scannez avec Freighter mobile ou tout portefeuille compatible WalletConnect", + "pairingFailed": "Échec de l'appairage WalletConnect", + "userRejected": "La demande de connexion a été annulée. Approuvez-la dans votre portefeuille pour continuer.", + "walletConnectUnavailable": "WalletConnect n'est pas disponible pour le moment. Essayez Freighter ou configurez un ID de projet WalletConnect valide.", + "extensionNotFound": "Ce portefeuille est introuvable sur cet appareil. Installez l'extension et réessayez.", + "noAccountFound": "Votre portefeuille est connecté, mais aucun compte Stellar n'a été renvoyé. Ouvrez un compte Stellar dans le portefeuille et réessayez.", + "walletConnectFailed": "Impossible de finaliser la connexion WalletConnect. Veuillez réessayer.", + "connectionFailed": "Impossible de se connecter à ce portefeuille pour le moment. Veuillez réessayer." + }, + "support": { + "panelAriaLabel": "Panneau d'aide et d'assistance", + "heading": "Aide et assistance", + "title": "Besoin d'aide pour vos paiements ?", + "description": "Besoin d'aide ? Envoyez-nous un message ou un pourboire pour bénéficier d'une assistance prioritaire.", + "contactEmail": "Contacter par e-mail", + "openChat": "Ouvrir le chat d'assistance", + "closeChat": "Fermer le chat d'assistance", + "yourWallet": "Votre portefeuille", + "notConnected": "Non connecté", + "connectingWallet": "Connexion…", + "amount": "Montant", + "loadingBalance": "Chargement du solde...", + "unfunded": "Non approvisionné (approvisionnez-le sur le Testnet)", + "available": "Disponible : {balance} {asset}", + "availableAriaLabel": "Solde disponible : {balance} {asset}", + "balanceSyncing": "Synchronisation du solde du portefeuille…", + "walletUnfunded": "Le portefeuille n'est pas approvisionné.", + "balanceSynced": "Solde synchronisé : {balance} {asset}.", + "insufficientBalance": "Solde insuffisant", + "messageLabel": "Laisser un message (facultatif)", + "messagePlaceholder": "Merci pour votre excellent travail !", + "sendSupport": "Envoyer le soutien", + "sending": "Envoi…", + "submissionFailed": "Échec de l'envoi", + "failedToRecord": "Impossible d'enregistrer la transaction de soutien" + }, + "checkout": { + "paymentRequest": "Demande de paiement", + "completePayment": "Finaliser le paiement", + "paymentNotFound": "Paiement introuvable", + "loadFailed": "Impossible de charger les détails du paiement.", + "paymentMissing": "Paiement introuvable.", + "loadPaymentFailed": "Impossible de charger le paiement.", + "errorTitle": "Erreur", + "errorDescription": "Vérifiez le lien de paiement et réessayez, ou contactez l'expéditeur.", + "processingFallback": "Traitement de la transaction...", + "doNotClose": "Ne fermez pas cet onglet", + "recipient": "Destinataire", + "scanToPay": "Scanner pour payer", + "scanDescription": "Scannez avec Freighter ou tout portefeuille compatible", + "openQrModal": "Payer avec un code QR", + "qrModalTitle": "Payer avec un code QR", + "qrModalDescription": "Scannez ce code sur un autre appareil ou partagez-le avec votre application de portefeuille.", + "downloadQr": "Télécharger en PNG", + "downloadQrSuccess": "Code QR téléchargé.", + "downloadQrError": "Impossible de télécharger le code QR.", + "viewRawIntent": "Afficher le lien d'intention brut", + "hideRawIntent": "Masquer le lien d'intention brut", + "created": "Créé", + "activeViewers": "{count, plural, one {# personne consulte ce lien de paiement} other {# personnes consultent ce lien de paiement}}", + "transaction": "Transaction", + "paymentConfirmed": "Paiement confirmé !", + "paymentSent": "Paiement envoyé !", + "paymentFailed": "Échec du paiement. Veuillez réessayer.", + "connectedVia": "Connecté via {provider}", + "payInXlm": "Payer en XLM", + "standardAssetOption": "Régler la facture en {asset}", + "pathPaymentOption": "Payer en XLM", + "approximateCost": "Coût approximatif en XLM", + "merchantReceives": "Le marchand reçoit {amount} {asset}", + "quoteLoading": "Vérification des chemins de paiement XLM...", + "quoteUnavailable": "Le routage des paiements en XLM n'est pas disponible pour cette facture pour le moment.", + "quoteBuffer": "Vous pouvez envoyer jusqu'à {amount} XLM pour couvrir les variations de prix.", + "pathPaymentHint": "Utilise les path payments Stellar via votre portefeuille connecté.", + "approximateCostLabel": "Coût approximatif en XLM", + "approximateCostHelp": "Montant estimé pour livrer {amount} {asset} au marchand.", + "slippageBuffer": "Marge de sécurité de {percent} % incluse. Envoi max. : {sendMax} {asset}", + "pathPaymentTogglePrefix": "Payer avec", + "pathPaymentToggleSuffix": "à la place", + "processing": "Traitement...", + "payWith": "Payer avec {provider}", + "payWithFallback": "Payer avec le portefeuille", + "reviewPaymentTitle": "Vérifier le paiement", + "loadingNetworkFee": "Récupération des frais de réseau actuels...", + "networkFeeLabel": "Frais de réseau : ~{amount} XLM", + "networkFeeUnavailable": "Frais de réseau indisponibles pour le moment.", + "cancel": "Annuler", + "confirmPayment": "Confirmer le paiement", + "receivedTitle": "Ce paiement a été reçu.", + "receivedDescription": "La transaction a été confirmée on-chain via PLUTO.", + "downloadReceipt": "Télécharger le reçu", + "downloadReceiptLoading": "Préparation du reçu...", + "receiptDownloaded": "Téléchargement du reçu lancé.", + "receiptDownloadFailed": "Impossible de générer le reçu PDF.", + "receiptHashUnavailable": "Indisponible", + "failedTitle": "Ce paiement a échoué.", + "failedDescription": "Contactez le marchand si vous pensez qu'il s'agit d'une erreur.", + "status": { + "pending": "En attente de paiement", + "confirmed": "Confirmé", + "completed": "Terminé", + "failed": "Échoué" + } + }, + "profileCard": { + "account": "Compte", + "openProfileMenu": "Ouvrir le menu du profil", + "dashboard": "Tableau de bord", + "settings": "Paramètres", + "createPayment": "Créer un paiement", + "logoutAccount": "Se déconnecter", + "loadingProfile": "Chargement du profil…" + }, + "settingsPage": { + "eyebrow": "Paramètres", + "eyebrowAccount": "Compte", + "title": "Paramètres", + "description": "Gérez vos identifiants, votre image de marque et vos intégrations.", + "loadingSettings": "Chargement des paramètres…", + "noApiKeyTitle": "Aucune clé API trouvée", + "noApiKeyDescription": "Créez d'abord un compte marchand pour gérer vos identifiants ici.", + "registerAsMerchant": "S'inscrire comme marchand", + "navAriaLabel": "Navigation des paramètres", + "navApiKeys": "Clés API", + "navBranding": "Image de marque", + "navDisplay": "Affichage", + "navWebhooks": "Webhooks", + "navPermissions": "Autorisations", + "navDanger": "Zone de danger", + "apiAuthTitle": "Authentification API", + "apiAuthDescription": "Votre clé secrète pour les requêtes API côté serveur.", + "liveApiKey": "Clé API de production", + "hide": "Masquer", + "reveal": "Afficher", + "apiKeyHeaderHintPrefix": "À transmettre dans l'en-tête", + "apiKeyHeaderHintSuffix": "à chaque requête.", + "apiKeyVisible": "La clé API de production est actuellement visible.", + "apiKeyHidden": "La clé API de production est actuellement masquée.", + "rotateApiKeyTitle": "Renouveler la clé API", + "rotateApiKeyDescription": "Génère une nouvelle clé et invalide immédiatement la clé actuelle.", + "rotateKeyEllipsis": "Renouveler la clé…", + "confirmAction": "Confirmer l'action", + "rotateKeyWarning": "L'ancienne clé cessera de fonctionner immédiatement.", + "confirm": "Confirmer", + "rotating": "Renouvellement…", + "cancel": "Annuler", + "checkoutBrandingTitle": "Personnalisation du paiement", + "checkoutBrandingDescription": "Personnalisez l'apparence de votre page de paiement.", + "storeLogo": "Logo de la boutique", + "clickOrDragToReplace": "Cliquez ou glissez pour remplacer", + "dropHere": "Déposez ici", + "uploadLogo": "Importer un logo", + "logoFormats": "PNG, JPG ou SVG · 2 Mo max.", + "removeLogo": "Supprimer le logo", + "colors": "Couleurs", + "primary": "Principale", + "secondary": "Secondaire", + "background": "Arrière-plan", + "preview": "Aperçu", + "sampleCheckout": "Exemple de paiement", + "payNow": "Payer maintenant", + "lowContrastWarning": "Les couleurs risquent de ne pas respecter les exigences de contraste WCAG. Pensez à les ajuster.", + "saving": "Enregistrement…", + "saveBranding": "Enregistrer l'image de marque", + "previewReceipt": "Aperçu du reçu", + "displayPreferencesTitle": "Préférences d'affichage", + "displayPreferencesDescription": "Ajustez l'affichage des valeurs dans le tableau de bord.", + "hideTrailingCents": "Masquer les centimes nuls", + "hideTrailingCentsDescription": "Les montants entiers comme 50 s'afficheront sans le suffixe .00.", + "webhookEndpointTitle": "Point de terminaison du webhook", + "webhookEndpointDescription": "Recevez les événements de paiement en temps réel sur votre serveur.", + "verified": "Vérifié", + "unverified": "Non vérifié", + "endpointUrl": "URL du point de terminaison", + "saveUrl": "Enregistrer l'URL", + "testing": "Test en cours…", + "sendTest": "Envoyer un test", + "domainVerification": "Vérification du domaine", + "hostTokenAt": "Hébergez ce jeton à l'adresse", + "verifying": "Vérification…", + "verifyDomain": "Vérifier le domaine", + "signingSecretTitle": "Secret de signature", + "signingSecretDescriptionPrefix": "Validez l'en-tête", + "signingSecretDescriptionSuffix": "sur les webhooks entrants.", + "hideWebhookSecret": "Masquer le secret du webhook", + "showWebhookSecret": "Afficher le secret du webhook", + "webhookSecretVisible": "Le secret du webhook est visible. Copiez-le maintenant — il ne s'affiche qu'une fois.", + "webhookSecretHidden": "Le secret du webhook est masqué. Activez l'affichage pour le consulter.", + "regenerateSecretEllipsis": "Régénérer le secret…", + "regenerateSecretWarning": "Le secret actuel cessera de fonctionner immédiatement.", + "regenerating": "Régénération…", + "dangerZoneDescription": "Actions irréversibles. Procédez avec prudence.", + "failedToLoadBranding": "Impossible de charger l'image de marque", + "failedToLoadWebhookSettings": "Impossible de charger les paramètres du webhook", + "failedToRotateKey": "Impossible de renouveler la clé", + "apiKeyRotatedSuccess": "Clé API renouvelée — mettez à jour les intégrations qui utilisent l'ancienne clé.", + "imageSizeLimit": "L'image doit faire moins de 2 Mo", + "logoUploaded": "Logo importé !", + "hexColorRequired": "{field} doit être une couleur hexadécimale valide", + "failedToSaveBranding": "Impossible d'enregistrer l'image de marque", + "brandingSaved": "Image de marque enregistrée", + "webhookUrlMustBeHttps": "L'URL du webhook doit utiliser HTTPS", + "webhookUrlInvalid": "URL invalide (ex. https://example.com/webhook)", + "failedToSaveWebhookUrl": "Impossible d'enregistrer l'URL du webhook", + "webhookUrlSaved": "URL du webhook enregistrée", + "webhookUrlCleared": "URL du webhook effacée", + "failedToVerifyDomain": "Impossible de vérifier le domaine du webhook", + "domainVerified": "Domaine vérifié", + "domainStillUnverified": "Domaine toujours non vérifié", + "failedToRegenerateSecret": "Impossible de régénérer le secret", + "webhookSecretRegenerated": "Secret du webhook régénéré — mettez à jour vos intégrations.", + "testWebhookRequestFailed": "La requête de webhook de test a échoué", + "failedToTestWebhook": "Impossible de tester le webhook", + "testWebhookSentStatus": "Webhook de test envoyé — statut {status}", + "colorPickerLabel": "Sélecteur de couleur {label}" + }, + "notificationCenter": { + "buttonLabel": "Notifications", + "buttonLabelWithCount": "Notifications ({count} non lues)", + "panelLabel": "Centre de notifications", + "heading": "Notifications", + "unreadCount": "{count} non lues", + "noAlerts": "Aucune nouvelle alerte", + "alertLabel": "Alerte", + "dismissLabel": "Ignorer la notification : {message}", + "notificationLabel": "Notification : {message}", + "timestampLabel": "Horodatage : {timestamp}" + }, + "fiatOnramp": { + "title": "Acheter / Déposer des fonds", + "triggerLabel": "Acheter / Déposer", + "description": "Déposez des devises fiduciaires via un anchor Stellar (SEP-0024). Les fonds arrivent sous forme de jetons directement dans votre portefeuille connecté.", + "selectAsset": "Sélectionner un actif", + "amountLabel": "Montant", + "amountOptional": "(facultatif)", + "amountPlaceholder": "ex. {amount} {asset}", + "anchorDomainLabel": "Domaine de l'anchor", + "anchorDomainPlaceholder": "ex. testanchor.stellar.org", + "continueButton": "Continuer vers l'anchor", + "footerNote": "Sécurisé par le réseau Stellar - norme SEP-0024", + "stepConnecting": "Connexion à l'anchor...", + "stepAuth": "En attente de la signature du portefeuille...", + "stepGenerating": "Préparation de votre formulaire de dépôt...", + "authHint": "Veuillez signer la transaction de challenge dans votre portefeuille pour vous connecter en toute sécurité à {domain}.", + "genericHint": "Cela ne devrait prendre qu'un instant.", + "iframeTitle": "Formulaire de dépôt de l'anchor", + "genericError": "Échec du dépôt" + }, + "kycPage": { + "title": "Vérification KYC", + "description": "Effectuez la vérification pour respecter les exigences réglementaires et accéder à toutes les fonctionnalités de la plateforme.", + "whyKyc": "Pourquoi le KYC ?", + "reasonComply": "Respecter les exigences réglementaires", + "reasonLimits": "Augmenter les plafonds de transaction", + "reasonFeatures": "Accéder aux fonctionnalités premium", + "reasonSecurity": "Sécurité du compte renforcée" + }, + "kycForm": { + "formTitle": "Formulaire de soumission KYC", + "step": "Étape", + "of": "sur", + "steps": "Étapes", + "completed": "terminée", + "current": "en cours", + "upcoming": "à venir", + "personalInfo": "Informations personnelles", + "addressInfo": "Adresse", + "documents": "Documents", + "review": "Vérification", + "firstName": "Prénom", + "lastName": "Nom", + "email": "E-mail", + "nationality": "Nationalité", + "dateOfBirth": "Date de naissance", + "street": "Rue", + "city": "Ville", + "state": "État / Région", + "postalCode": "Code postal", + "country": "Pays", + "idType": "Type de pièce d'identité", + "selectIdType": "Sélectionnez le type de pièce", + "passport": "Passeport", + "driversLicense": "Permis de conduire", + "nationalId": "Carte d'identité nationale", + "idNumber": "Numéro de pièce d'identité", + "idFront": "Recto de la pièce", + "idBack": "Verso de la pièce", + "selfie": "Selfie", + "required": "Obligatoire", + "validationError": "Veuillez remplir tous les champs obligatoires", + "back": "Retour", + "next": "Suivant", + "submit": "Soumettre le KYC", + "submitting": "Envoi...", + "submitError": "Échec de l'envoi", + "submitAnother": "Soumettre un autre dossier", + "successTitle": "KYC soumis avec succès !", + "successDescription": "Votre vérification KYC a été soumise et est en cours d'examen.", + "dash": "—" + }, + "network": { + "status": "État du réseau", + "refresh": "Vérifier l'état du réseau", + "latency": "Latence", + "connection": "Connexion", + "error": "Erreur", + "lastChecked": "Dernière vérification", + "online": "En ligne", + "offline": "Hors ligne", + "slow": "Lent", + "checking": "Vérification...", + "connectionQuality": "Qualité de la connexion", + "excellent": "Excellente", + "good": "Bonne", + "fair": "Moyenne", + "poor": "Faible" + }, + "darkModeTheme": { + "loading": "Chargement des paramètres du thème", + "errorCleared": "Erreur effacée. Nouvelle tentative de changement de thème.", + "switchingTo": "Passage au thème {theme}.", + "themeChanged": "Thème changé : {theme}.", + "themeError": "Impossible de changer de thème. Veuillez réessayer.", + "currentTheme": "Thème actuel : {theme}", + "systemTheme": "Système ({theme})", + "ariaError": "Sélecteur de thème en erreur — appuyez pour réessayer", + "ariaLabel": "Sélecteur de thème, thème actuel : {theme}. Appuyez pour passer au thème suivant.", + "ariaLoading": "Chargement du thème", + "errorTitle": "Erreur de thème : {error}. Appuyez pour réessayer.", + "titleSystem": "Thème : Système ({theme})", + "titleLight": "Thème : Clair", + "titleDark": "Thème : Sombre", + "description": "Alternez entre les thèmes clair, sombre et système. Le thème système suit les préférences de votre appareil. Appuyez sur Espace ou Entrée pour l'activer.", + "preferenceLabel": "Préférence de thème", + "preferenceSaved": "Préférence de thème enregistrée", + "preferenceRestored": "Préférence de thème restaurée", + "themes": { + "light": "clair", + "dark": "sombre", + "system": "système" + }, + "themeNames": { + "light": "Clair", + "dark": "Sombre", + "system": "Système" + }, + "sr": { + "sunIcon": "Icône soleil — mode clair actif", + "moonIcon": "Icône lune — mode sombre actif", + "systemIcon": "Icône écran — thème système actif", + "errorIcon": "Icône d'erreur — le thème n'a pas pu être appliqué" + } + }, + "networkStatus": { + "label": "État du réseau", + "ariaLabel": "Indicateur d'état du réseau Stellar", + "ariaLive": "État du réseau : {status}", + "statusOperational": "Opérationnel", + "statusDegraded": "Dégradé", + "statusPartialOutage": "Panne partielle", + "statusMajorOutage": "Panne majeure", + "statusUnknown": "Inconnu", + "statusChecking": "Vérification…", + "statusOffline": "Hors ligne", + "loading": "Vérification de l'état du réseau…", + "checking": "Nouvelle vérification…", + "lastChecked": "Dernière vérification {time}", + "lastCheckedNever": "Pas encore vérifié", + "checkNow": "Vérifier maintenant", + "retry": "Réessayer", + "retryCount": "Réessayer ({count})", + "dismiss": "Ignorer", + "errorHeading": "Échec de la vérification de l'état", + "errorMessage": "Impossible de joindre le réseau Stellar. Vérifiez votre connexion.", + "networkError": "Erreur réseau — nouvelle tentative…", + "latency": "{ms} ms", + "latencyLabel": "Temps de réponse : {ms} ms", + "ledger": "Ledger {sequence}", + "ledgerLabel": "Ledger actuel : {sequence}", + "horizon": "Horizon", + "stellar": "Stellar", + "mainnet": "Mainnet", + "testnet": "Testnet", + "networkLabel": "Réseau {network}", + "expandPanel": "Afficher les détails du réseau", + "collapsePanel": "Masquer les détails du réseau", + "detailsHeading": "Détails du réseau", + "horizonUrl": "URL Horizon", + "tooltipOperational": "Tous les systèmes fonctionnent normalement.", + "tooltipDegraded": "Certains services rencontrent des problèmes.", + "tooltipPartialOutage": "Certains services sont indisponibles.", + "tooltipMajorOutage": "Des services critiques sont indisponibles.", + "tooltipOffline": "Impossible de se connecter au réseau.", + "tooltipChecking": "Vérification de la connectivité réseau…", + "pollingPaused": "Interrogation de l'état en pause", + "pollingActive": "Interrogation de l'état active", + "autoRefresh": "Actualisation automatique toutes les {seconds} s" + }, + "portfolioChartWidget": { + "title": "Valeur du portefeuille", + "allocation": "Répartition", + "trend": "Tendance", + "loading": "Chargement du graphique du portefeuille...", + "historyLoading": "Chargement de l'historique de performance...", + "emptyAllocation": "Aucune donnée de répartition disponible pour l'instant.", + "emptyAssets": "Aucun actif de portefeuille disponible pour l'instant.", + "emptyHistory": "Aucun historique de performance disponible pour l'instant." + }, + "payment": { + "successTitle": "Paiement réussi !", + "successMessage": "Votre paiement a bien été traité. La transaction est désormais confirmée sur le réseau Stellar.", + "amountReceived": "Montant reçu", + "transactionId": "ID de transaction", + "successAnnounce": "Paiement réussi ! {amount} {asset} ont été reçus.", + "networkConfirmed": "Paiement confirmé sur le réseau Stellar.", + "dismissing": "Traitement…", + "optimisticNote": "Transaction soumise — en attente de confirmation du réseau.", + "successHint": "Appuyez sur Échap ou sur le bouton de fermeture pour masquer ce message." + }, + "common": { + "close": "Fermer l'animation de réussite", + "continue": "Continuer", + "error": "Une erreur s'est produite. Veuillez réessayer." + }, + "multisigModal": { + "title": "Approbation multisignature", + "closeModal": "Fermer la fenêtre", + "stepOf": "Étape {step} sur {total}", + "review": { + "heading": "Vérifier la transaction", + "description": "Veuillez vérifier les détails de la transaction et confirmer votre approbation.", + "sectionAriaLabel": "Section de vérification de la transaction", + "amount": "Montant", + "to": "Destinataire", + "issuer": "Émetteur", + "memo": "Mémo", + "signaturesLabel": "Signatures ({signed}/{required})", + "progressAriaLabel": "Progression des signatures", + "signersLabel": "Signataires", + "signersListAriaLabel": "Liste des signataires", + "signerFallbackName": "Signataire {id}", + "signedStatus": "Signé", + "notSignedStatus": "Non signé", + "signerWeight": "Poids : {weight} • {publicKey}...", + "signerAriaLabel": "{name}, poids : {weight}, statut : {status}", + "signButton": "Signer", + "signingButton": "Signature...", + "signButtonAriaLabel": "{action} pour {name}", + "expiresIn": "Expire dans {time}", + "submitButton": "Soumettre la transaction", + "submittingButton": "Envoi..." + }, + "processing": { + "heading": "Traitement de la transaction", + "description": "Traitement de votre transaction. Veuillez patienter." + }, + "confirm": { + "pendingHeading": "Transaction soumise", + "pendingDescription": "En attente de confirmation du réseau...", + "hashPendingLabel": "Hash de la transaction", + "confirmingButton": "Confirmation...", + "approvedHeading": "Transaction approuvée", + "approvedDescription": "Votre transaction multisignature a bien été soumise", + "hashLabel": "Hash de la transaction", + "closeButton": "Fermer" + }, + "error": { + "heading": "Échec de la transaction", + "defaultMessage": "Une erreur s'est produite lors du traitement de votre transaction", + "tryAgainButton": "Réessayer", + "closeButton": "Fermer", + "bannerTitle": "Erreur", + "clearErrorAriaLabel": "Effacer l'erreur", + "statusLabel": "Échoué" + }, + "expired": { + "heading": "Transaction expirée", + "statusLabel": "Transaction expirée", + "badge": "Expirée", + "description": "Cette transaction a expiré et ne peut plus être signée ni soumise.", + "closeButton": "Fermer" + }, + "announcements": { + "processing": "Traitement de votre transaction. Veuillez patienter.", + "confirmed": "Transaction approuvée avec succès.", + "failed": "Échec de la transaction. Consultez le détail de l'erreur ci-dessous." + }, + "toasts": { + "signed": "Transaction signée avec succès", + "submitted": "Transaction soumise avec succès" + } + }, + "transactionFilters": { + "title": "Filtres", + "landmarkLabel": "Filtres des transactions", + "closeFilters": "Fermer les filtres", + "ariaLabel": "Effacer la recherche", + "applyingToResults": "Application aux résultats…", + "clearAll": "Effacer tous les filtres", + "clearAllAriaLabel": "Effacer {count, plural, one {1 filtre actif} other {# filtres actifs}}", + "clearing": "Effacement…", + "search": { + "label": "Rechercher", + "placeholder": "Rechercher par ID ou description", + "syncLabel": "Recherche…" + }, + "status": { + "label": "Statut", + "all": "Tous les statuts", + "pending": "En attente", + "confirmed": "Confirmé", + "failed": "Échoué", + "refunded": "Remboursé", + "syncLabel": "Mise à jour du statut…" + }, + "asset": { + "label": "Filtre d'actif", + "all": "Tous", + "syncLabel": "Mise à jour de l'actif…" + }, + "dateRange": { + "label": "Période", + "from": "Du", + "to": "Au", + "fromSyncLabel": "Mise à jour de la date de début…", + "toSyncLabel": "Mise à jour de la date de fin…" + } + }, + "dashboardShell": { + "openNavigationMenu": "Ouvrir le menu de navigation" + } +} diff --git a/frontend/messages/pt.json b/frontend/messages/pt.json index d3396e96..ffeddeb0 100644 --- a/frontend/messages/pt.json +++ b/frontend/messages/pt.json @@ -33,7 +33,8 @@ "options": { "en": "English", "es": "Espanol", - "pt": "Portugues" + "pt": "Portugues", + "fr": "Frances" } }, "nav": { @@ -205,6 +206,11 @@ "title": "Historico de pagamentos", "description": "Monitore todas as transacoes recebidas e seu status de verificacao." }, + "portfolioChart": { + "valueTitle": "Valor do portfólio", + "allocation": "Alocação", + "trend": "Tendência" + }, "recentPayments": { "missingApiKey": "Chave API nao encontrada. Cadastre-se ou faca login.", "fetchFailed": "Falha ao buscar pagamentos", @@ -577,6 +583,23 @@ "successDescription": "Sua verificacao KYC foi enviada e esta em revisao.", "dash": "—" }, + "network": { + "status": "Status da rede", + "refresh": "Verificar status da rede", + "latency": "Latência", + "connection": "Conexão", + "error": "Erro", + "lastChecked": "Última verificação", + "online": "Online", + "offline": "Offline", + "slow": "Lenta", + "checking": "Verificando...", + "connectionQuality": "Qualidade da conexão", + "excellent": "Excelente", + "good": "Boa", + "fair": "Regular", + "poor": "Ruim" + }, "darkModeTheme": { "loading": "Carregando configuracoes de tema", "errorCleared": "Erro eliminado. Tentando alternar o tema.", @@ -787,5 +810,8 @@ "fromSyncLabel": "Atualizando data inicial…", "toSyncLabel": "Atualizando data final…" } + }, + "dashboardShell": { + "openNavigationMenu": "Abrir menu de navegação" } } diff --git a/frontend/src/app/(authenticated)/layout.tsx b/frontend/src/app/(authenticated)/layout.tsx index 544513a8..e6afc530 100644 --- a/frontend/src/app/(authenticated)/layout.tsx +++ b/frontend/src/app/(authenticated)/layout.tsx @@ -11,6 +11,7 @@ import SupportOverlay from "@/components/SupportOverlay"; import UserNav from "@/components/UserNav"; import { useHydrateMerchantStore } from "@/lib/merchant-store"; import { motion } from "framer-motion"; +import { useTranslations } from "next-intl"; /** * #1189 — Bundle size optimisation: @@ -29,6 +30,7 @@ export default function AuthenticatedLayout({ }: { children: React.ReactNode; }) { + const t = useTranslations("dashboardShell"); const [mobileSidebarOpen, setMobileSidebarOpen] = useState(false); useHydrateMerchantStore(); @@ -50,7 +52,7 @@ export default function AuthenticatedLayout({ type="button" onClick={() => setMobileSidebarOpen(true)} className="inline-flex h-10 w-10 items-center justify-center rounded-xl border border-[#E8E8E8] bg-white text-[#6B6B6B] transition-all hover:border-[#0A0A0A] hover:bg-[#F9F9F9] hover:text-[#0A0A0A] lg:hidden" - aria-label="Open navigation menu" + aria-label={t("openNavigationMenu")} aria-controls="dashboard-sidebar-mobile" aria-expanded={mobileSidebarOpen} > diff --git a/frontend/src/components/LocaleSwitcher.test.tsx b/frontend/src/components/LocaleSwitcher.test.tsx new file mode 100644 index 00000000..4b903e98 --- /dev/null +++ b/frontend/src/components/LocaleSwitcher.test.tsx @@ -0,0 +1,64 @@ +import React from "react"; +import { fireEvent, render, screen } from "@testing-library/react"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import en from "../../messages/en.json"; + +const replace = vi.fn(); +let mockLocale = "en"; + +vi.mock("next-intl", () => ({ + useLocale: () => mockLocale, + useTranslations: (namespace: string) => (key: string) => { + const scope = (en as Record)[namespace] as Record; + return key + .split(".") + .reduce((node, part) => (node as Record)?.[part], scope) as string; + }, +})); + +vi.mock("next/navigation", () => ({ + useSearchParams: () => new URLSearchParams("tab=all"), +})); + +vi.mock("@/i18n/navigation", () => ({ + usePathname: () => "/dashboard", + useRouter: () => ({ replace }), +})); + +import LocaleSwitcher from "./LocaleSwitcher"; + +describe("LocaleSwitcher (#428)", () => { + beforeEach(() => { + replace.mockClear(); + mockLocale = "en"; + document.cookie = "NEXT_LOCALE=; path=/; max-age=0"; + }); + + it("offers English, Spanish, Portuguese and French", () => { + render(); + const options = screen.getAllByRole("option").map((option) => option.getAttribute("value")); + expect(options).toEqual(["en", "es", "pt", "fr"]); + expect(screen.getByRole("option", { name: "Francais" })).toBeInTheDocument(); + }); + + it("persists the chosen locale and re-renders the current dashboard route", () => { + render(); + fireEvent.change(screen.getByLabelText("Select language"), { target: { value: "fr" } }); + + expect(document.cookie).toContain("NEXT_LOCALE=fr"); + expect(replace).toHaveBeenCalledWith("/dashboard?tab=all", { locale: "fr", scroll: false }); + }); + + it("does nothing when the active locale is selected again", () => { + mockLocale = "fr"; + render(); + fireEvent.change(screen.getByLabelText("Select language"), { target: { value: "fr" } }); + expect(replace).not.toHaveBeenCalled(); + }); + + it("falls back to English when the active locale is unsupported", () => { + mockLocale = "de"; + render(); + expect((screen.getByLabelText("Select language") as HTMLSelectElement).value).toBe("en"); + }); +}); diff --git a/frontend/src/i18n/config.ts b/frontend/src/i18n/config.ts index 71a664a4..25714e82 100644 --- a/frontend/src/i18n/config.ts +++ b/frontend/src/i18n/config.ts @@ -1,4 +1,4 @@ -export const locales = ["en", "es", "pt"] as const; +export const locales = ["en", "es", "pt", "fr"] as const; export type AppLocale = (typeof locales)[number]; @@ -22,6 +22,8 @@ export function localeToLanguageTag(locale: string): string { return "es-ES"; case "pt": return "pt-BR"; + case "fr": + return "fr-FR"; case "en": default: return "en-US"; diff --git a/frontend/src/i18n/messages.test.ts b/frontend/src/i18n/messages.test.ts new file mode 100644 index 00000000..1b425dae --- /dev/null +++ b/frontend/src/i18n/messages.test.ts @@ -0,0 +1,109 @@ +import { describe, expect, it } from "vitest"; +import en from "../../messages/en.json"; +import es from "../../messages/es.json"; +import pt from "../../messages/pt.json"; +import fr from "../../messages/fr.json"; +import { + defaultLocale, + isValidLocale, + localeToLanguageTag, + locales, + resolveAppLocale, +} from "@/i18n/config"; + +type Messages = { [key: string]: string | Messages }; + +const catalogues: Record = { en, es, pt, fr }; + +function flatten(messages: Messages, prefix = ""): Record { + return Object.entries(messages).reduce>((acc, [key, value]) => { + const path = prefix ? `${prefix}.${key}` : key; + if (typeof value === "string") { + acc[path] = value; + } else { + Object.assign(acc, flatten(value, path)); + } + return acc; + }, {}); +} + +/** Top-level ICU argument names, e.g. "{count, plural, one {# x}}" -> ["count"]. */ +function icuArguments(message: string): string[] { + const names = new Set(); + let depth = 0; + for (let i = 0; i < message.length; i += 1) { + const char = message[i]; + if (char === "{") { + if (depth === 0) { + const match = /^\{\s*(\w+)\s*[,}]/.exec(message.slice(i)); + if (match) names.add(match[1]); + } + depth += 1; + } else if (char === "}") { + depth = Math.max(0, depth - 1); + } + } + return [...names].sort(); +} + +const english = flatten(en as Messages); + +describe("Dashboard i18n catalogues (#428)", () => { + it("ships a message catalogue for every configured locale", () => { + expect([...locales].sort()).toEqual(["en", "es", "fr", "pt"]); + for (const locale of locales) { + expect(catalogues[locale], `missing messages/${locale}.json`).toBeDefined(); + } + }); + + it.each(["es", "pt", "fr"])("%s has exactly the same keys as en", (locale) => { + const translated = flatten(catalogues[locale]); + const missing = Object.keys(english).filter((key) => !(key in translated)); + const extra = Object.keys(translated).filter((key) => !(key in english)); + expect(missing).toEqual([]); + expect(extra).toEqual([]); + }); + + it.each(["es", "pt", "fr"])("%s keeps every ICU placeholder used by en", (locale) => { + const translated = flatten(catalogues[locale]); + const mismatches = Object.keys(english).filter( + (key) => icuArguments(english[key]).join() !== icuArguments(translated[key] ?? "").join(), + ); + expect(mismatches).toEqual([]); + }); + + it.each(["es", "pt", "fr"])("%s has no empty translations", (locale) => { + const empty = Object.entries(flatten(catalogues[locale])) + .filter(([, value]) => value.trim() === "") + .map(([key]) => key); + expect(empty).toEqual([]); + }); + + it("labels every locale in the header switcher in every catalogue", () => { + for (const catalogue of Object.values(catalogues)) { + const options = (catalogue.localeSwitcher as Messages).options as Messages; + expect(Object.keys(options).sort()).toEqual([...locales].sort()); + } + }); + + it("translates the dashboard shell navigation label", () => { + expect(flatten(fr as Messages)["dashboardShell.openNavigationMenu"]).toBe( + "Ouvrir le menu de navigation", + ); + }); +}); + +describe("i18n config", () => { + it("accepts fr as a valid locale and maps it to fr-FR", () => { + expect(isValidLocale("fr")).toBe(true); + expect(resolveAppLocale("fr")).toBe("fr"); + expect(localeToLanguageTag("fr")).toBe("fr-FR"); + }); + + it("falls back to the default locale for unknown or empty values", () => { + expect(isValidLocale("de")).toBe(false); + expect(resolveAppLocale("de")).toBe(defaultLocale); + expect(resolveAppLocale(undefined)).toBe(defaultLocale); + expect(localeToLanguageTag("xx")).toBe("en-US"); + }); +});