[Simultech369]

Summary

Closes #550.

This pull request delivers a production-grade resilience layer for upstream price oracle and external rate lookups, resolving cascading failure vulnerabilities, stampedes, and unbounded hanging requests.

Core Enhancements

1. Exponential Backoff with Full Jitter (src/oracle/retry.ts):

   • Strictly bounded execution attempts governed by maxAttempts (default 3: 1 initial attempt + 2 retries).
   • Exponential scaling delay: $\text{delay} = \text{random}(0, , \min(\text{baseDelayMs} \cdot 2^{\text{attempt}-1}, , \text{maxDelayMs}))$.
   • Full jitter prevents thundering herd spikes against recovering upstream providers.
   • Configurable ceilings (maxDelayMs), multiplier, and injectable sleep timer for testing.

2. Per-Dependency Circuit Breaker (src/oracle/circuitBreaker.ts):

   • Complete 3-state finite state machine: CLOSED -> OPEN -> HALF_OPEN -> CLOSED.
   • Fail-Fast: Once consecutive failures exceed failureThreshold (default 5), the breaker enters OPEN and immediately rejects subsequent traffic with typed CircuitBreakerOpenError / HTTP 503 upstream_unavailable without making network calls.
   • Canary Probing: After cooldownMs (default 30s) elapses, transitions to HALF_OPEN and admits a single probe request. Success closes the circuit; failure immediately reopens it.
   • Per-Dependency Isolation: State is maintained per dependency name via circuitBreakerRegistry. An outage in one upstream oracle never trips or impacts another.

3. Intelligent Error Classification & Idempotency:

   • Only transient errors are retried: network timeouts/drops (ECONNRESET, ETIMEDOUT, ECONNREFUSED, ENOTFOUND, EAI_AGAIN, EPIPE), HTTP 5xx responses, and HTTP 429 rate limits.
   • Non-retryable client errors (HTTP 4xx, schema validation errors, syntax errors) fail fast immediately on attempt 1.
   • Non-idempotent operations (isIdempotent: false) bypass retries completely to prevent duplicate side effects.

4. API Endpoints & Runtime Tuning (src/index.ts):

   • GET /api/v1/oracle/status: Query breaker state and metrics (per dependency or globally).
   • GET /api/v1/oracle/rate/:source/:destination: Exchange rate lookups guarded by the retry and circuit breaker pipeline.
   • PATCH /api/v1/config: Live dynamic tuning of oracleRetryAttempts, oracleBackoffBaseMs, oracleBreakerThreshold, and oracleBreakerCooldownMs.

5. Prometheus Metrics Exposition (GET /api/v1/metrics):

   • stableroute_circuit_breaker_state{dependency="..."} (0=CLOSED, 1=HALF_OPEN, 2=OPEN).
   • stableroute_circuit_breaker_failures_total{dependency="..."}.
   • stableroute_circuit_breaker_successes_total{dependency="..."}.
   • stableroute_circuit_breaker_short_circuits_total{dependency="..."}.

6. OpenAPI 3.0.3 Spec & Technical Documentation:

   • Documented endpoints in src/openapi.ts passing the router drift guard test.
   • Full technical architecture reference guide in docs/price-oracle-resilience.md.

---

Edge Cases Covered & Tested

All 5 required edge cases are covered with dedicated tests in src/__tests__/circuitBreaker.test.ts and src/__tests__/oracleIntegration.test.ts:

1. Transient failure then success: Verified retried and succeeds on attempt $N$.
2. $N$ consecutive failures: Verified breaker transitions from CLOSED to OPEN and subsequent calls fail fast with typed upstream_unavailable.
3. Cooldown elapses: Verified transition to HALF_OPEN; probe success transitions back to CLOSED.
4. Non-retryable error: Verified non-retryable 4xx/syntax errors abort immediately on attempt 1.
5. Max attempt bound: Verified retry loop strictly halts when maxAttempts is reached.
6. HALF_OPEN probe failure: Verified probe failure immediately returns to OPEN.
7. HALF_OPEN probe limit: Verified concurrent calls fail fast while probe is in-flight.
8. Idempotency guard: Verified non-idempotent operations skip retries even on 500 error.
9. Per-dependency isolation: Verified dependency A outage does not affect dependency B.

---

Verification Results

• npm run lint: Passed with 0 errors.
• npm run build: Passed with 0 errors (tsc clean).
• npm test: 37 / 37 test suites passed, 1004 / 1004 tests passed.

[mikewheeleer]

Closing as part of a repository reset; new issues will be opened shortly. Thanks for contributing!