feat: add provider circuit breakers, graceful shutdown, health probes, and outbound retries

[Omoboi-dev]

Summary of Changes

This pull request addresses issues #1792, #1793, #1794, and #1795 by introducing provider circuit breaking, graceful shutdown with connection draining, distinct liveness/readiness probes, and bounded outbound timeouts with idempotent retries.

---

#1793: Add circuit breakers for provider calls

• Provider Breaker Registry: Implemented ProviderCircuitBreakerRegistry managing isolated circuit breaker instances per external provider (stripe, paddle, circle, stellar, etc.) to prevent cascading outages.
• Fallback Handling: Added support for provider-specific registered fallback functions as well as inline fallbacks executed when breakers are open or requests fail.
• Tests: Added comprehensive unit tests verifying breaker trip, open-circuit fast failure, fallback invocation, half-open transitions, and successful recovery.

#1792: Add graceful shutdown

• Connection Draining: Server tracks active sockets and in-flight HTTP requests (inFlightRequests). On shutdown, stops accepting new requests (returning 503 while draining), closes idle keep-alive connections, and waits for active requests to finish up to drainTimeoutMs (default 10,000ms).
• Forced Teardown: Enforces drain timeout bounds, forcefully destroying remaining sockets if in-flight requests exceed the timeout window.
• Signal Handling: Listens to SIGTERM and SIGINT signals with clean event handler cleanup.
• Tests: Added tests verifying that in-flight requests complete before shutdown, forced destruction triggers on timeout expiry, and new requests are rejected with 503 during draining.

#1795: Add health and readiness probes

• Distinct Endpoints:
  • /health/live and /livez: Liveness probe returning 200 OK with uptime.
  • /health/ready and /readyz: Readiness probe evaluating critical dependencies (database query SELECT 1 and cache isHealthy()). Returns 503 Service Unavailable if database/cache is down or if the server is in draining state.
• Backward Compatibility: Preserved /health endpoint for existing monitors.
• Tests: Added unit tests validating 200 on healthy dependencies, 503 when the database or cache fails, and 503 during server shutdown draining.

#1794: Add outbound request timeouts and retries

• Request Timeouts: Built outboundRequest wrapping fetch with AbortController (default 5,000ms timeout) that throws OutboundTimeoutError.
• Bounded Idempotent Retries: Automatically retries transient network errors (timeouts, 429, 5xx) only on idempotent methods (GET, PUT, DELETE, HEAD, OPTIONS) or requests carrying an Idempotency-Key / X-Idempotency-Key header with linear backoff. Restricts non-idempotent POST calls without idempotency keys to a single attempt.
• Tests: Added unit tests verifying timeouts, idempotency detection, bounded retries on 500/503/429, and non-retry on non-idempotent requests.

---

Closes #1792
Closes #1793
Closes #1794
Closes #1795

[drips-wave]

@Omoboi-dev Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits