feat(health): add dependency-aware readiness and recovery diagnostics

.env.example

@@ -56,3 +56,8 @@ PAGINATION_MAX_SCAN=10000
 # Valid scopes: transfers:read, transfers:write, users:read, users:write, audit:read
 # Example (single line, properly escaped for shell):
 # API_TOKENS={"my-secret-token":["transfers:read","transfers:write","users:read","users:write","audit:read"]}
+
+# Health / readiness probes
+# Per-dependency time budget for GET /api/health/ready. Kept well below
+# REQUEST_TIMEOUT_MS so a hung dependency cannot stall the probe.
+HEALTH_CHECK_TIMEOUT_MS=1000

CHANGELOG.md

@@ -18,6 +18,12 @@ When preparing a new release:
 
 ### Added
 
+- Dependency-aware readiness diagnostics: `GET /api/health/ready` now probes
+  store, payments (Stellar), and FX under a per-check timeout
+  (`HEALTH_CHECK_TIMEOUT_MS`), returns redacted reason codes on failure, and
+  recovers without a process restart. Liveness (`GET /api/health/live`) stays
+  process-only so orchestrators do not flap restarts during dependency outages.
+
 - Cursor pagination for `GET /api/transfers` and `GET /api/audit`. Pass
   `?cursor=` (with optional `?order=asc|desc`) to page by an indexed position
   instead of a row offset; responses carry a `pageInfo` block with

README.md

@@ -227,9 +227,21 @@ The API implements Cache-Control response headers for security and efficiency:
 
 ### Health
 
-- `GET /api/health` — service health snapshot (version, uptime, env).
-- `GET /api/health/live` — liveness probe.
-- `GET /api/health/ready` — readiness probe with dependency checks.
+- `GET /api/health` — service health snapshot (version, uptime, env). Not a dependency gate.
+- `GET /api/health/live` (also `HEAD`) — liveness probe. Process-only; stays responsive during dependency outages
+  and exhausted API quotas. Liveness polls do not consume the business API budget. Common security, cache,
+  parsing, timeout, request-ID and logging middleware still apply. Readiness, health snapshots and business
+  routes retain the API quota; unmatched methods or subpaths continue through the existing route stack.
+- `GET /api/health/ready` — readiness probe with bounded checks for store, payments (Stellar), and FX.
+  Returns `200` when ready and `503` with redacted reason codes when a dependency is down or times out.
+  Probes re-run on every request so recovery does not require a restart.
+  Tune the per-check budget with `HEALTH_CHECK_TIMEOUT_MS` (default `1000`).
+  Payment and FX `ping()` adapters may return their result immediately or as a Promise;
+  the default probes await that result before checking `ok`. Rejections remain attached
+  to the readiness error path, including after a timeout. The deadline bounds waiting,
+  but does not cancel the underlying Promise or interrupt synchronous blocking work.
+  The shipped adapters remain synchronous mocks: Stellar configuration and loaded FX
+  data checks do not establish live provider reachability.
 - `GET /api/version` — service name and version.
 
 ### Rates & quotes

docs/READINESS_PROBE_SHARING.md

@@ -0,0 +1,34 @@
+# Bounded readiness probe work
+
+Follow-up to the existing dependency-readiness contribution, PR #140 / issue #134.
+
+## Behavior
+
+A request deadline previously stopped waiting without stopping the underlying operation. Repeated readiness requests could therefore accumulate unfinished calls to a hung payment or FX adapter. The service now shares only an unfinished operation for each dependency and adapter generation. Every request retains its own timeout; a timed-out request removes its subscriber instead of leaving another callback attached to the hung operation. Completed results are discarded, not cached as healthy or failed.
+
+The payment and FX method references and their receiver objects are captured before execution. Replacing an adapter does not inherit the old adapter's pending operation; an old completion cannot evict its successor. Existing stable reason codes, redaction, forced test states and the liveness/business-API rate-limit distinction are unchanged.
+
+**Limit:** this bounds outstanding work for an unchanged adapter; it does not cancel an arbitrary Promise. A truly never-settling adapter stays not-ready rather than spawning more calls. Recovery occurs when the operation settles or the adapter is replaced. A real external driver still needs transport-level cancellation/timeouts. Deliberately replacing adapters repeatedly can leave their old uncancellable operations outstanding; this is not a global provider quota or fleet-wide limiter.
+
+## Executed evidence — October 4, 2026
+
+Source commit `b4c9505ee1ac758ebb3fff659bbe39747febaa6b`, tree `9a49ce0645d3b95eed16aa8d2671dbc19682e989`, sole parent `67c4e52d8bcb6b286922104702e717c75cdfa053`.
+
+- Production service blob: `f1d574ff23df4440aa350ca57ace9df51b88080b`.
+- Shared-operation helper: `5dfabe9ed6437b00e336b0eb8e3dca58e9bc5279`.
+- New regression file: `6148d44d1e1f46d05a2920531ae0bf9986028b44`.
+
+[Final run 37193635966](https://github.com/woahwhattheheck/RemitFlow-Backend/actions/runs/37193635966), Node 24.21.0 on Ubuntu, executed:
+
+```sh
+npm ci --no-audit --no-fund
+node --unhandled-rejections=strict --test --test-reporter=tap test/dependencyHealth.test.js test/readinessProbeSharing.test.js
+```
+
+**29 tests passed; zero failures, skips or cancellations.** Existing health tests and dependency manifests stayed byte-identical. This includes actual Express HTTP requests and default adapter wrappers, with controlled local provider promises; no live Stellar/FX/database connection is claimed.
+
+The repeated-timeout regression sent 40 readiness requests. Before the repair it made 40 provider calls; afterward it made one unfinished provider call, retained liveness HTTP 200 and recovered after completion. The other new cases cover independent caller deadlines, shared redacted rejection and a late predecessor completion. The earlier default-adapter tests also prove replacement recovery.
+
+The initial baseline run [37192889105](https://github.com/woahwhattheheck/RemitFlow-Backend/actions/runs/37192889105) reproduced all four new failures. Its wrapper then stopped because it expected TAP while Node emitted the spec reporter; that run is not represented as a repair pass. The first repaired candidate [37193040714](https://github.com/woahwhattheheck/RemitFlow-Backend/actions/runs/37193040714) passed 27 tests but failed two existing adapter-replacement recovery cases. Capturing and keying the actual adapter generation repaired those failures without changing any existing assertion. The original baseline was not rerun.
+
+[Final artifact 11299902454](https://github.com/woahwhattheheck/RemitFlow-Backend/actions/runs/37193635966/artifacts/11299902454) retains TAP output, commands, source diff and identities. ZIP SHA-256: `a08f9dad1374587e4b202b190fa97b897a602522d20888a670d45dd726995c5f`. Earlier baseline and failed-candidate artifacts remain attached to their runs. This result does not establish full-suite, deployment, live-provider, bounty-acceptance or payment status.

docs/READINESS_TIMER_RANGE.md

@@ -0,0 +1,54 @@
+# Readiness timer range — 4 October 2026
+
+Continuation of RemitFlow-Backend PR 140 / issue 134 on the original branch.
+
+## Behavior
+
+An overflowing configured delay or an infinite override previously reached Node's
+`setTimeout`. Node replaces delays above 2,147,483,647 milliseconds with one
+millisecond, so a healthy asynchronous dependency could incorrectly time out.
+The returned `timeoutMs` also failed to describe the effective timer.
+
+The service now normalizes configured budgets, readiness overrides and direct
+`runCheck` arguments. Invalid overrides fall back to the configured valid budget;
+an invalid configured budget falls back to the existing 1,000-millisecond default.
+Positive supported values retain Node's integer truncation and one-millisecond
+minimum. The supported maximum remains valid. Probe sharing, release, adapter
+identity, reason codes and timeout recovery are unchanged.
+
+Node timer contract: https://nodejs.org/api/timers.html#settimeoutcallback-delay-args
+
+## Focused execution
+
+Node.js v22.16.0, Linux x64. Exactly the same three maintained cases were run
+against the original and changed complete service source, with the unchanged
+complete `inFlightProbe.js`. The baseline failed all three cases and emitted
+`TimeoutOverflowWarning`; the changed source passed all three with no failures,
+skips, cancellations or overflow warning. No full suite or build was run.
+
+The cases cover configured overflow, override fallback, reported integer budgets,
+the maximum supported delay, the direct-call path, a genuine short timeout and
+recovery after completion. Healthy probes use actual Node timers, not a fake clock.
+
+Execution used a task-local CommonJS preload providing four explicit collaborator
+modules: mutable health config, two empty store maps and unused payment/FX ping
+adapters. The maintained test replaces probes through the service's existing test
+hook. The real service, promise race, timer handling and probe pool ran unchanged
+between the two executions except for the timeout-normalization patch. This is
+source-level service execution, not HTTP routing, live dependencies or full-app CI.
+In a regular checkout with its dependencies installed, run:
+
+```sh
+node --test test/dependencyHealthTimerRange.test.js
+```
+
+| Source | Git blob |
+| --- | --- |
+| Baseline service at `4fd4284ccead15abede008a95779c680aab2682b` | `f1d574ff23df4440aa350ca57ace9df51b88080b` |
+| Changed complete service | `7cc1334e9cf79ccdf621ad59b6b015480e10c45c` |
+| Unchanged complete probe pool | `5dfabe9ed6437b00e336b0eb8e3dca58e9bc5279` |
+| Three-case maintained regression | `329dd0c28acaf5ca71f0bd5474204b167b547ca9` |
+
+The downloaded earlier dependency artifact was not needed or executed for this
+bounded run. Historical test results retain their original source pins; this
+continuation makes no deployment, maintainer acceptance, award or payment claim.

src/app.js

@@ -6,6 +6,7 @@ const morgan = require('morgan');
 
 const config = require('./config');
 const routes = require('./routes');
+const healthController = require('./controllers/healthController');
 const securityHeaders = require('./middleware/securityHeaders');
 const cacheControl = require('./middleware/cacheControl');
 const requestTimeout = require('./middleware/requestTimeout');
@@ -46,6 +47,11 @@ function createApp() {
   }
   app.use(requestLogger);
 
+  // Process liveness must not consume or depend on the business API budget.
+  // Express also serves HEAD through this GET route; other paths/methods
+  // continue through the normal API middleware below.
+  app.get('/api/health/live', healthController.getLiveness);
+
   // Basic abuse protection on the API surface.
   app.use('/api', rateLimit(config.rateLimit));
 

src/config/index.js

@@ -69,6 +69,13 @@ const config = {
     maxScan: parseInt(process.env.PAGINATION_MAX_SCAN, 10) || 10000,
   },
 
+
+  health: {
+    // Per-dependency budget for readiness probes. Must stay well below the
+    // request timeout so a hung dependency cannot stall the readiness route.
+    checkTimeoutMs: parseInt(process.env.HEALTH_CHECK_TIMEOUT_MS, 10) || 1000,
+  },
+
   apiTokens: (() => {
     try {
       if (process.env.API_TOKENS) {

src/controllers/healthController.js

@@ -2,14 +2,21 @@
 
 const config = require('../config');
 const { name, version } = require('../../package.json');
+const dependencyHealth = require('../services/dependencyHealthService');
 
 /**
  * Health controller.
+ *
+ * Liveness answers "is the process up?" and must stay cheap and dependency-
+ * free so orchestrators do not restart a process that is merely waiting on
+ * a degraded dependency. Readiness answers "can this instance serve traffic?"
+ * by running bounded, redacted dependency probes that recover automatically
+ * once the dependency is healthy again.
  */
 
 /**
  * GET /api/health
- * Reports basic liveness information.
+ * Reports basic process information (not a dependency gate).
  */
 function getHealth(req, res) {
   res.json({
@@ -36,23 +43,28 @@ function getVersion(req, res) {
 
 /**
  * GET /api/health/live
- * Liveness probe: confirms the process is up and responding.
+ * Liveness probe: confirms the process is up and responding. Intentionally
+ * ignores store / payment / FX state so outages do not flap restarts.
  */
 function getLiveness(req, res) {
   res.json({ status: 'alive', timestamp: new Date().toISOString() });
 }
 
 /**
  * GET /api/health/ready
- * Readiness probe: confirms dependencies needed to serve traffic are up.
- * The demo store is always in-memory, so readiness simply mirrors liveness.
+ * Readiness probe: bounded checks against store, payments, and FX.
+ * Returns 200 when every dependency is healthy and 503 otherwise. Reason
+ * codes are redacted; recovery is automatic on the next successful probe.
  */
-function getReadiness(req, res) {
-  res.json({
-    status: 'ready',
-    checks: { store: 'ok' },
+async function getReadiness(req, res) {
+  const result = await dependencyHealth.evaluateReadiness();
+  const payload = {
+    status: result.status,
+    checks: result.checks,
+    timeoutMs: result.timeoutMs,
     timestamp: new Date().toISOString(),
-  });
+  };
+  res.status(result.ready ? 200 : 503).json(payload);
 }
 
 module.exports = {

src/routes/healthRoutes.js

@@ -9,9 +9,6 @@ const router = express.Router();
 // GET /api/health
 router.get('/', asyncHandler(healthController.getHealth));
 
-// GET /api/health/live
-router.get('/live', asyncHandler(healthController.getLiveness));
-
 // GET /api/health/ready
 router.get('/ready', asyncHandler(healthController.getReadiness));