feat: Process Resiliency, Infrastructure Health Probes, and Reliability Telemetry Cluster

.changeset/reliability-resilience-cluster.md

@@ -0,0 +1,15 @@
+---
+'lens': minor
+---
+
+Comprehensive Process Resiliency, Reliability, Health Probes & Telemetry Cluster:
+- #348: Handle Horizon 429 rate limiting with Retry-After backoff, non-2xx pre-parse detection, and remaining quota gauge metrics
+- #345: Active retention path mode telemetry, Redis periodic reconnection recovery, and advisory-locked safe concurrent pruning
+- #344: Unhandled rejection and uncaught exception guard logging full stack trace and exiting non-zero cleanly
+- #342: Webhook dead-letter store recording exhausted deliveries without persisting secrets, with failure categorisation
+- #341: Full jitter on exponential webhook retry delays with injectable randomSource
+- #339: Configurable database connection pool with poolSize and timeout arithmetic, pool utilisation gauges, and DatabasePoolExhaustedError
+- #334: GET /live lightweight liveness probe and GET /ready boot & dependency readiness probe
+- #331: Ingester crash supervisor with capped exponential backoff, healthy run reset, and crash history telemetry
+- #329: Distinct VenueFetchError and failure tracking preventing empty-venue confusion and unalerted reweighting
+- #350: Comprehensive operational service runbook in docs/RUNBOOK.md and README.md linkage

.env.example

@@ -16,6 +16,13 @@ HOST=0.0.0.0
 DATABASE_URL=postgresql://lens:lens@localhost:5432/lens
 # Direct unpooled PostgreSQL connection string (used by Prisma migrations and CLI to bypass connection poolers)
 DIRECT_DATABASE_URL=postgresql://lens:lens@localhost:5432/lens
+# Maximum connections in the shared pg connection pool (default: 20)
+# NOTE: Ingesters and API share this pool. 3 replicas x 20 conn = 60 max conn, within Postgres 100 limit.
+DATABASE_POOL_SIZE=20
+# Optional alias for DATABASE_POOL_SIZE
+DATABASE_POOL_MAX=20
+# Connection acquisition timeout in milliseconds (default: 10000)
+DATABASE_POOL_TIMEOUT_MS=10000
 # Retention window in days for partitioned price_points table (default: 30)
 PRICE_POINTS_RETENTION_DAYS=30
 # Explicit opt-in flag to drop expired partitions older than PRICE_POINTS_RETENTION_DAYS (default: false, DESTRUCTIVE).

README.md

@@ -31,7 +31,9 @@ Aggregates price data from Stellar's Classic Order Book (SDEX) and AMM Liquidity
 | GET | `/screener?sortBy=volume&order=desc&limit=20` | Screen pairs by volume, 24h change, liquidity and price |
 | GET | `/benchmark/:asset?target=USD` | Peg-deviation statistics versus a target |
 | GET | `/basket?asset=XLM&asset=USDC&weight=0.6&weight=0.4` | Weighted price of a basket of assets |
-| GET | `/status` | Indexer health |
+| GET | `/status` | Indexer health and operational telemetry |
+| GET | `/live` | Liveness check (process event loop responsiveness) |
+| GET | `/ready` | Readiness check (validates database and cache connectivity) |
 | GET | `/discovery/resources?type=&payTo=&network=&extensions=&limit=&offset=` | Bazaar catalog of x402-discoverable resources (spec: [`bazaar`](https://github.com/x402-foundation/x402/blob/main/specs/extensions/bazaar.md)) |
 | GET | `/usage/me` | Usage and quota for the calling API key (requires `Authorization: Bearer <key>`) |
 | GET | `/supported` | Payment kinds and extensions this facilitator supports |
@@ -530,9 +532,17 @@ curl -s http://localhost:3002/price/XLM/USDC | jq '{price, ammPrice, lastUpdated
 | `ORACLE_PAYMENT_ADDRESS` | Stellar address for x402 API payments | - | No* |
 | `ORACLE_PAYMENT_ADDRESS_TESTNET` / `ORACLE_PAYMENT_ADDRESS_MAINNET` | Per-network override for the address above | - | No |
 | `X402_FACILITATOR_URL` | x402 facilitator service URL | - | No |
+| `DATABASE_POOL_SIZE` | Maximum PostgreSQL connections (shared across ingesters and API) | `20` | No |
+| `DATABASE_POOL_TIMEOUT_MS` | Connection acquisition timeout in ms | `10000` | No |
 
 *\*Required if enabling x402 payment gating.*
 
+## Operations & Runbook
+
+For production architecture, failure mode diagnostics, degraded operational states, and deployment procedures, see [`docs/RUNBOOK.md`](docs/RUNBOOK.md).
+
+> **Note on Connection Pooling:** The Fastify API server and background venue ingesters (SDEX, AMM, Soroswap, Aquarius) share a single database connection pool (`DATABASE_POOL_SIZE`). At 20 connections per instance, 3 running replicas require at most 60 connections, safely under the default PostgreSQL `max_connections` ceiling of 100.
+
 ## Stack
 - **Runtime:** Node.js 20 + TypeScript
 - **API:** Fastify + Mercurius (GraphQL)

docs/RUNBOOK.md

@@ -0,0 +1,107 @@
+# Lens Service Runbook
+
+This runbook documents the operational architecture, dependencies, degraded operating modes, diagnostics, and deployment procedures for the Lens service.
+
+---
+
+## 1. Architecture & Background Processes
+
+A single Lens process runs both user-facing API handlers and background data pipelines:
+
+1. **Fastify REST & GraphQL API Server**
+   - Serves price queries, OHLCV candles, orderbook depth, aggregates, screener, and x402 facilitator routes.
+   - Listens on `PORT` (default 3002) and binds to `HOST` (default `0.0.0.0`).
+   - Serves health and telemetry probes: `/live`, `/ready`, `/status`, and Prometheus metrics on `/metrics`.
+
+2. **WebSocket Price Fan-Out**
+   - Broadcasts real-time price updates for watched asset pairs to connected WebSocket clients at `/ws`.
+
+3. **Venue Ingestion Loops (Per Enabled Network)**
+   - Independently fault-isolated worker loops per network (`testnet`, `mainnet`):
+     - **SDEX Ingester:** Ingests Classic Stellar orderbook trades.
+     - **AMM Ingester:** Indexes Stellar liquidity pool reserves from Horizon.
+     - **Soroswap Ingester:** Polls Soroswap pool reserves via Soroban RPC.
+     - **Aquarius Ingester:** Queries Aquarius AMM pool states.
+     - **Snapshot Ingester:** Periodically records 1-minute price snapshots.
+   - Ingesters use exponential backoff (1s up to 60s ceiling) on repeated crashes, resetting after 60s of sustained healthy execution.
+
+4. **Background Jobs (BullMQ + Redis)**
+   - **Aggregate Refresh Worker:** Computes and refreshes rolling VWAP / OHLCV aggregates.
+   - **Snapshot Retention Worker:** Enforces 30-day retention on `price_snapshots`. Degrades to an advisory-locked in-process hourly timer if Redis is unavailable.
+
+---
+
+## 2. Shared Database Connection Pool
+
+The Fastify API handlers, background venue ingesters, and retention jobs **share a single PostgreSQL connection pool**.
+
+### Pool Sizing Arithmetic
+- Standard Postgres `max_connections` limit: **100**.
+- Lens pool size per instance: **20 connections** (`DATABASE_POOL_SIZE`, default 20).
+- Acquisition timeout: **10,000ms** (`DATABASE_POOL_TIMEOUT_MS`, default 10s).
+- Across 3 deployed instances: `3 × 20 = 60 connections`, leaving 40 connections headroom for administrative tasks, migrations, and external monitoring.
+
+---
+
+## 3. External Dependencies & Degraded Modes
+
+| Dependency | Purpose | Failure Symptom | Degraded Behavior |
+|---|---|---|---|
+| **PostgreSQL** | Primary source of truth for trades, snapshots, and indexer state | `/ready` returns 503; queries fail with `DatabasePoolExhaustedError` or connection errors | API cannot serve fresh data; writes fail immediately. |
+| **Redis** | Response cache and BullMQ job queue | `redis_connected` metric = 0; log warning on connection | Price reads fall back to PostgreSQL; retention degrades to in-process hourly timer with Postgres advisory locking (`retentionMode: in_process`). Restores automatically to queue mode upon reconnection. |
+| **Horizon (Stellar)** | SDEX trade feeds and AMM pool reserve queries | `horizon_rate_limited_total` increments; 429 logs with `Retry-After` | Ingester respects `Retry-After` header or performs exponential backoff; venue is excluded from aggregate instead of reporting zero pools. |
+| **Soroban RPC** | Soroswap reserves & Reflector oracle quotes | RPC simulation errors; `VenueFetchError` logged | Excluded from aggregate; Reflector oracle returns null and logs error; last error recorded in `/status`. |
+| **Aquarius API** | Aquarius AMM liquidity pools | Non-2xx response; `VenueFetchError` logged | Excluded from aggregate; does not dilute or zero-weight overall market price. |
+
+---
+
+## 4. Diagnostics: /status, Probes, & /metrics
+
+### Health Probes
+- `GET /live`: Liveness probe. Returns HTTP 200 `{"status": "live"}` if the event loop is turning. Does NOT query Postgres or Redis, preventing restart loops during dependency blips.
+- `GET /ready`: Readiness probe. Returns HTTP 200 `{"status": "ready", "checks": {"database": true, "redis": true}}` only after boot completes and dependencies are reachable; returns HTTP 503 otherwise.
+- Both `/live` and `/ready` are public, unauthenticated, exempt from rate limiting, and excluded from `httpMetrics` route cardinality.
+
+### `/status` Monitoring Endpoint
+Returns operational telemetry:
+- `ok`: Service status boolean.
+- `network`: Current network (`testnet` or `mainnet`).
+- `watchedPairs`: List of watched trading pairs.
+- `lastIndexedLedger`: Latest ingested ledger sequence.
+- `lastProcessedAt`: Timestamp of last indexed trade.
+- `ingestLagSeconds`: Seconds elapsed since last trade was ingested.
+- `retentionMode`: Active snapshot retention mode (`queue` or `in_process`).
+- `ingesterCrashes`: History of recent crashes per ingester, restart counts, and current backoff.
+- `failedVenues`: Map of venues experiencing upstream fetch errors.
+- `failedWebhooks`: Recent webhook deliveries that exhausted all retries (payload body preserved, secret excluded).
+
+### Key Prometheus Metrics (`/metrics`)
+- `trades_ingested_total{pair, network}`: Counter of trades ingested.
+- `last_trade_timestamp{pair, network}`: Unix timestamp of latest trade.
+- `horizon_rate_limit_remaining{network}`: Gauge of remaining Horizon rate limit quota.
+- `horizon_rate_limited_total{network}`: Counter of 429 rate limit responses from Horizon.
+- `redis_connected`: Gauge (1 = connected, 0 = disconnected).
+- `retention_path_mode{mode}`: Gauge indicating active retention mode (`queue` vs `in_process`).
+- `pg_pool_total_connections`, `pg_pool_idle_connections`, `pg_pool_waiting_count`: Pool utilisation gauges.
+- `pg_pool_exhausted_total`: Counter of queries that timed out waiting for an available DB connection.
+- `ingester_restarts_total{ingester, network}`: Ingester crash restart counter.
+- `webhook_delivery_failures_total{category}`: Counter of permanently failed webhook deliveries by category (`upstream_5xx`, `timeout`, `network`, `unknown`).
+- `unhandled_rejections_total`, `uncaught_exceptions_total`: Counters of fatal process error events.
+
+---
+
+## 5. Deployment & Rollback Procedures
+
+### Standard Deployment
+1. Build the application: `npm run build`
+2. Run database migrations: `npm run prisma:push`
+3. Launch application: `npm start` (or container start)
+4. Verify `/ready` returns HTTP 200 before routing live traffic.
+
+### Rollback Procedure
+1. If deployment fails or exhibits regressions, redeploy the previous stable container image or git commit.
+2. Lens ingesters are idempotent and resume automatically from `indexer_state.last_ledger`.
+3. In-flight WebSocket connections and HTTP requests terminate during restart and reconnect automatically.
+
+### Known Non-Blocking Test Failures
+- **`Load test (prices)`** is failing on `main` and is a known, pre-existing issue unrelated to application reliability or this release.

openapi.json

@@ -156,6 +156,21 @@
                     "ingestLagSeconds": {
                       "type": "integer",
                       "nullable": true
+                    },
+                    "retentionMode": {
+                      "type": "string"
+                    },
+                    "ingesterCrashes": {
+                      "type": "object"
+                    },
+                    "failedVenues": {
+                      "type": "object"
+                    },
+                    "failedWebhooks": {
+                      "type": "array",
+                      "items": {
+                        "type": "object"
+                      }
                     }
                   }
                 }
@@ -168,6 +183,94 @@
         }
       }
     },
+    "/live": {
+      "get": {
+        "operationId": "getLive",
+        "summary": "Liveness check probe",
+        "security": [],
+        "responses": {
+          "200": {
+            "description": "Process event loop is healthy and alive",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "properties": {
+                    "status": {
+                      "type": "string",
+                      "example": "live"
+                    }
+                  }
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/ready": {
+      "get": {
+        "operationId": "getReady",
+        "summary": "Readiness check probe",
+        "security": [],
+        "responses": {
+          "200": {
+            "description": "Process has finished booting and dependencies are ready",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "properties": {
+                    "status": {
+                      "type": "string",
+                      "example": "ready"
+                    },
+                    "checks": {
+                      "type": "object",
+                      "properties": {
+                        "database": {
+                          "type": "boolean"
+                        },
+                        "redis": {
+                          "type": "boolean"
+                        }
+                      }
+                    }
+                  }
+                }
+              }
+            }
+          },
+          "503": {
+            "description": "Process is starting up or dependencies are unreachable",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "properties": {
+                    "status": {
+                      "type": "string",
+                      "example": "unhealthy"
+                    },
+                    "checks": {
+                      "type": "object",
+                      "properties": {
+                        "database": {
+                          "type": "boolean"
+                        },
+                        "redis": {
+                          "type": "boolean"
+                        }
+                      }
+                    }
+                  }
+                }
+              }
+            }
+          }
+        }
+      }
+    },
     "/price/{assetA}/{assetB}": {
       "get": {
         "operationId": "getPrice",

openapi.yaml

@@ -114,9 +114,77 @@ paths:
                   ingestLagSeconds:
                     type: integer
                     nullable: true
+                  retentionMode:
+                    type: string
+                  ingesterCrashes:
+                    type: object
+                  failedVenues:
+                    type: object
+                  failedWebhooks:
+                    type: array
+                    items:
+                      type: object
         '429':
           $ref: '#/components/responses/TooManyRequests'
 
+  /live:
+    get:
+      operationId: getLive
+      summary: Liveness check probe
+      security: []
+      responses:
+        '200':
+          description: Process event loop is healthy and alive
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  status:
+                    type: string
+                    example: live
+
+  /ready:
+    get:
+      operationId: getReady
+      summary: Readiness check probe
+      security: []
+      responses:
+        '200':
+          description: Process has finished booting and dependencies are ready
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  status:
+                    type: string
+                    example: ready
+                  checks:
+                    type: object
+                    properties:
+                      database:
+                        type: boolean
+                      redis:
+                        type: boolean
+        '503':
+          description: Process is starting up or dependencies are unreachable
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  status:
+                    type: string
+                    example: unhealthy
+                  checks:
+                    type: object
+                    properties:
+                      database:
+                        type: boolean
+                      redis:
+                        type: boolean
+
   /price/{assetA}/{assetB}:
     get:
       operationId: getPrice

src/__tests__/aquariusIngester.test.ts

@@ -66,11 +66,10 @@ describe('fetchAquariusPools', () => {
     )
   })
 
-  it('returns an empty array on a non-ok response', async () => {
-    ;(global.fetch as ReturnType<typeof vi.fn>).mockResolvedValue({ ok: false })
+  it('throws on a non-ok response (#329)', async () => {
+    ;(global.fetch as ReturnType<typeof vi.fn>).mockResolvedValue({ ok: false, status: 500 })
 
-    const result = await fetchAquariusPools(mockPair as any)
-    expect(result).toEqual([])
+    await expect(fetchAquariusPools(mockPair as any)).rejects.toThrow()
   })
 })