feat: core engine data quality, CEX adapters, TVL, routing and README sync (#295, #275, #260, #249, #248, #243, #238, #236, #235, #233, #231, #229, #226, #224, #223, #222, #220, #217, #216, #215)

.env.example

@@ -225,3 +225,20 @@ MCP_PAID_CALL_ALLOWED_HOSTS=
 # against a table that has never been pruned cannot hold a long lock or blow
 # the WAL. Default 1000.
 SNAPSHOT_PRUNE_BATCH_SIZE=1000
+
+# ─── Data Quality & Plausibility ───────────────────────────────────────────
+# Maximum age in seconds before price observation is flagged as stale. Default 300.
+PRICE_STALE_THRESHOLD_SECONDS=300
+
+# Maximum allowed percentage deviation over rolling median before an incoming tick is quarantined. Default 40.
+INGEST_OUTLIER_BOUND_PCT=40
+
+# Comma-separated list of enabled centralized exchange adapters (binance, coinbase, kraken, mock). Default binance,coinbase,kraken.
+ENABLED_CEX_VENUES=binance,coinbase,kraken
+
+# Days to retain raw price_points rows before rolling them up into hourly aggregates and pruning. Default 30.
+PRICE_POINTS_RETENTION_DAYS=30
+
+# Maximum raw price_points rows deleted per batch during retention pruning. Default 1000.
+PRICE_POINTS_PRUNE_BATCH_SIZE=1000
+

README.md

@@ -9,37 +9,59 @@ Aggregates price data from Stellar's Classic Order Book (SDEX) and AMM Liquidity
 
 **What Horizon doesn't do:** Horizon's `/trade_aggregations` only covers SDEX. AMM pool prices (derived from reserve ratios) are not aggregated anywhere. Lens fills this gap.
 
+## Supported Venues
+
+<!-- VENUES:START -->
+| Venue | Type | Description |
+|---|---|---|
+| **SDEX** | DEX | Stellar's native order book matching engine |
+| **Stellar AMM** | AMM | Protocol-level constant-product liquidity pools |
+| **Soroswap** | AMM / DEX | Uniswap v2-style automated market maker on Soroban |
+| **Aquarius** | AMM / Liquidity | Liquidity protocol and automated market maker |
+| **Phoenix** | AMM / DEX | Next-generation Soroban decentralized exchange |
+| **Binance** | CEX | Centralized exchange public ticker and order book depth |
+| **Coinbase** | CEX | Centralized exchange public ticker and order book depth |
+| **Kraken** | CEX | Centralized exchange public ticker and order book depth |
+<!-- VENUES:END -->
+
 ## Endpoints
 
 ### REST
+
+<!-- ENDPOINTS:START -->
 | Method | Path | Description |
 |---|---|---|
-| GET | `/price/:assetA/:assetB` | Current VWAP, 24h volume, best route |
-| GET | `/price/:assetA/:assetB/route?amount=1000` | Best execution route for a given amount |
-| GET | `/price/:assetA/:assetB/depth?amount=1000` | Simulated order-book depth and execution slippage |
-| GET | `/book/:assetA/:assetB?depth=0.1&step=0.01&network=testnet` | Aggregated SDEX book and synthetic AMM price ladder with cumulative size and venue provenance |
-| GET | `/price/:assetA/:assetB/history?window=1h` | OHLCV history (`1m`, `5m`, `1h`, `24h`) |
-| GET | `/price/twap/:assetA/:assetB?window=60&sampleInterval=60&method=iqr` | Manipulation-resistant time-weighted average price |
-| GET | `/price/vwap/:assetA/:assetB?window=60&source=SDEX&method=iqr` | Volume-weighted average price |
-| GET | `/candles/:assetA/:assetB?interval=1h&from=…&to=…` | OHLCV candles (`1m`, `5m`, `15m`, `1h`, `4h`, `1d`) |
-| GET | `/prices/history?pair=XLM/USDC&from=…&to=…&interval=1m` | Historical 1-minute price snapshots, optionally aggregated (`1m`, `5m`, `1h`); honours `?network=`; `?format=csv` serves the points as `text/csv` (`ts,price,volume`); ~30-day retention |
-| GET | `/pools` | Active AMM pools being watched |
-| GET | `/pairs` | Watched trading pairs |
-| GET | `/volumes/:asset?window=24h` | Cross-venue traded volume (`24h`, `7d`, `30d`) with a per-venue breakdown |
-| GET | `/spreads/:asset?window=5m` | Per-venue bid/ask spread, tightest first (`5m`, `15m`, `1h`, `24h`) |
-| GET | `/compare/:asset` | Lens vs configured oracle prices, with per-oracle deviation and availability |
-| GET | `/screener?sortBy=volume&order=desc&limit=20` | Screen pairs by volume, 24h change, liquidity and price |
-| GET | `/benchmark/:asset?target=USD` | Peg-deviation statistics versus a target |
-| GET | `/basket?asset=XLM&asset=USDC&weight=0.6&weight=0.4` | Weighted price of a basket of assets |
-| GET | `/status` | Indexer health |
-| GET | `/discovery/resources?type=&payTo=&network=&extensions=&limit=&offset=` | Bazaar catalog of x402-discoverable resources (spec: [`bazaar`](https://github.com/x402-foundation/x402/blob/main/specs/extensions/bazaar.md)) |
-| GET | `/usage/me` | Usage and quota for the calling API key (requires `Authorization: Bearer <key>`) |
-| GET | `/supported` | Payment kinds and extensions this facilitator supports |
-| POST | `/verify` | Verify an x402 payment payload |
-| POST | `/settle` | Settle an x402 payment (idempotent — retries replay the stored result) |
+| GET | `/venues/registry` | List registered venues and their network liveness |
+| GET | `/status` | Get API status and indexer state |
+| GET | `/price/:assetA/:assetB` | Get aggregated price for a pair |
+| GET | `/price/:assetA/:assetB/route` | Get best swap route for a pair and amount |
+| GET | `/price/:assetA/:assetB/history` | Get price history buckets |
+| GET | `/pools` | Get AMM liquidity pool reserves and spot prices |
+| GET | `/candles/:assetA/:assetB` | Get OHLCV candle data |
+| GET | `/prices/history` | Get historical 1-minute price snapshots |
+| GET/POST | `/pairs` | List all active trading pairs |
+| GET | `/price/:assetA/:assetB/depth` | Get order-book depth and execution slippage for a pair |
+| GET | `/book/:assetA/:assetB` | Get the aggregated SDEX and AMM price ladder for a pair |
+| GET | `/price/twap/:assetA/:assetB` | Get a manipulation-resistant time-weighted average price |
+| GET | `/price/vwap/:assetA/:assetB` | Get a volume-weighted average price |
+| GET | `/volumes/:asset` | Get cross-venue traded volume for an asset |
+| GET | `/spreads/:asset` | Get the tightest per-venue bid/ask spread for an asset |
+| GET | `/compare/:asset` | Compare Lens prices with every configured external oracle |
+| GET | `/premium/:assetA/:assetB` | Get CEX vs on-chain price premium and spread in basis points |
+| GET | `/venues` | Get cross-venue traded volume, market share percentages and period change |
+| GET | `/tvl` | Get venue-level Total Value Locked with unpriceable pool reporting |
+| GET | `/screener` | Screen pairs by volume, change, liquidity and price |
+| GET | `/benchmark/:asset` | Get peg-deviation statistics for an asset against a target |
+| GET | `/basket` | Compute the weighted price of a basket of assets |
+| GET | `/discovery/resources` | Browse the x402 Bazaar catalog of discoverable resources |
 | POST | `/webhooks` | Subscribe to a price-threshold webhook |
 | DELETE | `/webhooks/:id` | Delete a webhook subscription |
-| GET/POST | `/graphql` | GraphQL query/mutation endpoint (GraphiQL IDE at `/graphiql`, docs: [`docs/graphql.md`](docs/graphql.md)) |
+| GET | `/usage/me` | Get usage and quota for the calling API key |
+| GET | `/supported` | List the payment kinds and extensions this facilitator supports |
+| POST | `/verify` | Verify an x402 payment payload |
+| POST | `/settle` | Settle an x402 payment (idempotent) |
+| GET/POST | `/graphql` | Execute a GraphQL query over HTTP GET |
+<!-- ENDPOINTS:END -->
 
 Every non-internal route above has a matching entry in [`openapi.yaml`](openapi.yaml);
 `tests/openapi.test.ts` boots the server and fails if a registered route is

docs/data-quality.md

@@ -0,0 +1,42 @@
+# Data Quality & Service Level Guarantees
+
+This document defines the precision, freshness boundaries, completeness SLA, and outlier filtering guarantees provided by Lens.
+
+---
+
+## 1. Freshness Guarantees
+* **Standard Staleness Threshold:** `300 seconds` (5 minutes)
+  - Enforced in: [`src/aggregator/vwap.ts`](file:///d:/PersonalAI/lens/src/aggregator/vwap.ts) via `PRICE_STALE_THRESHOLD_SECONDS`.
+  - Behavior when exceeded: Responses explicitly flag `"stale": true` and confidence drops to `"low"`.
+* **Indexer Lag Alarm:** `300 seconds`
+  - Enforced in: [`src/api/rest.ts`](file:///d:/PersonalAI/lens/src/api/rest.ts) `/status` health probe.
+  - Behavior when exceeded: `/status` reports `"status": "degraded"` with `"health": { "data": "stale" }`.
+
+---
+
+## 2. Ingest-Time Outlier & Plausibility Check
+* **Deviation Bound:** `40%`
+  - Enforced in: [`src/db.ts`](file:///d:/PersonalAI/lens/src/db.ts) via `INGEST_OUTLIER_BOUND_PCT`.
+  - Behavior when exceeded: Aberrant ticks deviating > 40% from the recent rolling median are rejected from the primary `price_points` table and stored in the quarantine audit store, incrementing `price_points_ingested_total{status="rejected"}`.
+
+---
+
+## 3. Precision & Decimal Handling
+* **Prices:** Exact 18 decimal places (`Decimal(36, 18)`).
+* **Volumes:** Exact 7 decimal places (`Decimal(36, 7)`).
+* Enforced in: [`prisma/schema.prisma`](file:///d:/PersonalAI/lens/prisma/schema.prisma).
+* Lossless String Formatting: Financial clients can pass header `X-Format-Decimals: string` to receive exact string representations without double-precision truncation.
+
+---
+
+## 4. Retention & Pruning
+* **Snapshot Retention Window:** `30 days` (`SNAPSHOT_RETENTION_DAYS` in [`src/jobs/snapshotRetention.ts`](file:///d:/PersonalAI/lens/src/jobs/snapshotRetention.ts)).
+* **Price Points Retention Window:** `30 days` (`PRICE_POINTS_RETENTION_DAYS` in [`src/jobs/snapshotRetention.ts`](file:///d:/PersonalAI/lens/src/jobs/snapshotRetention.ts)).
+* Rollup Requirement: Raw price points older than 30 days are rolled up into 1-hour `price_aggregates` buckets before batch deletion.
+
+---
+
+## 5. Rate Limiting Bounds
+* **Public Unauthenticated IP Limit:** `100 requests / minute` (`RATE_LIMITS.ipDefault` in [`src/app.ts`](file:///d:/PersonalAI/lens/src/app.ts)).
+* **Authenticated API Key Limit:** `1000 requests / minute` (`RATE_LIMITS.authenticatedDefault` in [`src/app.ts`](file:///d:/PersonalAI/lens/src/app.ts)).
+* **Monitoring Endpoint (`/status`) Limit:** `1000 requests / minute` (`RATE_LIMITS.monitoring` in [`src/app.ts`](file:///d:/PersonalAI/lens/src/app.ts)).

docs/deploy.md

@@ -0,0 +1,33 @@
+# Deployment and Database Migration Guide
+
+This document describes the deployment procedure for the Lens database and application services.
+
+---
+
+## Database Migrations: `prisma migrate deploy`
+
+Lens uses **Prisma Migrate** for all schema updates.
+
+### Production Migration Command
+```bash
+npx prisma migrate deploy
+```
+
+### Why `prisma db push` is Strictly Prohibited in Production
+1. **Unpredictable Schema Diffs:** `db push` inspects the live database connection and immediately alters tables without reviewable SQL statements.
+2. **Data Loss Risk:** `db push` can silently drop columns, indexes, or composite partitions when models are modified or renamed.
+3. **No Migration History:** `db push` leaves no audit trail or rollback record in `_prisma_migrations`, making rollbacks impossible.
+4. **Reproducibility:** Every deployment stage (local, test, staging, production) must execute the exact byte-for-byte migration sequence recorded under `prisma/migrations/`.
+
+---
+
+## Zero-Downtime Rollout Steps
+1. **Pre-Deployment:**
+   - Execute tests: `npm test`
+   - Verify migration drift: `npx prisma migrate diff --from-schema-datamodel prisma/schema.prisma --to-migrations prisma/migrations --exit-code`
+2. **Database Migration:**
+   - Run `npx prisma migrate deploy`
+3. **Application Deploy:**
+   - Start or restart worker and API containers using `npm start`
+4. **Post-Deployment Verification:**
+   - Verify health probe: `GET /status` returns HTTP 200 `{ "ok": true, "status": "healthy" }`