docs(ops): add operations runbook — processes, dependencies, health checks, failure modes, rollback

[felixumeh]

Closes #350

Summary

Adds docs/RUNBOOK.md — the operational runbook issue #350 asks for — and links it from the README's Documentation section. Every claim in the document names the file or symbol it is checkable against, and where the system does the wrong thing (no shutdown handler, no 429 handling, aggregate refresh has no fallback) the runbook says so instead of describing an aspirational system.

Why

The operational knowledge for Lens — that Redis loss silently freezes price_aggregates while /price keeps serving, that startup runs prisma db push --accept-data-loss before the API listens, that /status's ok: true is a DB-liveness signal and not a freshness signal — currently lives only in the code and in whoever last read it. An operator at 03:00 needs one document that maps symptom → endpoint or metric → cause.

What the runbook covers (mapped to the issue)

Issue #350 requirement	Where it lands
The processes that run and what each one does	§1 — the single npm start process (API, WebSocket fan-out, venue + snapshot ingesters, BullMQ workers, webhook dispatcher, boot-time auto-migration, health probes), its exact startup order with hard vs soft failure points (ending in markBootComplete()), compose services, optional example processes, and all five GitHub Actions workflows
Every external dependency and what degrades when unavailable	§2 — PostgreSQL, Redis, Horizon, Soroban RPC/Soroswap/Aquarius, the x402 facilitator, outbound webhook targets, the Render keep-alive monitor, and env-var misconfiguration — each with down-at-boot vs dies-at-runtime behaviour plus the concrete symptom (including the new /ready signals: Postgres reachability and migration state, Redis PING)
How to tell whether ingestion is healthy, using /status and /metrics	§3 — field-by-field /status reading (why ok: true is not freshness, why lag can grow on an idle market), the /live and /ready probes with their semantics, /snapshot as a one-shot freshness view, the ingestion/API metric table with PromQL recipes (including the new auth/rate-limit/shed counters), known blind spots, and a 5-step triage sequence
The known failure modes in this batch and the symptom each presents	§4 — nine failure modes (#338, #339, #341, #342, #344, the unfixed half of #345, #347, #348, #351), each stating what the code does today and naming the endpoint, metric, or log line that shows it
How to deploy, how to roll back, and what a deploy interrupts	§5 — deploy paths (Render + compose), the six things a restart interrupts (including the destructive --accept-data-loss auto-migration and holding the instance out of rotation via /ready), and a 5-step rollback: code → schema → data → config → verification
Load test (prices) failing on main is a known, unrelated red	§6 — stated explicitly, with the local CI gate to run instead

Acceptance criteria

• docs/RUNBOOK.md exists and is linked from the README
• Every external dependency is listed with its degraded behaviour
• Each documented symptom names the endpoint or metric that shows it
• Rollback is documented
• Nothing in it is aspirational — every claim is checkable against the code today (file/symbol cited inline)

Verification

The full CI sequence from .github/workflows/ci.yml was run locally after rebasing onto the current main, with exit statuses captured (pipefail):

• npm ci — ✅
• node node_modules/prisma/build/index.js generate — ✅ (required: main added expiresAt to the Prisma schema)
• npx tsc --noEmit — ✅ 0 errors
• npm run build — ✅
• npm test --if-present — ✅ 84 files, 908 passed, 1 skipped (tests/integration/e2e.test.ts, skipped in CI as well), exit 0

The branch was rebased onto main after the first draft, which had landed the health-probe, auth-metric, and admin-token-refactor PRs in the meantime. Every claim was re-verified against the new code and the runbook updated accordingly: /live + /ready + /snapshot and the four new counters are now covered, the admin-auth citation moved to src/api/auth.ts, and readiness became a symptom signal in §2.

Claims that were least obvious were verified against the code, not assumed:

• No SIGTERM/SIGINT/unhandledRejection/uncaughtException handler exists anywhere in src/ (re-checked after the rebase) → §4 rows 1–2
• Aggregate refresh has no in-process fallback while retention and partition maintenance do → §4 row 9
• /status returns ok: true unconditionally, and only the SDEX/AMM ingesters write indexer_state → §3 caveat
• GitHub Actions: Load test (prices) has failed on every daily run for the last 8+ days on main, while ci.yml is green — confirming the issue's note that it is an unrelated red

CI on this PR: the pull_request run was created (run 39290710090) but reports action_required — this repository requires maintainer approval before workflows run for fork PRs, and I cannot approve my own run. The identical job (Typecheck & build) passed locally as listed above; a maintainer just needs to approve the run.

Notes

• Docs-only diff: docs/RUNBOOK.md (new) + 2 lines in README.md. No source, schema, or lockfile changes (a stray local package-lock.json modification was deliberately excluded to keep the PR focused), and no changeset since nothing user-facing changed.
• The runbook documents today's behaviour including the ugly parts (auto-migration that accepts data loss, abrupt shutdowns, dropped webhooks after 3 attempts). Fixing those belongs to their own issues (An unhandled rejection takes the process down with no record #344, Websocket clients are dropped without a close frame on shutdown #347, Horizon rate limiting is not handled #348, …) — this PR only makes them findable.
• Two pre-existing open PRs touch adjacent ground and should not conflict textually: feat(ingest): bound every ingester cycle with a deadline, abandoned-cycle metric, and overlap-proof abandonment (#351) #471 (ingester cycle deadline, §4 row 4) — if it merges, row 4 can be retired.

[drips-wave]

@felixumeh Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits