# ISP-082: Distributed rate-limiter counting divergence — MemoryStore vs Redis across Render replicas lets surge traffic through one door #752

• Type: Integration + Spike
• Severity: P0/Critical
• Difficulty: Extreme
• Estimate: 24h
• Touch surface: 13 files

1. Background & Context

HelPhone protects dispatch creation, wallet-nonce issuance, and GraphQL mutations with express-rate-limit@8.6.2 configured in server/middleware/rateLimiter.ts. The default store is MemoryStore, an in-process Map<string, Client> with windowMs, localKeys, and interval expiry that works perfectly on a single Node process. Production on Render, however, runs multiple replicas behind a round-robin load balancer as declared in render.yaml (multiple instances for HA), each with its own MemoryStore. There is no shared counter, so an attacker or a flash-crowd surge gets N times the configured limit where N is the replica count.

The repo already vendors ioredis@5.4.1 for caching and pub/sub, and server/db/poolManager.ts manages Postgres pooling, but rate limiting was never wired to Redis. server/env.ts and .env.example expose REDIS_URL optionally, yet server/middleware/rateLimiter.ts ignores it and always constructs new MemoryStore(). server/index.ts and server/index.js both mount the limiter before server/middleware/auth.ts and before server/graphql/server.ts, so GraphQL mutations inherit the same per-replica hole. server/middleware/metrics.ts counts hits per process, which masks the divergence because dashboards sum per-replica counters without showing that any single IP exceeded the global budget.

Disaster traffic is the worst case. During an active emergency, legitimate responder polling (src/lib/wsClient.ts reconnect loops, src/lib/syncManager.ts retries) plus malicious replay can hit /api/dispatch and /graphql at 10-50x baseline. With three Render replicas and a 100 req/min limit, the effective limit is 300 req/min per IP, enough to exhaust Soroban RPC quotas via src/lib/rpcHealth.ts and Postgres connections via server/db/poolManager.ts. server/middleware/logger.ts logs per-process 429s, so operators see sporadic 429s on one replica while the others sail through, misdiagnosing it as client flakiness rather than a distributed-systems bug.

The spike must also reckon with express-rate-limit@8 store contract subtleties: increment, decrement, resetKey, get, plus init, shutdown, and validations that differ between MemoryStore and rate-limit-redis. Key generation via ipKeyGenerator must handle Render's X-Forwarded-For plus trust proxy settings, IPv6 subnets, and authenticated-user keys for wallet-bound limits. Fail-open vs fail-closed on Redis outage is a life-safety decision: fail open lets abuse through during an outage; fail closed blocks emergency calls when Redis is down.

2. Current Broken Behaviour

With 3 local processes simulating Render replicas (PORT=3001/3002/3003 node --experimental-strip-types server/index.ts), sending 300 requests from one IP with a 100 req/min limit yields zero 429s when load-balanced round-robin (100 per replica, each under its local window), but 200 429s when pinned to one replica. In production, surge tests show effective limit scaling linearly with replica count. server/middleware/metrics.ts reports per-replica rate_limited_total that never fires globally. Redis is running but idle (MONITOR shows no INCR from rate limiting). During Redis failover drills, the limiter either crashes (unhandled rejection) or silently passes all traffic depending on which replica loses connection first.

3. Desired End State

A shared, atomic, globally-correct rate limiter backed by Redis (or a proven alternative) that enforces the configured window regardless of replica count, with deterministic key generation behind Render's proxy, graceful failover semantics explicitly chosen and tested (fail-closed for auth/mutation routes, fail-open with alerting for read routes, or global fail-closed with circuit-breaker), per-route limits preserved (/graphql mutations stricter than /api/docs), and observability showing global vs local counts. MemoryStore remains only for single-process dev/test. All behavior covered by multi-process integration tests.

4. Files / Systems Involved

• server/middleware/rateLimiter.ts
• server/index.ts
• server/index.js
• server/middleware/metrics.ts
• server/middleware/logger.ts
• server/middleware/auth.ts
• server/graphql/server.ts
• server/env.ts
• server/db/poolManager.ts
• server/telemetry.js
• render.yaml
• .env.example
• server/package.json

5. Reproduction Steps

1. Run npm ci && npm --prefix server ci to install express-rate-limit@8 and ioredis@5.
2. Start Redis locally: docker run --rm -p 6379:6379 redis:7-alpine and verify with redis-cli ping.
3. Start three replicas: PORT=3001 node --experimental-strip-types server/index.ts & PORT=3002 node --experimental-strip-types server/index.ts & PORT=3003 node --experimental-strip-types server/index.ts &.
4. Run surge probe round-robin: for i in $(seq 1 300); do p=$((3001 + i % 3)); curl -s -o /dev/null -w "%{http_code}\n" http://localhost:$p/api/dispatch; done | sort | uniq -c — observe zero 429s despite 300 > 100 limit.
5. Repeat pinned to one replica: for i in $(seq 1 300); do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3001/api/dispatch; done | sort | uniq -c — observe ~200 429s.
6. Run redis-cli MONITOR during both runs to confirm zero Redis interaction from limiter.
7. Kill Redis mid-test (docker stop) and observe per-replica crash vs pass-through divergence.
8. Run node server/tests/ws-cluster.js to see WS reconnect amplification under divergent limits.

6. Spike Investigation Plan

Experiment A (4h): Instrument rateLimiter.ts to log key, windowMs, hits, replicaId per increment. Run round-robin vs pinned load with autocannon -c 50 -d 20 http://localhost:3001/api/dispatch. Quantify divergence factor vs replica count. Decide key shape (ipKeyGenerator + user + route).

Experiment B (6h): Prototype rate-limit-redis with ioredis Sentinel/Cluster client from server/env.ts REDIS_URL. Test atomic INCR + PEXPIRE Lua, sliding vs fixed windows, resetKey on auth success, and shutdown drain. Measure added p99 latency (target <5ms) and Redis ops/sec under 10k req burst.

Experiment C (4h): Failure-mode matrix. Kill Redis, partition network (iptables -j DROP), expire Redis memory (CONFIG SET maxmemory 1mb). Compare fail-open vs fail-closed vs stale-cache-serve. Wire decision into server/middleware/errorHandler.ts + server/telemetry.js alerts. Validate Render REDIS_URL rotation without restart interplay (see ISP-090).

Experiment D (4h): Proxy + IPv6 correctness. Set trust proxy per Render docs in server/index.ts, test X-Forwarded-For spoofing, ipKeyGenerator(ip, ipv6Subnet) subnet masking, and authenticated-key fallback in server/middleware/auth.ts. Ensure scripts/detect-api-drift.js --check still passes for express-rate-limit surface.

Experiment E (6h): Soak + metrics. Update server/middleware/metrics.ts to export global-limit hits via OTel, run 3-replica soak with k6, confirm global 429 rate matches single-replica semantics and Postgres pool stays healthy via server/db/poolManager.ts stats.

7. Acceptance Criteria

• 300-req round-robin burst with 100 req/min limit yields ~200 global 429s regardless of replica count (within 5% tolerance)
• Redis-backed store uses atomic increments; no lost updates under 50-concurrency autocannon burst
• Failover behavior explicitly implemented and tested: documented fail-open vs fail-closed per route class with alert emitted to OTel
• trust proxy + ipKeyGenerator correctly keys IPv4, IPv6 /64, and Render forwarded IPs; spoofed X-Forwarded-For does not bypass
• Authenticated routes key on user/wallet in addition to IP via server/middleware/auth.ts integration
• Added p99 latency from Redis check is <8ms at 1k rps; Redis MONITOR shows bounded ops (no N+1)
• server/middleware/metrics.ts exposes ratelimit_global_hits and ratelimit_redis_errors counters
• server/index.ts and server/index.js share one limiter factory; no MemoryStore leak in production path
• Multi-process integration test added and passes in CI (3 processes + real Redis service)
• render.yaml documents REDIS_URL requirement and replica count assumption
• No unhandled Redis rejection crashes process; shutdown drains cleanly on SIGTERM
• scripts/detect-api-drift.js --check green for rate-limit surface

8. Why This Takes 24h / Why It Is Extremely Hard

• Distributed counting looks trivial until clock skew, sliding-window edge effects, and Redis PEXPIRE races produce off-by-N flakiness that only appears under concurrent multi-replica load, requiring custom harnesses not present in the repo.
• Fail-open vs fail-closed is a life-safety tradeoff with no safe default: blocking emergency dispatch during a Redis blip kills people, allowing abuse during the same blip kills the backend — the spike must quantify both and get product sign-off.
• Render networking (shared private network, ephemeral IPs, proxy headers) interacts with trust proxy and IPv6 subnetting; a wrong subnet mask either merges a whole carrier NAT into one bucket (mass false 429s) or splits one attacker into thousands of buckets (no limiting).
• express-rate-limit@8 store interface plus validations enforcement rejects naive custom stores at startup, forcing deep reading of its validation code and correct init/shutdown lifecycle inside both boot files.
• Observability is per-process today; proving global correctness requires cross-replica log correlation plus OTel metrics that don't yet exist, and load generators that defeat round-robin affinity.

9. Risks & Blast Radius

Switching to Redis makes Redis a hard dependency for every request: a Redis outage or REDIS_URL rotation mishap (ISP-090) can take down all replicas simultaneously instead of degrading gracefully. Wrong trust proxy config rate-limits the load-balancer IP and causes a total outage. Overly strict global limits block legitimate disaster-surge responder polling via src/lib/wsClient.ts retries, causing cascading reconnect storms. Underly strict limits let abuse exhaust server/db/poolManager.ts connections and Soroban RPC quotas, cascading into indexer lag (server/indexer/daemon.ts) and wallet failures. Metrics cardinality from per-IP logging can blow up OTel quotas (ISP-089).