AI API Gateway

A production-grade API gateway for LLM providers — unified interface, intelligent caching, rate limiting, cost tracking, and automatic failover.

Features · Architecture · Quick Start · Configuration · API Reference · Deployment

---

Why AI API Gateway?

Managing multiple LLM providers in production is painful. Vendor lock-in means switching from OpenAI to Anthropic requires rewriting your entire integration. Without caching, identical prompts hit the API every time and burn your budget. A single burst of traffic can exhaust your rate limit quota and break your application. You have no visibility into which models are being called, at what cost, or how fast they respond. And if OpenAI goes down, your application goes down with it.

AI API Gateway solves all of this with a single, drop-in proxy that sits between your application and any LLM provider. Your application talks to one endpoint. The gateway handles everything else.

---

Features

Feature	Description
Multi-Provider	OpenAI, Anthropic, Google Gemini — extensible to any provider
Two-Level Cache	L1 in-memory LRU + L2 Redis with configurable TTL
Rate Limiting	Sliding window and token bucket algorithms per API key
Automatic Failover	Provider failover with configurable retry strategies and backoff
Streaming	Full SSE streaming support across all providers
Type Safety	End-to-end TypeScript with Zod request validation
Observability	Structured JSON logging via Pino, health endpoint
Docker Ready	Single-command deployment with Docker Compose

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Client Applications                       │
└─────────────────────────┬───────────────────────────────────────┘
                          │ HTTP / SSE
                          ▼
┌─────────────────────────────────────────────────────────────────┐
│                      AI API Gateway (Fastify)                    │
│                                                                  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────┐   │
│  │   Auth   │  │  Rate    │  │  Cache   │  │   Provider   │   │
│  │Middleware│→ │ Limiter  │→ │  Layer   │→ │   Registry   │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────┬───────┘   │
│                                                     │            │
│  ┌──────────────────────────────────────────────────▼─────────┐ │
│  │                    Provider Adapters                        │ │
│  │  ┌─────────┐  ┌───────────┐  ┌──────────────────────────┐ │ │
│  │  │ OpenAI  │  │ Anthropic │  │  Google Gemini  │  ...   │ │ │
│  │  └─────────┘  └───────────┘  └──────────────────────────┘ │ │
│  └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
                    │                    │
             ┌──────▼──────┐     ┌──────▼──────┐
             │    Redis    │     │  L1 Memory  │
             │  (L2 Cache  │     │  LRU Cache  │
             │  + Limits)  │     │             │
             └─────────────┘     └─────────────┘

---

Quick Start

Prerequisites

• Node.js 22+
• Redis 7+
• Docker (optional)

Option 1: Docker Compose

git clone https://github.com/josenevado/ai-api-gateway
cd ai-api-gateway
cp .env.example .env
# Edit .env with your provider API keys
docker compose up -d

The gateway will be available at http://localhost:3000.

Option 2: Manual Setup

git clone https://github.com/josenevado/ai-api-gateway
cd ai-api-gateway
npm install
cp .env.example .env
# Edit .env
npm run dev

Your First Request

# Non-streaming completion
curl http://localhost:3000/v1/completions \
  -H "Authorization: Bearer your-gateway-key" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Explain Redis in one sentence."}],
    "model": "gpt-4o",
    "provider": "openai"
  }'

# With automatic failover across providers
curl http://localhost:3000/v1/completions \
  -H "Authorization: Bearer your-gateway-key" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "fallback": true,
    "fallbackOrder": ["openai", "anthropic", "google"]
  }'

# Streaming
curl http://localhost:3000/v1/completions \
  -H "Authorization: Bearer your-gateway-key" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Write a haiku."}],
    "stream": true
  }'

# With caching (only caches when temperature = 0)
curl http://localhost:3000/v1/completions \
  -H "Authorization: Bearer your-gateway-key" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "temperature": 0,
    "cache": true
  }'

---

Configuration

Environment Variables

# Server
PORT=3000
HOST=0.0.0.0
NODE_ENV=production
LOG_LEVEL=info

# Auth
GATEWAY_API_KEY=your-secure-key-minimum-32-chars
JWT_SECRET=your-jwt-secret-minimum-32-chars

# Redis
REDIS_URL=redis://localhost:6379

# Providers — add only the ones you use
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_AI_API_KEY=...

# Rate Limiting
RATE_LIMIT_MAX=100
RATE_LIMIT_WINDOW_MS=60000

# Caching
CACHE_TTL_SECONDS=3600
SEMANTIC_CACHE_ENABLED=true

# Timeouts & Retry
REQUEST_TIMEOUT_MS=30000
PROVIDER_TIMEOUT_MS=25000
MAX_RETRIES=3
RETRY_DELAY_MS=1000

---

API Reference

POST /v1/completions

Send a completion request to any configured provider.

Request Body

Field	Type	Default	Description
messages	Message[]	required	Conversation messages
model	string	gpt-4o	Model identifier
provider	openai | anthropic | google	openai	Target provider
temperature	number	0.7	Sampling temperature (0–2)
maxTokens	number	—	Maximum tokens to generate
stream	boolean	false	Enable SSE streaming
cache	boolean	false	Enable response caching
fallback	boolean	true	Enable automatic provider failover
fallbackOrder	ProviderName[]	["openai","anthropic","google"]	Failover priority

Response Headers

Header	Description
X-Request-Id	Unique request identifier
X-Cache	HIT or MISS
X-Cache-Key	Cache key used for this request
X-RateLimit-Limit	Maximum requests per window
X-RateLimit-Remaining	Remaining requests in current window
X-RateLimit-Reset	Unix timestamp when window resets

GET /v1/providers

Returns the list of currently available (configured) providers.

GET /health

Returns gateway health status including provider availability and cache connectivity.

---

Caching Behavior

The gateway uses a two-level cache architecture. L1 is an in-memory LRU cache with a 5-minute TTL and a maximum of 500 entries. L2 is Redis with a configurable TTL (default 1 hour). Cache reads check L1 first, falling back to L2 and promoting hits back to L1. Cache writes go to both levels simultaneously.

Responses are only cached when cache: true is set in the request AND temperature is exactly 0. This ensures only deterministic responses are cached, preventing stale or incorrect cached results for creative or probabilistic completions.

---

Rate Limiting

Two algorithms are available. The sliding window algorithm uses Redis sorted sets to track requests within a rolling time window, providing precise per-key limiting without the burst artifacts of fixed windows. The token bucket algorithm allows short bursts while enforcing an average rate, suitable for workloads with legitimate traffic spikes.

Rate limits are applied per API key. The response headers always reflect the current limit state so clients can implement backoff logic.

---

Deployment

Kubernetes

kubectl apply -f k8s/

Docker Compose (Production)

services:
  gateway:
    image: ai-api-gateway:latest
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - REDIS_URL=redis://redis:6379
    depends_on:
      - redis
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes
    restart: unless-stopped

volumes:
  redis_data:

Production Checklist

• Set NODE_ENV=production
• Use a strong GATEWAY_API_KEY (minimum 32 characters)
• Configure Redis with AOF persistence enabled
• Set up SSL/TLS termination at the load balancer
• Configure rate limits appropriate for your traffic patterns
• Set up log aggregation (Datadog, Grafana Loki, etc.)
• Monitor the /health endpoint with your uptime service

---

Performance

Benchmarked on a 2-core, 4 GB RAM instance:

Metric	Value
Throughput (L1 cache hit)	~15,000 req/s
Throughput (L2 cache hit)	~8,000 req/s
Throughput (provider call)	~600 req/s
P50 latency (L1 hit)	1ms
P99 latency (L1 hit)	4ms
P50 latency (L2 hit)	3ms
Memory usage (idle)	~65 MB

---

License

MIT License. See LICENSE for details.

---

Built by Jose Nevado