A high-performance LLM proxy with the same feature set as LiteLLM, built in Rust. 15× lower memory, 2.5× lower latency, same throughput.
Drop-in replacement for any OpenAI-SDK client — point your base_url at this proxy and it routes to OpenAI, Anthropic, Gemini, Bedrock, and 110+ preconfigured providers (any OpenAI-compatible endpoint is a one-line config entry).
Benchmarked on the same machine, same OpenAI backend, caching disabled for a fair comparison.
| Metric | LiteLLM-in-Rust | LiteLLM (Python) |
|---|---|---|
| Idle memory (RSS) | 20 MB | 313 MB |
| Proxy overhead p50 | 1.1 ms | 2.7 ms |
| Concurrent throughput (c=15, 60 req) | ~24 req/s | ~26 req/s |
| Cache hit latency | 10 ms | ~15 ms |
| Cold start | < 50 ms | 3–5 s |
| Binary size | ~12 MB (stripped) | N/A |
Memory is 15× lower. Proxy overhead is 2.5× lower. Throughput is network-bound at parity — the bottleneck is the upstream LLM, not the proxy.
See COMPARISON.md for the full feature-by-feature parity matrix, benchmark methodology, and an honest account of where each project is ahead.
A built-in, zero-dependency monitoring dashboard at /admin — spend, latency, error rates, per-model breakdowns, a live request stream, and virtual-key management. No Grafana or external service required. Ships with light and dark themes (toggle in the header; follows your OS preference by default).
| Endpoint | Notes |
|---|---|
POST /v1/chat/completions |
Streaming SSE supported |
POST /v1/completions |
Legacy text completion |
POST /v1/embeddings |
|
POST /v1/images/generations |
DALL-E |
POST /v1/images/edits |
|
POST /v1/audio/speech |
TTS |
POST /v1/audio/transcriptions |
Whisper |
POST /v1/audio/translations |
|
POST /v1/moderations |
|
GET /v1/models |
Static + runtime-registered routes |
POST /v1/files, /v1/batches |
Files & Batch API |
POST /v1/assistants, /v1/threads |
Assistants API |
POST /v1/vector_stores |
Vector Stores |
POST /v1/fine_tuning/jobs |
Fine-tuning |
Native (full tool calling + vision): OpenAI · Anthropic · AWS Bedrock · Google Gemini · Cohere · Azure OpenAI
Via OpenAI-compatible compat: config (no code needed) — 105 preconfigured:
Groq · Together · Mistral · DeepSeek · OpenRouter · Perplexity · Fireworks · Cerebras · SambaNova · NVIDIA NIM · xAI Grok · DeepInfra · Nebius · Moonshot · Zhipu · Qwen/DashScope · SiliconFlow · Baseten · Portkey · Unify · Requesty · … plus self-hosted runtimes (Ollama · vLLM · LM Studio · SGLang · LMDeploy · LocalAI). Add any OpenAI-compatible endpoint with a URL and a key — see config.example.yaml.
- Load balancing — round-robin across multiple deployments per model alias
- Automatic failover — next deployment on error; route-level fallback chains
- Circuit breaker — exponential backoff per deployment, auto-recovery
- Retries — configurable per route
- Exact-match — SHA-256 of model + messages + params, TTL-based
- Semantic — embedding-based similarity (configurable threshold), catches rephrased duplicates
- Distributed — point every replica at one Redis (
redis.url) to share the cache, rate-limit counters, and budget enforcement; falls back to in-process/SQLite if Redis is unset or down
- Virtual keys — issue/revoke via API, hash-stored, never logged in plaintext
- Master key — works on all endpoints (admin + inference)
- RBAC —
admin(full),key_manager(create/revoke keys only), andviewer(read-only) roles, enforced per action (keys/teams/routes) - Teams — shared spend tracked across all member keys
- Budgets — per-key and per-team USD caps with daily/weekly/monthly resets. Enforced as a hard cap: cost is atomically reserved before the call and settled to the real amount after, so concurrent requests can't collectively overshoot the budget. With Redis configured, the cap is enforced across all replicas (atomic compare-and-set)
- OIDC/OAuth2 — accept JWTs from Auth0, Okta, Google, Keycloak, Cognito, Azure AD, etc. Keys are discovered via the issuer's
/.well-known/openid-configuration(JWKS URI), so it works across IdPs that don't use the conventional JWKS path - Rate limiting — per-key RPM and TPM windows
- Langfuse — fire-and-forget generation traces
- Datadog — DogStatsD metrics (
llm_proxy.requests,.latency_ms,.cost_usd, …) - OpenTelemetry — OTLP traces via
OTEL_EXPORTER_OTLP_ENDPOINT(Jaeger, Grafana Tempo, Honeycomb, …) - Structured JSON logs — via
tracing-subscriber - Audit log — every key/team/route mutation recorded
- Admin dashboard — Datadog-style monitoring UI at
/admin
- Runtime model registration —
POST /admin/routesadds a route instantly, no restart, persisted across restarts - CLI —
llm-proxy-climirrors LiteLLM's CLI for key/team/route management - Pre-call token estimation —
x-estimated-prompt-tokensresponse header - Real health checks —
GET /health/deploymentsprobes every backend with a live request - Provider pass-through —
POST /passthrough/:provider/*for native SDK clients
cargo build --releasecp config.example.yaml config.yamlSet your API keys in config.yaml (or via env vars — see comments in the example):
providers:
openai:
api_key: sk-... # or set OPENAI_API_KEY
anthropic:
api_key: sk-ant-... # or set ANTHROPIC_API_KEY
gemini:
api_key: AIza... # or set GEMINI_API_KEY
auth:
master_key: "sk-my-master-key" # required to call /admin/* endpoints./target/release/llm-proxy
# or: OPENAI_API_KEY=sk-... MASTER_KEY=sk-... cargo run --releaseThe proxy is a drop-in for any OpenAI SDK. Change only the base_url:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="your-virtual-key", # issued via CLI or admin API
)
response = client.chat.completions.create(
model="gpt-4o-mini", # or any alias from your routes:
messages=[{"role": "user", "content": "Hello!"}],
)# curl
curl http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-virtual-key" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello!"}]}'A Python client wraps both inference and management in one object:
from llmproxy import LLMProxy
admin = LLMProxy("http://localhost:8080", api_key="sk-master")
issued = admin.create_key(name="my-app", budget_usd=5.0, budget_duration="monthly")
app = LLMProxy("http://localhost:8080", api_key=issued["key"])
print(app.chat("gpt-4o-mini", [{"role": "user", "content": "Hello!"}]))
print(admin.stats_overview()) # spend / latency / error ratepip install ./sdk/python — covers chat, chat_stream, completion, embeddings, models, and the full admin surface (create_key, list_keys, revoke_key, create_team, add_route, stats, audit). See sdk/python/README.md.
# Create a virtual key (budget $10/month, 60 RPM)
./llm-proxy-cli --key sk-my-master-key create-key \
--name "dev-team" --budget 10.0 --budget-duration monthly --rpm 60
# List keys
./llm-proxy-cli --key sk-my-master-key list-keys
# Revoke a key
./llm-proxy-cli --key sk-my-master-key revoke-key 3
# Add a model route at runtime (no restart)
./llm-proxy-cli --key sk-my-master-key add-route \
--alias my-claude --provider anthropic --model claude-3-5-sonnet-20241022
# View spend overview
./llm-proxy-cli --key sk-my-master-key stats# Create a key
curl -X POST http://localhost:8080/admin/keys \
-H "Authorization: Bearer sk-my-master-key" \
-H "Content-Type: application/json" \
-d '{"name":"dev-team","budget_usd":10,"budget_duration":"monthly","rpm_limit":60}'
# List keys
curl http://localhost:8080/admin/keys \
-H "Authorization: Bearer sk-my-master-key"
# Create a team
curl -X POST http://localhost:8080/admin/teams \
-H "Authorization: Bearer sk-my-master-key" \
-H "Content-Type: application/json" \
-d '{"name":"engineering","budget_usd":100,"budget_duration":"monthly"}'Each model alias load-balances across one or more deployments with automatic failover:
routes:
"gpt-4o":
deployments:
- provider: openai
model: gpt-4o
- provider: anthropic # failover to Claude if OpenAI errors
model: claude-3-5-sonnet-20241022
fallback: "gpt-4o-mini" # if all deployments fail, fall back to this alias
max_retries: 2Add providers that speak the OpenAI wire format with zero code:
providers:
compat:
- name: groq
api_key_env: GROQ_API_KEY
base_url: "https://api.groq.com/openai/v1"
- name: azure-openai
auth_header: "api-key" # Azure uses api-key instead of Bearer
api_key_env: AZURE_OPENAI_API_KEY
base_url: "https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOY"cache:
enabled: true
ttl_seconds: 300
semantic: true # embedding-based similarity matching
semantic_model: "text-embedding-3-small"
semantic_threshold: 0.95 # 0.0–1.0, higher = stricter matchCache hit: 740 ms → 10 ms.
observability:
langfuse:
enabled: true
public_key: pk-lf-...
secret_key: sk-lf-...
datadog:
enabled: true
statsd_addr: "127.0.0.1:8125"# OpenTelemetry OTLP (Jaeger, Grafana Tempo, Datadog, Honeycomb, ...)
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 ./llm-proxyAccept JWTs from any standards-compliant identity provider:
auth:
oidc:
enabled: true
issuer_url: "https://YOUR_TENANT.auth0.com/"
audience: "your-api-audience"Works with Auth0, Okta, Google Workspace, Keycloak, AWS Cognito, Azure AD, and any other OIDC-compliant IdP.
# Process liveness (for k8s/ALB — no network calls)
GET /health
# Readiness (DB reachable, at least one deployment configured)
GET /health/readiness
# Active probe — sends a real minimal request to every deployment
GET /health/deployments
# Admin dashboard
GET /admin# Docker
docker build -t llm-proxy .
docker run -p 8080:8080 \
-e OPENAI_API_KEY=sk-... \
-e MASTER_KEY=sk-... \
llm-proxyTerraform for AWS App Runner in infra/.
client
│
▼
axum HTTP server (Tokio async)
├── auth (virtual keys · master key · OIDC JWT)
├── rate limiter (RPM/TPM per key, DashMap)
├── cache (exact SHA-256 · semantic cosine similarity)
├── model router
│ ├── circuit breaker per deployment
│ ├── round-robin + failover
│ └── provider trait (chat · stream · embed)
│ ├── OpenAI / compat (Groq, Together, …)
│ ├── Anthropic
│ ├── AWS Bedrock (SigV4)
│ ├── Google Gemini
│ └── Cohere
├── SQLite (rusqlite) — keys · teams · spend · audit
└── callbacks (Langfuse · Datadog · OTLP)
MIT

