Skip to content

About

LiteLLM in Rust — 15× lower memory, 2.5× lower latency. Full OpenAI-compatible API, 20+ providers, OIDC, Langfuse, Datadog, semantic caching.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

LiteLLM in Rust

CI License: MIT

A high-performance LLM proxy with the same feature set as LiteLLM, built in Rust. 15× lower memory, 2.5× lower latency, same throughput.

Drop-in replacement for any OpenAI-SDK client — point your base_url at this proxy and it routes to OpenAI, Anthropic, Gemini, Bedrock, and 110+ preconfigured providers (any OpenAI-compatible endpoint is a one-line config entry).


Performance vs LiteLLM

Benchmarked on the same machine, same OpenAI backend, caching disabled for a fair comparison.

Performance comparison: 15× lower memory, 2.5× lower proxy overhead, 1.5× faster cache hits

Metric LiteLLM-in-Rust LiteLLM (Python)
Idle memory (RSS) 20 MB 313 MB
Proxy overhead p50 1.1 ms 2.7 ms
Concurrent throughput (c=15, 60 req) ~24 req/s ~26 req/s
Cache hit latency 10 ms ~15 ms
Cold start < 50 ms 3–5 s
Binary size ~12 MB (stripped) N/A

Memory is 15× lower. Proxy overhead is 2.5× lower. Throughput is network-bound at parity — the bottleneck is the upstream LLM, not the proxy.

See COMPARISON.md for the full feature-by-feature parity matrix, benchmark methodology, and an honest account of where each project is ahead.


Dashboard

A built-in, zero-dependency monitoring dashboard at /admin — spend, latency, error rates, per-model breakdowns, a live request stream, and virtual-key management. No Grafana or external service required. Ships with light and dark themes (toggle in the header; follows your OS preference by default).

llm-proxy monitoring dashboard in dark mode
Dark mode

Light mode

llm-proxy monitoring dashboard in light mode


Features

Full OpenAI-compatible API surface

Endpoint Notes
POST /v1/chat/completions Streaming SSE supported
POST /v1/completions Legacy text completion
POST /v1/embeddings
POST /v1/images/generations DALL-E
POST /v1/images/edits
POST /v1/audio/speech TTS
POST /v1/audio/transcriptions Whisper
POST /v1/audio/translations
POST /v1/moderations
GET /v1/models Static + runtime-registered routes
POST /v1/files, /v1/batches Files & Batch API
POST /v1/assistants, /v1/threads Assistants API
POST /v1/vector_stores Vector Stores
POST /v1/fine_tuning/jobs Fine-tuning

Providers

Native (full tool calling + vision): OpenAI · Anthropic · AWS Bedrock · Google Gemini · Cohere · Azure OpenAI

Via OpenAI-compatible compat: config (no code needed) — 105 preconfigured: Groq · Together · Mistral · DeepSeek · OpenRouter · Perplexity · Fireworks · Cerebras · SambaNova · NVIDIA NIM · xAI Grok · DeepInfra · Nebius · Moonshot · Zhipu · Qwen/DashScope · SiliconFlow · Baseten · Portkey · Unify · Requesty · … plus self-hosted runtimes (Ollama · vLLM · LM Studio · SGLang · LMDeploy · LocalAI). Add any OpenAI-compatible endpoint with a URL and a key — see config.example.yaml.

Routing & reliability

  • Load balancing — round-robin across multiple deployments per model alias
  • Automatic failover — next deployment on error; route-level fallback chains
  • Circuit breaker — exponential backoff per deployment, auto-recovery
  • Retries — configurable per route

Caching

  • Exact-match — SHA-256 of model + messages + params, TTL-based
  • Semantic — embedding-based similarity (configurable threshold), catches rephrased duplicates
  • Distributed — point every replica at one Redis (redis.url) to share the cache, rate-limit counters, and budget enforcement; falls back to in-process/SQLite if Redis is unset or down

Auth & budgets

  • Virtual keys — issue/revoke via API, hash-stored, never logged in plaintext
  • Master key — works on all endpoints (admin + inference)
  • RBAC — admin (full), key_manager (create/revoke keys only), and viewer (read-only) roles, enforced per action (keys/teams/routes)
  • Teams — shared spend tracked across all member keys
  • Budgets — per-key and per-team USD caps with daily/weekly/monthly resets. Enforced as a hard cap: cost is atomically reserved before the call and settled to the real amount after, so concurrent requests can't collectively overshoot the budget. With Redis configured, the cap is enforced across all replicas (atomic compare-and-set)
  • OIDC/OAuth2 — accept JWTs from Auth0, Okta, Google, Keycloak, Cognito, Azure AD, etc. Keys are discovered via the issuer's /.well-known/openid-configuration (JWKS URI), so it works across IdPs that don't use the conventional JWKS path
  • Rate limiting — per-key RPM and TPM windows

Observability

  • Langfuse — fire-and-forget generation traces
  • Datadog — DogStatsD metrics (llm_proxy.requests, .latency_ms, .cost_usd, …)
  • OpenTelemetry — OTLP traces via OTEL_EXPORTER_OTLP_ENDPOINT (Jaeger, Grafana Tempo, Honeycomb, …)
  • Structured JSON logs — via tracing-subscriber
  • Audit log — every key/team/route mutation recorded
  • Admin dashboard — Datadog-style monitoring UI at /admin

Operations

  • Runtime model registration — POST /admin/routes adds a route instantly, no restart, persisted across restarts
  • CLI — llm-proxy-cli mirrors LiteLLM's CLI for key/team/route management
  • Pre-call token estimation — x-estimated-prompt-tokens response header
  • Real health checks — GET /health/deployments probes every backend with a live request
  • Provider pass-through — POST /passthrough/:provider/* for native SDK clients

Quick start

1. Build

cargo build --release

2. Configure

cp config.example.yaml config.yaml

Set your API keys in config.yaml (or via env vars — see comments in the example):

providers:
  openai:
    api_key: sk-...         # or set OPENAI_API_KEY
  anthropic:
    api_key: sk-ant-...     # or set ANTHROPIC_API_KEY
  gemini:
    api_key: AIza...        # or set GEMINI_API_KEY

auth:
  master_key: "sk-my-master-key"   # required to call /admin/* endpoints

3. Run

./target/release/llm-proxy
# or: OPENAI_API_KEY=sk-... MASTER_KEY=sk-... cargo run --release

4. Call it

The proxy is a drop-in for any OpenAI SDK. Change only the base_url:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-virtual-key",   # issued via CLI or admin API
)

response = client.chat.completions.create(
    model="gpt-4o-mini",          # or any alias from your routes:
    messages=[{"role": "user", "content": "Hello!"}],
)
# curl
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello!"}]}'

Python SDK

A Python client wraps both inference and management in one object:

from llmproxy import LLMProxy

admin = LLMProxy("http://localhost:8080", api_key="sk-master")
issued = admin.create_key(name="my-app", budget_usd=5.0, budget_duration="monthly")

app = LLMProxy("http://localhost:8080", api_key=issued["key"])
print(app.chat("gpt-4o-mini", [{"role": "user", "content": "Hello!"}]))

print(admin.stats_overview())   # spend / latency / error rate

pip install ./sdk/python — covers chat, chat_stream, completion, embeddings, models, and the full admin surface (create_key, list_keys, revoke_key, create_team, add_route, stats, audit). See sdk/python/README.md.

Key management

Via CLI

# Create a virtual key (budget $10/month, 60 RPM)
./llm-proxy-cli --key sk-my-master-key create-key \
  --name "dev-team" --budget 10.0 --budget-duration monthly --rpm 60

# List keys
./llm-proxy-cli --key sk-my-master-key list-keys

# Revoke a key
./llm-proxy-cli --key sk-my-master-key revoke-key 3

# Add a model route at runtime (no restart)
./llm-proxy-cli --key sk-my-master-key add-route \
  --alias my-claude --provider anthropic --model claude-3-5-sonnet-20241022

# View spend overview
./llm-proxy-cli --key sk-my-master-key stats

Via API

# Create a key
curl -X POST http://localhost:8080/admin/keys \
  -H "Authorization: Bearer sk-my-master-key" \
  -H "Content-Type: application/json" \
  -d '{"name":"dev-team","budget_usd":10,"budget_duration":"monthly","rpm_limit":60}'

# List keys
curl http://localhost:8080/admin/keys \
  -H "Authorization: Bearer sk-my-master-key"

# Create a team
curl -X POST http://localhost:8080/admin/teams \
  -H "Authorization: Bearer sk-my-master-key" \
  -H "Content-Type: application/json" \
  -d '{"name":"engineering","budget_usd":100,"budget_duration":"monthly"}'

Routing

Each model alias load-balances across one or more deployments with automatic failover:

routes:
  "gpt-4o":
    deployments:
      - provider: openai
        model: gpt-4o
      - provider: anthropic       # failover to Claude if OpenAI errors
        model: claude-3-5-sonnet-20241022
    fallback: "gpt-4o-mini"       # if all deployments fail, fall back to this alias
    max_retries: 2

Add providers that speak the OpenAI wire format with zero code:

providers:
  compat:
    - name: groq
      api_key_env: GROQ_API_KEY
      base_url: "https://api.groq.com/openai/v1"
    - name: azure-openai
      auth_header: "api-key"      # Azure uses api-key instead of Bearer
      api_key_env: AZURE_OPENAI_API_KEY
      base_url: "https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOY"

Caching

cache:
  enabled: true
  ttl_seconds: 300
  semantic: true                  # embedding-based similarity matching
  semantic_model: "text-embedding-3-small"
  semantic_threshold: 0.95        # 0.0–1.0, higher = stricter match

Cache hit: 740 ms → 10 ms.


Observability

observability:
  langfuse:
    enabled: true
    public_key: pk-lf-...
    secret_key: sk-lf-...

  datadog:
    enabled: true
    statsd_addr: "127.0.0.1:8125"
# OpenTelemetry OTLP (Jaeger, Grafana Tempo, Datadog, Honeycomb, ...)
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 ./llm-proxy

OIDC / SSO

Accept JWTs from any standards-compliant identity provider:

auth:
  oidc:
    enabled: true
    issuer_url: "https://YOUR_TENANT.auth0.com/"
    audience: "your-api-audience"

Works with Auth0, Okta, Google Workspace, Keycloak, AWS Cognito, Azure AD, and any other OIDC-compliant IdP.


Health & monitoring

# Process liveness (for k8s/ALB — no network calls)
GET /health

# Readiness (DB reachable, at least one deployment configured)
GET /health/readiness

# Active probe — sends a real minimal request to every deployment
GET /health/deployments

# Admin dashboard
GET /admin

Docker / AWS

# Docker
docker build -t llm-proxy .
docker run -p 8080:8080 \
  -e OPENAI_API_KEY=sk-... \
  -e MASTER_KEY=sk-... \
  llm-proxy

Terraform for AWS App Runner in infra/.


Architecture

client
  │
  ▼
axum HTTP server (Tokio async)
  ├── auth (virtual keys · master key · OIDC JWT)
  ├── rate limiter (RPM/TPM per key, DashMap)
  ├── cache (exact SHA-256 · semantic cosine similarity)
  ├── model router
  │     ├── circuit breaker per deployment
  │     ├── round-robin + failover
  │     └── provider trait (chat · stream · embed)
  │           ├── OpenAI / compat (Groq, Together, …)
  │           ├── Anthropic
  │           ├── AWS Bedrock (SigV4)
  │           ├── Google Gemini
  │           └── Cohere
  ├── SQLite (rusqlite) — keys · teams · spend · audit
  └── callbacks (Langfuse · Datadog · OTLP)

License

MIT

About

LiteLLM in Rust — 15× lower memory, 2.5× lower latency. Full OpenAI-compatible API, 20+ providers, OIDC, Langfuse, Datadog, semantic caching.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages