orfora

Route each LLM request to the right-sized model. Cut cost, keep quality.

Most LLM apps send every request to one big model. But most requests do not need it: a short rewrite, a classification, or a quick factual answer runs just as well on a model that costs a fraction. orfora (Optimal Routes For All) looks at each request and picks the model that actually fits it, so you stop overpaying for the easy ones without giving up quality on the hard ones.

It decides by meaning, not by length. Length is a poor proxy: a two-line prompt can need deep reasoning, and a long one can be a trivial reformat. orfora compares each request to a handful of example prompts (yours, or the built-in defaults) and routes to the closest match. The decision is pure math plus a small embedding, so no extra LLM call is made just to route. It is fast (milliseconds) and cheap.

The guiding principle: route to the most appropriate model, not the cheapest and not always the biggest. A trivial request goes to a small model, a genuinely hard one goes to a strong model. Low cost is a consequence of good fit, not the goal.

By default orfora returns a decision and does not sit in front of your model calls: no proxy hop, no extra service, no single point of failure. When you want a drop-in instead, an optional gateway wraps that same decision as an OpenAI-compatible proxy.

Status: early development (0.1.0). The core works, is covered by tests, and is on npm. The API may still change before 1.0.

Install

npm install orfora

orfora needs two things: an embedder (to turn a request into a vector and route it) and, if you want orfora to call the model too, a way to reach models. The embedder is required even for the gateway. Pick an embedder:

• orfora/openai, any OpenAI-compatible embeddings API (needs a key).
• orfora/local, a small model on device via transformers.js, no key. Install the peer once (npm install @huggingface/transformers); the model downloads on first use.
• orfora/openrouter, one key that both embeds and reaches every model in the catalog.

// keyless, on-device embedding
import { localEmbedder } from "orfora/local";
const embed = localEmbedder(); // downloads all-MiniLM-L6-v2 on first call

Which router do I use?

You want	Use
Cheapest split between a small and a strong model	complexityRouter
Route across many models by real benchmark fitness	createVectorRouter
Keep OpenAI-shaped code, auto-route with no logic change	the gateway (createOrforaClient / orforaHandler)
Fully custom named routes with your own seeds	createRouter
Route image / video / speech / music generation too	createMultimodalRouter

Start with complexityRouter or the gateway. createVectorRouter is the advanced engine (model-as-vector fitness); createCatalogRouter is its capability x tier grid predecessor.

Quickstart: two-tier cost routing

The headline use case, with zero hand-written seeds: a cheap model for easy requests, a strong model for the ones that actually need it.

import { complexityRouter } from "orfora";
import { openaiEmbedder } from "orfora/openai";

const router = complexityRouter({
  simple: "openai/gpt-5-nano",
  complex: "anthropic/claude-opus-5",
  embed: openaiEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
});

const { model } = await router.route("Summarize this paragraph in one line.");
// model === "openai/gpt-5-nano"  (an easy request, so it takes the cheap model)

// orfora only decides which model. You make the call, with your own client.

Route across many models: the catalog router

For real apps you usually want more than two models. createCatalogRouter decides on two independent axes and maps the result to a concrete model:

• Capability (what kind of task): code, math_reasoning, creative_writing, live_web_search, general_qa.
• Tier (how strong a model it warrants): cheap, mid, premium, plus an opt-in ultra for the hardest code and reasoning.

It ships over the same current catalog as the vector router (one shared source of model identity) and a default grid that maps each capability and tier to a best-fit model. Deterministic signals run alongside the semantic match: an attachment routes to a vision model, a very long prompt to a long-context model, freshness words ("today", "latest") to live web search, and a multi-intent premium request escalates to ultra. Anything unclear fails open to a capable general model.

Paired with orfora/openrouter, one key reaches every model in the catalog:

import { createCatalogRouter } from "orfora";
import { openrouterEmbedder, openrouterHandlers } from "orfora/openrouter";

const apiKey = process.env.OPENROUTER_API_KEY;

const router = createCatalogRouter({
  embed: openrouterEmbedder({ apiKey }),
  handlers: openrouterHandlers({ apiKey }),
});

const decision = await router.route("Refactor this module to remove the duplication.");
// decision.target === "code", tier by difficulty, decision.model a real coding model

const answer = await router.run("What is the USD to EUR rate today?");
// the freshness signal routes to live web search, then orfora calls the model for you

The catalog, the grid, and every seed set are just defaults. Pass your own grid, capabilitySeeds, or tierSeeds to fit your traffic.

Route by real benchmarks: the vector router

createVectorRouter goes one step further: instead of a hand-mapped grid, each model carries a capability VECTOR scored from real published benchmarks (SWE-bench, GPQA, IFEval, BFCL, LMArena Elo, and more). A request picks its capability by meaning, and the router picks within the tier by fitness: the cheapest model that clears the bar at lower tiers, or the strongest at premium and ultra, subject to hard gates (an image needs a vision-capable model, live search needs a web-search model).

import { createVectorRouter } from "orfora";
import { openrouterEmbedder } from "orfora/openrouter";

const router = createVectorRouter({
  embed: openrouterEmbedder({ apiKey: process.env.OPENROUTER_API_KEY }),
});

const decision = await router.route("Fix the failing test in this module.");
// decision.capability === "code", decision.model = the best-fit coding model for
// the request's tier, decision.fitness = its score for the request

It ships with a catalog of ~40 current (August 2026) models scored from LIVE, cited benchmarks, one canonical benchmark per capability axis normalised to [0,1]. Because vendors report different suites (SWE-bench Verified vs Pro, Terminal-Bench 2.0/2.1/3.0), a score is a relative fitness proxy, not an absolute truth; profileFrom names the real benchmark and source on every model. A missing score is left out (never guessed) and the router degrades gracefully to a neutral prior and price-tier routing over the gap. If a current model has no number of its own, one is carried forward from its family's latest measured version, recorded in profileFrom.

Beyond text: the generative branch

Not every request wants a text answer. createMultimodalRouter adds a second branch in front of the chat router: it detects a request's OUTPUT modality (text, image, video, speech, or music) from the same embedding, then either delegates text to the chat router above, untouched, or matches a best-fit GENERATIVE model.

import { createMultimodalRouter } from "orfora";
import { openaiEmbedder } from "orfora/openai";

const router = createMultimodalRouter({
  embed: openaiEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
});

await router.route("Refactor this module.");           // { modality: "text", ...chat decision }
await router.route("Design a poster with bold text."); // { modality: "image", model: "ideogram/ideogram-v3", ... }
await router.route("Read this paragraph aloud.");        // { modality: "speech", ... }

Output modality is orthogonal to the chat capability: it asks what the user wants PRODUCED, not what task the text is. Describing an image to generate is image output; an uploaded image would be an input gate instead. Detection reuses the capability seeds as the "text" anchor and biases to text, so a normal request is never mis-sent to an image generator. Within a modality the same right-fit rule applies: a specialty match (typography to Ideogram, dialogue video to Veo, voice-clone to ElevenLabs), then a draft-vs-final quality need, then the cheapest that fits. The generative catalog carries ~30 current image / video / speech / music models with live-sourced pricing, coarse quality tiers, specialty tags, and a cited source per model.

Cascade: verify, then escalate

Predicting the tier is one paradigm; verifying an answer is the other. router.plan() returns an escalation ladder (the best model per price tier, cheapest first) for a request's capability, and runCascade executes it: call the cheapest, check the answer, and climb only when the check fails, so frontier tokens go only to the requests that provably needed them.

const plan = await router.plan("Explain this stack trace and propose a fix.");
const result = await runCascade(
  plan,
  (step) => callModel(step.model, prompt), // your model call
  (answer) => heuristicVerify(answer), // your verifier: a judge, a test, a schema check
);
// result.step is the model that answered; result.escalated tells you if it climbed.

orfora stays LLM-free here: it plans and runs the ladder, the verifier is yours (a judge model, a unit test, a JSON-schema check). heuristicVerify is a cheap default that rejects empty, too-short, or refusal answers so they escalate.

Custom routes

If you want full control, createRouter is the primitive underneath both wrappers. Define any named routes you like, each with a model and a few seed examples:

import { createRouter } from "orfora";
import { openaiEmbedder } from "orfora/openai";

const router = createRouter({
  routes: {
    coding: { model: "anthropic/claude-opus-5", seeds: ["write a function that...", "fix this bug..."] },
    chat: { model: "openai/gpt-5.6-luna", seeds: ["hi", "what's the weather like"] },
    vision: { model: "google/gemini-3.7-flash", seeds: ["describe this image"] },
  },
  fallback: "chat",
  embed: openaiEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
  signals: {
    maxChars: 4000, // very long prompts escalate to the fallback route
    multiIntent: true, // several questions or tasks escalate too
    onModality: { image: "vision", video: "vision" }, // attachments go to the right model
  },
});

const decision = await router.route({ prompt: "what's in this photo?", attachments: ["photo.png"] });
// decision.route === "vision", decision.reason === "signal:modality:image"

openaiEmbedder takes a baseURL, so it works with any OpenAI-compatible provider (Together, OpenRouter, a local Ollama). Or bring your own embedder: any object with embed(texts: string[]): Promise<number[][]> works.

Let orfora call the model (optional)

By default orfora returns a decision and you make the call. If you would rather it call the model for you, give it a handler per model and use the router's .run() method (every router exposes it). orfora still never imports a provider SDK, it only calls the function you supply; if any reachable model has no handler, creation throws once, listing all of them.

const router = createRouter({
  routes: {
    chat: { model: "openai/gpt-5.6-luna", seeds: ["hi", "what's the weather like"] },
    reasoning: { model: "anthropic/claude-opus-5", seeds: ["prove that...", "design a system that..."] },
  },
  fallback: "reasoning",
  embed: openaiEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
  handlers: {
    "openai/gpt-5.6-luna": (input) => callModel("openai/gpt-5.6-luna", input.prompt),
    "anthropic/claude-opus-5": (input) => callModel("anthropic/claude-opus-5", input.prompt),
  },
});

const answer = await router.run("Summarize this in one line.");

With orfora/openrouter this is a one-liner: openrouterHandlers({ apiKey }) builds a handler for every model in the shared catalog, so it works with both the catalog router and the vector router, and run() decides and calls through your key.

Drop it in: the OpenAI-compatible gateway

orfora/gateway is the wrapping surface. Keep your OpenAI-shaped code, set model: "auto", and every request routes to its best-fit model. The prompt is embedded ONCE for routing, then the call is forwarded verbatim with the model swapped, streaming and all, and the routing shows up as metadata.

In-process (an OpenAI-SDK-shaped client):

import { createOrforaClient } from "orfora/gateway";
import { openaiEmbedder } from "orfora/openai";

const client = createOrforaClient({
  embed: openaiEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
  forward: { mode: "openrouter", apiKey: process.env.OPENROUTER_API_KEY },
});

const res = await client.chat.completions.create({
  model: "auto", // or pin a model id to skip routing
  messages: [{ role: "user", content: "Refactor this module and explain the change." }],
});
// res.orfora = { routed, model, capability, tier, fitness, estCostPerMTokens }
// res.choices[0].message.content is the answer, typed (no cast needed).

Streaming returns an async iterable of chunks; the first carries the orfora metadata:

const stream = await client.chat.completions.create({
  model: "auto",
  stream: true,
  messages: [{ role: "user", content: "Write a haiku about routing." }],
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content ?? "");
}

As a proxy (any language, any stack). orforaHandler is a fetch-style (Request) => Response, so it drops onto Web-standard runtimes (Vercel edge, Bun, Deno) as-is. On Node (http.createServer, Express, Fastify) wrap it with toNodeHandler from orfora/node. Point your OpenAI base URL at it and set model: "auto":

// api/chat.ts (Vercel edge)
import { orforaHandler } from "orfora/gateway";
import { openaiEmbedder } from "orfora/openai";

export const config = { runtime: "edge" };
export default orforaHandler({
  embed: openaiEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
  forward: { mode: "openrouter", apiKey: process.env.OPENROUTER_API_KEY },
  // The handler forwards with YOUR provider key, so gate who may call it.
  authorize: (req) => req.headers.get("authorization") === `Bearer ${process.env.PROXY_KEY}`,
});

On Node, wrap the same handler with toNodeHandler:

import { createServer } from "node:http";
import { orforaHandler } from "orfora/gateway";
import { toNodeHandler } from "orfora/node";

createServer(toNodeHandler(orforaHandler({ embed, forward }))).listen(3000);

The handler is a keyed proxy, so it is unauthenticated unless you pass authorize (return false to reject with 401); a body-size cap (maxBodyBytes, default 1MB) guards oversized requests, and you should rate-limit at the edge on a public deployment.

Forwarding is flexible: { mode: "openrouter", apiKey } reaches every model with one key, or { mode: "providers", providers: { anthropic: { baseURL, apiKey }, ... } } sends each request to your own provider keys (the bare model name to each provider's OpenAI-compatible endpoint). Streaming passes through as text/event-stream, and the routing decision rides along in response headers.

The routing metadata, on the response body as orfora (in-process) and as headers (proxy):

OrforaMeta field	Header	Meaning
routed	x-orfora-routed	true if orfora chose the model, false if you pinned it
model	x-orfora-model	the model the request was sent to
capability	x-orfora-capability	the detected capability (e.g. code, general_qa)
tier	x-orfora-tier	the tier the request landed in
fitness	x-orfora-fitness	the chosen model's fitness for the capability
estCostPerMTokens	—	the model's output price per 1M tokens (its pricePerMTokens)

Tuning

• Seeds. A handful per route is enough to start, five to ten each. The closer they are to your real traffic, the better it routes. Override the defaults any time with your own labelled examples.
• threshold. The minimum confidence to trust a match. Leave it at 0 to always take the nearest route, or raise it to fall back to the strong model when unsure.
• signals. Deterministic guards that run before the embedding: escalate very long or multi-intent prompts, and send attachments to a model that can handle them.
• fallback. Where routing fails open when it cannot decide. Defaults to "strongest" so a bad guess never trades quality; set "cheapest" (or a specific model id) for a cost-safe fail-open instead.

How it works

1. Setup (once): each route's seed examples are embedded to vectors.
2. Per request:
   • Deterministic signals run first (length, multi-intent, attachment modality, and in the catalog router also freshness and long-context). They can decide without any embedding, which is both a safety guard and a cost saving.
   • Otherwise the request is embedded once and compared (cosine similarity) to the nearest seed. The catalog router compares against the capability and tier seed banks at the same time, so the second axis costs no extra embedding.
   • If confidence is below threshold, or anything errors, orfora fails open to a capable model, so quality is never sacrificed on a bad guess.

It routes by task type and difficulty, not by subject, so it generalises across domains.

Does it actually route well?

On a 40-example labelled set for the two-tier complexity case (kept separate from the seeds, embedded with text-embedding-3-small), orfora scored:

• 95% accuracy
• 100% complex-recall: it never sent a hard request to the cheap model
• 90% simple-recall: it sent 18 of 20 easy requests to the cheap model
• 45% of requests routed to the cheap model

Complex-recall is the number that matters most. orfora is built never to trade quality for cost, and on this set it never did. Reproduce it with npm run eval using your own OpenAI-compatible key.

For the vector router, on a 40-example main set plus a 19-example adversarial set (ambiguous, out-of-distribution, non-English, multi-intent), with centroid capability matching and out-of-distribution abstention:

• 97.5% capability accuracy on the main set, 78.9% on the adversarial set: the request's task type (code, reasoning, writing, live search, general) is picked correctly, and unclear or out-of-distribution inputs fall open to a general model instead of a confident wrong specialist.
• Tier is the hard axis, so it is LEARNED from real outcomes, not hand-tuned. Which strength tier a request warrants is fuzzy, and seed-based tiering barely beats chance on held-out data (35% on a balanced 3-tier set). An opt-in tierPredictor, a logistic regression over prompt embeddings trained on RouterBench (each prompt labelled with the cheapest tier that actually solved it), reaches 56% under 5-fold cross-validation vs 35% for the seeds and 33% majority. It is embedding-space specific, so it ships off by default and is enabled where the embedder matches (as in the playground).
• Cost, with an honest caveat (not a quality guarantee): on the main set the routed models average 90% of the flagship's benchmark fitness (0.780 vs 0.866) at 11.5% of its cost ($5.73 vs $50.00 per M tokens), and 89% of fitness (0.775 vs 0.872) at 19% of cost on the adversarial set. Read this as a cost signal, not a measured-quality claim: the "quality" and the baseline are the same benchmark-fitness proxy (one score on both sides of the ratio), so it means "on paper we rarely fall far below the flagship," not "you get 90% of real answer quality." Cost is higher than a cheapest always router on purpose: premium and ultra requests now take the strongest model, so a hard request pays for the model it needs. Tier thresholds are derived from the catalog's own fitness distribution, so every tier is reachable rather than a hand-set bar.

Reproduce with npm run eval:catalog or npm run eval:vector using your own OpenAI-compatible key.

Why orfora (vs LLM gateways)

At its core orfora is a small decision library, not a gateway like LiteLLM or OpenRouter, and by default that is the point:

• No proxy in your request path. In decision mode orfora returns a model id and your app calls the model: no added latency on completions, no single point of failure.
• No infrastructure. npm install, runs in your process. No proxy, no Postgres, no Redis to operate.
• Your own keys. The decision library takes no cut and holds no credentials.
• Provider-agnostic and composable. It returns a model id, so you can use it standalone or feed that decision into a gateway you already run.

The optional orfora/gateway trades some of this for convenience: it IS an OpenAI-compatible proxy, so it does sit in your path and forwards with your provider key (gate it with authorize). Reach for the decision library when you want zero added surface, the gateway when you want a drop-in. Either way orfora does one thing, pick the model: it is not a feature-for-feature gateway replacement (no unified multi-provider API, no load balancing, no spend dashboards). Its decision still costs one embedding call, so use a local embedder if you want zero network.

Playground

A no-key browser demo lives in playground/ (it runs the local embedder in the browser). Try it locally with cd playground && npm install && npm run dev.

Roadmap

• Broader default seeds and domain presets (support bot, coding assistant, and so on)
• An optional multilingual capability preset (paired with a multilingual embedder)
• Auto-calibration that suggests thresholds from real traffic
• Optional feedback-based adjustment from real outcomes

License

MIT