[Feature]: Dynamic Model Selection and Smart Routing via TypeSafe Jev #5626

Area

Proxy and routing

What are you trying to accomplish?

Introduce dynamic model selection and intelligent request routing in OpenCodex proxy powered by TypeSafe Jev.

When downstream coding clients or developer tools (e.g. Codex, Claude Code, Cursor) send requests to OpenCodex specifying a virtual model alias (such as auto or jev-auto), OpenCodex should perform a low-latency (<50ms) structured semantic classification via Jev before forwarding the request. Based on task intent, code complexity, and reasoning depth, OpenCodex automatically dispatches the prompt to the most optimal model available in the user's configured provider pool (e.g., routing light edits to fast models, complex architecture/refactoring to flagship models, and algorithmic/formal reasoning to thinking models).

What prevents this today?

1. Static Model Binding: Model selection is currently tied to static configuration per client or fixed fallback chains. Users must manually switch models when alternating between simple tasks and complex reasoning.
2. Traditional LLM-as-a-Router Latency Overhead: Using a general LLM (such as GPT-4o-mini or Claude Haiku) for upstream routing introduces 500ms–2s of generation latency and extra token billing on every request, which is unacceptable in an interactive local proxy hot-path.
3. Fragile Regex / Keyword Heuristics: Rule-based prompt matching cannot gauge actual problem complexity, multilingual context, or nuanced developer intent, leading to frequent misrouting.

What should OpenCodex do?

1. Provide a Routing Strategy Extension: Add a Jev-powered routing provider/strategy in OpenCodex proxy configuration (routing.strategy = "jev").
2. Support Virtual Model Aliases: Recognize virtual model identifiers (e.g. auto, jev-auto) in incoming /v1/chat/completions and Responses requests.
3. Sub-50ms Semantic Judgments: Query TypeSafe Jev (a System One typed classifier primitive returning deterministic judgments without autoregressive generation overhead) to categorize task complexity (e.g., quick_edit, general_coding, deep_reasoning, tool_orchestration).
4. Map Decisions to Configured Models: Resolve the Jev decision through a configurable tier/mapping table against available providers in config.toml.
5. Observability & Response Headers: Transparently proxy the stream to the chosen target, appending metadata headers (e.g., x-opencodex-routed-model, x-opencodex-router-latency-ms).
6. Graceful Millisecond Fallback: Fall back cleanly to the configured default provider/model if Jev times out, errors, or returns low confidence.

Example usage or interface

1. Configuration in config.toml / ocx config

[routing]
strategy = "jev"
virtual_models = ["auto"]

[routing.jev]
api_key = "typesafe_..."
timeout_ms = 100
fallback_model = "gemini-3.8-flash"

[routing.jev.mappings]
simple_qa = "gemini-3.8-flash"
quick_edit = "gemini-3.8-flash"
complex_coding = "claude-sonnet-4-6"
deep_reasoning = "claude-opus-4-6-thinking"

2. Client Request Example

# Client sends request targeting virtual model 'auto'
curl http://localhost:10100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "Analyze potential race conditions in this Linux kernel driver and generate a patch..."}
    ]
  }'

3. Observability in Headers

HTTP/1.1 200 OK
x-opencodex-routed-model: claude-opus-4-6-thinking
x-opencodex-router-decision: deep_reasoning
x-opencodex-router-latency-ms: 41
Content-Type: text/event-stream
...

Alternatives or workarounds

• Manual Switching: Manually updating model configs or toggling client settings back and forth between light and heavy tasks.
• Keyword / Prompt Pattern Heuristics: Attempting regex rules on prompt keywords, which fails on complex multi-turn chats and mixed language requests.
• Full LLM Classifier Routers: Using upstream LLMs for routing, which adds unacceptable latency (>500ms) to every turn.

Additional context

• TypeSafe Jev is designed specifically as a System One programming primitive that yields typed judgments and probabilities with minimal latency, avoiding LLM generation overhead.
• OpenCodex already features mature account pooling, provider adapters, and proxy pipeline architecture; adding an intelligent routing phase fits naturally before provider dispatch.

Checks

• I searched existing issues and documentation.
• This request describes a concrete OpenCodex workflow rather than merely naming a desired technology.
• I removed secrets and personal data.