Skip to content

Document LLM Gateway as a model provider in the Quick start - #90

Open
smakosh wants to merge 1 commit into
CopilotKit:mainfrom
smakosh:docs/llm-gateway
Open

smakosh wants to merge 1 commit into
CopilotKit:mainfrom
smakosh:docs/llm-gateway

Conversation

@smakosh

@smakosh smakosh commented Oct 28, 2026

Copy link
Copy Markdown

Closes #89

The problem

The Quick start names two ways to run the OpenDuet Bot: OpenAI and Anthropic. LLM Gateway serves models from many providers behind one OpenAI-compatible key, and OpenDuet already reaches it through OPENAI_BASE_URL and BOT_MODEL. Nothing says so, or which models will work.

The approach

One entry under To use another provider or model, beside OpenAI and Anthropic:

  • Three variables: OPENAI_BASE_URL=https://api.llmgateway.io/v1, OPENAI_API_KEY, and BOT_MODEL with an ID from the gateway's model list. Unset, the Bot keeps gpt-5.6-terra from model.yaml.
  • Which models: a model needs tools, for the Bot's computer, and JSON output, because skill choice, routing and action review call /chat/completions with response_format: json_object.
  • Claude models: the gateway refuses OpenAI's JSON mode for them, so they go through the Anthropic settings instead: BOT_PROVIDER=anthropic, ANTHROPIC_BASE_URL=https://api.llmgateway.io. normalizeModelBaseUrls adds the /v1.
  • Setup reads Not checked: checkModelAvailability asks only OpenAI and Anthropic about a key, and skips custom endpoints.

What is not covered

No code change and no new provider id. Everything goes through settings OpenDuet already reads. A first-class provider row would belong in OpenBot.

Verification

  • bun run format:check and bun run lint in openbot/ pass.

  • bun test tests/copilot.test.ts tests/routing-model-url.test.ts tests/model-availability-check.test.ts tests/openduet-setup.test.ts passes: 144 tests, 0 failures.

  • A throwaway script, not committed, runs OpenDuet's own code with the README's settings: normalizeModelBaseUrls, runtimeModelForEnvironment, CopilotKit's resolveModel with a tool, and the router's createModelCompleter. Against a local capture server, the requests were:

    Settings Bot request Router / review request
    OpenAI, no BOT_MODEL POST /v1/responses, gpt-5.6-terra, 1 tool, stream, Bearer POST /v1/chat/completions, gpt-5.6-terra, json_object
    OpenAI, BOT_MODEL=gemini-3.6-flash POST /v1/responses, gemini-3.6-flash, 1 tool, stream, Bearer POST /v1/chat/completions, gemini-3.6-flash, json_object
    Anthropic, BOT_MODEL=claude-sonnet-5-5 POST /v1/messages, claude-sonnet-5-5, 1 tool, stream, x-api-key POST /v1/messages, claude-sonnet-5-5
  • The same script against https://api.llmgateway.io with a placeholder key got the gateway's own API key not found answer on all three endpoints, not a 404. So the URLs are the gateway's real routes.

  • The public model list (GET /v1/models) lists gpt-5.6-terra, gemini-3.6-flash and claude-sonnet-5-5. The first two have tools and JSON output. claude-sonnet-5-5 has tools but json_output: false, which is why Claude goes through the Anthropic settings.

Not run: a full OpenDuet session with a live LLM Gateway key. This change is README only, so there is no screenshot or video.

- Add an LLM Gateway entry beside OpenAI and Anthropic, using the existing OPENAI_BASE_URL and BOT_MODEL settings
- Name the tools and JSON output a model needs, and the Anthropic settings for Claude models
- Say why Settings → Setup reads Not checked for a gateway key
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document LLM Gateway as a model provider

1 participant