Repository navigation
ci: when the agent pin cannot think, say which models can - #23
Merged
Merged
Conversation
#22 made the failure fast and accurate, but its remedy -- "point LLM_GATEWAY_MODEL at a thinking-capable group" -- is not actionable on its own. The gateway registers model groups by role in the LiteLLM DB, not in git (litellm-config.yaml deliberately carries no model_list), so there is nothing to read and the names give nothing away: this key serves chat-default, tools-default, embed-default, vision-default and vision-accurate. So probe the rest of the allow-list and report. Two distinct verdicts, which matter because they need different people to do different things: - at least one passes -> name them; the fix is to repoint the secret. - none pass -> say plainly that repointing cannot fix it, and that the gateway needs a thinking-capable group registered and in the key's scope. Embedding models are skipped and the list is capped at 12. This runs only on the failure path, so a healthy run pays nothing for it. Verified against a mock gateway serving /v1/models plus per-model thinking behaviour: with two capable models it names exactly those two (skipping both the embedding model and the already-failed pin); with none capable it emits the "not fixable by repointing" error; and a healthy pin still exits 0 without probing anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DobwJ8YTnfowwqwhspU1kQ
aroff
enabled auto-merge (squash)
September 4, 2026 16:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#22 made the agent-route preflight fast and accurate — it now fails in ~30s instead of burning
60 minutes, and it quotes the backend's own error. But its remedy is not actionable:
Point it at which one? The gateway registers model groups in the LiteLLM DB
(
litellm-config.yamlsetsstore_model_in_db: trueand deliberately carries nomodel_list),so there is nothing in git to read, and the names give nothing away. This key's catalog, printed
by the judge preflight on run 33896269263, is:
Role names, not capabilities. Guessing costs a dispatch each.
What
On the failure path only, probe the rest of the allow-list with the same extended-thinking
request that just failed, and report one of two verdicts:
::error::set LLM_GATEWAY_MODEL to one of: <names>::error::no model in this key's allow-list can serve an extended-thinking request. This is not fixable by repointing the secret…That distinction is the point: the two outcomes need different people to do different things, and
today the log cannot tell them apart.
Embedding models are skipped (they cannot serve
/v1/messagesat all) and the list is capped at12 so a large catalog cannot turn the diagnostic into its own long job. A healthy run never
reaches this code and pays nothing.
Verification
Against a mock gateway serving
GET /v1/modelsplus per-model thinking behaviour, with the stepscript extracted from this YAML (
yaml.safe_load) rather than retyped:PASS chat-default,PASS vision-default, then::error::set LLM_GATEWAY_MODEL to one of: chat-default vision-default, exit 1.Both the embedding model and the already-failed pin were skipped.
failfor each, then the "not fixable by repointing" error, exit 1.chat-default) →minimal request: 200,thinking request: 200 — the agent route can serve this model., exit 0, no probing.🤖 Generated with Claude Code
https://claude.ai/code/session_01DobwJ8YTnfowwqwhspU1kQ