From 54e872a180efef2dc4563eae042b203b93b5b56b Mon Sep 17 00:00:00 2001 From: Alex Oliveira <4482374+aroff@users.noreply.github.com> Date: Fri, 4 Sep 2026 16:41:26 +0000 Subject: [PATCH] ci: when the agent pin cannot think, say which models can #22 made the failure fast and accurate, but its remedy -- "point LLM_GATEWAY_MODEL at a thinking-capable group" -- is not actionable on its own. The gateway registers model groups by role in the LiteLLM DB, not in git (litellm-config.yaml deliberately carries no model_list), so there is nothing to read and the names give nothing away: this key serves chat-default, tools-default, embed-default, vision-default and vision-accurate. So probe the rest of the allow-list and report. Two distinct verdicts, which matter because they need different people to do different things: - at least one passes -> name them; the fix is to repoint the secret. - none pass -> say plainly that repointing cannot fix it, and that the gateway needs a thinking-capable group registered and in the key's scope. Embedding models are skipped and the list is capped at 12. This runs only on the failure path, so a healthy run pays nothing for it. Verified against a mock gateway serving /v1/models plus per-model thinking behaviour: with two capable models it names exactly those two (skipping both the embedding model and the already-failed pin); with none capable it emits the "not fixable by repointing" error; and a healthy pin still exits 0 without probing anything. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DobwJ8YTnfowwqwhspU1kQ --- .github/workflows/skill-evals.yml | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/.github/workflows/skill-evals.yml b/.github/workflows/skill-evals.yml index b1a60f5..8f81ba4 100644 --- a/.github/workflows/skill-evals.yml +++ b/.github/workflows/skill-evals.yml @@ -340,6 +340,35 @@ jobs: if [ "$code" != "200" ]; then echo "::error::'$LLM_GATEWAY_MODEL' serves a plain request but FAILS one with extended thinking enabled (HTTP $code). Claude Code sends thinking on its very first call, so every case would fail this way. Point LLM_GATEWAY_MODEL at a model group that supports thinking; a small local model generally does not. It can still serve the judge -- set JUDGE_MODEL to it separately." sed 's/^/ /' "$RUNNER_TEMP/body.json" >&2 || true + + # "Pick a different model" is not actionable without knowing which + # ones would work, and the answer is not guessable from the names -- + # this gateway registers its groups by role (chat-default, + # tools-default, vision-*) in the LiteLLM DB, not in git, so there is + # nothing to read. Probe the rest of the allow-list and say. Runs + # only on this failure path, so the healthy case pays nothing. + echo "probing the rest of the allow-list for a thinking-capable model:" + others="$(curl -fsS --max-time 30 \ + -H "Authorization: Bearer $LLM_GATEWAY_KEY" \ + "$root/v1/models" 2>/dev/null \ + | jq -r '.data[]?.id' | grep -v '^embed' | head -12 || true)" + found="" + for m in $others; do + [ "$m" = "$LLM_GATEWAY_MODEL" ] && continue + body='{"model":"'"$m"'","max_tokens":1200,"thinking":{"type":"enabled","budget_tokens":1024},"messages":[{"role":"user","content":"ping"}]}' + c="$(probe "$body")" + if [ "$c" = "200" ]; then + echo " PASS $m" + found="$found $m" + else + echo " fail $m (HTTP $c)" + fi + done + if [ -n "$found" ]; then + echo "::error::set LLM_GATEWAY_MODEL to one of:$found" + else + echo "::error::no model in this key's allow-list can serve an extended-thinking request. This is not fixable by repointing the secret -- the gateway needs a thinking-capable model group registered (LiteLLM admin UI), and the virtual key needs it in scope. Embedding models were skipped." + fi exit 1 fi echo "thinking request: 200 -- the agent route can serve this model."