Skip to content

[Bug] Admin "Test connection" for OpenRouter always fails: LLMService.test_connection() does not exist #280

Description

@guofeng201507

Describe the bug

The admin endpoint POST /api/settings/test-connection (for service=openrouter) can never succeed. It calls LLMService.test_connection(), but that method does not exist on LLMService, so the handler raises AttributeError and returns a generic failure:

backend_api_python/app/routes/settings.py
    result = llm.test_connection()   # -> AttributeError
ERROR app.routes.settings: Connection test failed: 'LLMService' object has no attribute 'test_connection'

Response: {"code":0,"msg":"Test failed: 'LLMService' object has no attribute 'test_connection'"}

This is not a regression from a recent change — the method has never existed:

$ git log --oneline -S'def test_connection' -- backend_api_python/app/services/llm.py
# (no commits: the definition was never added)

Steps to reproduce

  1. Settings -> AI / LLM: select OpenRouter and save a valid API key.
  2. Click Test OpenRouter connection.
  3. The request fails with the AttributeError above, regardless of whether the key is valid.

Expected behaviour

A connectivity probe that actually validates the configured provider + credential and reports a meaningful reason on failure.

Impact

  • Operators cannot verify an LLM key from the UI; the button is effectively dead.
  • Because the handler short-circuits with an AttributeError, a genuinely bad key and a broken endpoint look identical from the UI.

Related: aiGenerate silently falls back to a template

POST /api/indicator/aiGenerate swallows model failures and returns its built-in template with HTTP 200. That template passes validation, so debug.returned_candidate is "initial" and the client cannot distinguish a failed model call from a successful response — the UI reports success. Suggest surfacing the fallback reason (currently only in edit_plan["error"]) in the streamed debug frame.

Suggested fix (from a local patch)

Add LLMService.test_connection() and validate against an authenticated endpoint. Two traps worth noting, both verified while fixing this against OpenRouter:

  • OpenRouter's GET /api/v1/models is public — it returns 200 even without a key, so it cannot validate credentials. Use GET /api/v1/auth/key instead.
  • OpenRouter provisioning / management keys pass /auth/key (200) but are rejected by chat completions (401, "User not found"). Detecting is_provisioning_key / is_management_key in the /auth/key payload gives an actionable error instead of a misleading "reachable".

Happy to open a PR with the patch if that is useful.

Environment

  • QuantDinger v5.5.4
  • Backend: backend_api_python (Flask), provider openrouter

Activity

  1. guofeng201507 commented on Oct 9, 2026

    @guofeng201507
    Author

    Additional finding while fixing this locally: the static fallback model is hardcoded and cannot be overridden.

    call_llm_api builds its candidate list as:

    [requested model] + [configured default model] + [PROVIDER_CONFIGS[p]["fallback_model"]]
    

    The third entry is a literal constant (e.g. openai/gpt-4o-mini for OpenRouter, app/services/llm.py) and there is no setting that can change it.

    Two consequences:

    1. On accounts where the built-in fallback is not usable, the "safety net" makes diagnosis worse. Observed on OpenRouter: the primary model failed with 400, the fallback then also failed with 400 "Provider returned error", and the raised exception concatenated both attempts. The operator has to parse a two-model error message to find the real cause — and, before fix: repair LLM test-connection and make silent model/template fallbacks visible #281, had no way to see that a different model was even being tried.
    2. There is no supported way to point the fallback at a model the account can actually use.

    Suggestion: resolve the fallback through <PROVIDER>_FALLBACK_MODEL (env, plus a settings-schema entry so it is editable in the AI/LLM page), keeping the current constant as the default when unset — i.e. purely additive, no behaviour change for existing deployments.

    I implemented this locally (~2 files, 14 lines) and exposed OPENROUTER_FALLBACK_MODEL. Worth noting it also dedupes naturally: when the override equals the configured default, the candidate list collapses to a single model, so a genuine failure surfaces immediately instead of being masked by a retry against a model that cannot serve the request.

    #281 makes the model that actually answered visible via debug.generation_model / generation_model_fallback; this configurability point is the natural follow-up. Happy to open a separate PR for it if that is welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions