[PeihuaChen]

Summary

No existing issue tracks this bug; the failure was reproduced and verified directly against a self-hosted vLLM endpoint (details below).

Fixes multimodal rerank in the openai_api_compatible plugin when the query or the documents are images (verified against vLLM v0.31.0 serving a Qwen3-VL-Reranker model via /v1/rerank).

Two problems in _invoke_multimodal:

1. Image query is inlined into the text query. The plugin assumed Qwen3-VL-Reranker does not support image queries and sent "query": "[Image: <base64>]". A real image is tens/hundreds of KB of base64, so the prompt exceeds the context window and vLLM answers HTTP 400:

   This model's maximum context length is 32768 tokens. However, ... your prompt contains at least 32769 input tokens.

   vLLM actually accepts an image query in the same ScoreMultiModalParam format as documents ({"content": [{"type": "image_url", ...}]}), so it is now sent that way.

2. Raw base64 document images are silently dropped. Dify passes images to the plugin as raw base64 (no data: prefix). _validate_image_url only allows http(s) / data:image, so these were turned into empty {"content": []} documents (and sending raw base64 to vLLM directly yields The URL must be either a HTTP, data or file URL).

Changes (all in models/openai_api_compatible):

• Send an image query as ScoreMultiModalParam; text queries are still a plain string.

• New _normalize_image_url: wraps raw base64 into data:image/<fmt>;base64,... after detecting the format from magic bytes (png/jpeg/gif/webp/bmp). URLs and existing data URIs are untouched; anything that is not a known image stays as-is and is still rejected by _validate_image_url.

• Raise InvokeBadRequestError for an invalid/unsupported image query instead of calling the endpoint.

• SSRF protections are unchanged (file://, localhost and private IPs are still blocked).

• Added tests/test_rerank_multimodal_image.py.

Release Notes

• Fix: multimodal rerank with an image query no longer fails with HTTP 400 on vLLM/Qwen3-VL-Reranker (the image is now sent as an image instead of being embedded as text).

• Fix: images from Dify knowledge-base documents (raw base64) are now actually passed to the rerank model instead of being dropped.

Change Type

• Documentation / non-plugin change

• Non-LLM plugin (tools, extensions, datasource, etc.)

• LLM plugin

Screenshots / Videos

Dify knowledge base hit testing with an image query against a vLLM-hosted Qwen3-VL reranker (/v1/rerank):

| Before | After |

| ------ | ----- |

| Rerank API request failed (HTTP 400) (vLLM: maximum context length is 32768 tokens ... at least 32769 input tokens) | Request succeeds; identical image scores highest (0.9583 vs 0.5325 for a different image and 0.0554 for unrelated text) |

LLM Plugin Checklist

Areas affected by this change (check all that apply)

• Message flow (system messages, user ↔ assistant turn-taking)

• Tool interaction flow (multi-round usage, Agent App and Agent Node)

• Multimodal input (images, PDFs, audio, video, etc.)

• Multimodal output (images, audio, video, etc.)

• Structured output (JSON, XML, etc.)

• Token consumption metrics

• Other LLM functionality (reasoning, grounding, prompt caching, etc.)

• New models / model parameter fixes

Version

• Bumped top-level version in manifest.yaml (not the one under meta): 0.0.69 → 0.0.71 (rebased on latest main, which is already at 0.0.69; 0.0.70 is claimed by fix(openai_api_compatible): strip think tags split across stream chunks #4066)
• dify_plugin>=0.3.0,<0.11.0 is declared in pyproject.toml and locked in uv.lock (dependencies are not touched by this PR)

Testing

• Local deployment — Dify version: 1.14.2 (self-hosted, vLLM v0.31.0 serving a Qwen3-VL reranker). Installed the packaged plugin and ran knowledge-base hit testing with image queries.

• SaaS (cloud.dify.ai)

Also:

• Existing tests/test_rerank_authorization.py (16 tests) and the new tests/test_rerank_multimodal_image.py pass (26 tests total). Tests were run with Python 3.11 and dify_plugin==0.7.4, not the Python 3.12 / uv.lock environment, so please let CI confirm.

• Called OpenAIRerankModel._invoke_multimodal directly against the vLLM endpoint with ~100 KB base64 images: image query + image docs, image query + text docs, text query + image docs and text query + text docs all return 200; an invalid image query raises InvokeBadRequestError.

[crazywoola]

Hi @PeihuaChen, thanks for fixing the image query and raw base64 images in multimodal rerank!

Heads-up about models/openai_api_compatible/manifest.yaml: the consolidated dependency refresh (#4106–#4111) bumped it to 0.0.69 on main, and several PRs are currently open for this plugin from different authors (#3795, #3869, #4060, #4066, #4068). Please rebase on latest main and bump the version to the next unused one so the releases don't collide. If there's a related issue for this fix, linking it in the description would help too.

Thanks again! 🙏

[PeihuaChen]

Thanks for the heads-up! Rebased on latest main (37722e8) and bumped manifest.yaml to 0.0.71, since 0.0.69 is already on main and 0.0.70 is taken by #4066. The PR is still a single commit touching manifest.yaml, models/rerank/rerank.py and the new tests/test_rerank_multimodal_image.py.

I couldn't find an existing issue for this bug. I hit it and reproduced it directly against a self-hosted vLLM Qwen3-VL reranker, and I've added a note about that to the description.