[nidcode]

Summary

Fixes #3940.

Image embedding only works against vLLM servers today. _invoke_multimodal() always sends the vLLM chat embeddings extension ({"messages": [...]}), and providers that implement the OpenAI embeddings spec reject the request before they ever look at the image:

• Gemini embedding endpoints: Invalid JSON payload received. Unknown name "messages": Cannot find field.
• OpenAI-compatible gateways such as LiteLLM: /embeddings: Missing required parameter: 'input'.

Those providers do accept images, but as data: URI elements of the standard input array.

This adds a multimodal_embedding_format credential to choose the wire format:

• chat_embeddings — the current vLLM payload, and the default, so existing deployments are untouched
• input_array — the standard {"input": [...]} payload carrying the image as a data URI

Why a credential rather than auto-detection

• Falling back on a 400 would add a failed round trip per document during indexing, and some gateways answer 200 while quietly dropping the image, so the response status is not a reliable signal.
• Absorbing gateway differences through an explicit credential is the existing pattern in this plugin (encoding_format, user_identity_support, compatibility_mode).

I kept chat_embeddings as the default to avoid silently changing behaviour for vLLM users, but I am happy to flip it if you would rather have the standard payload be the default.

Why one image per request

Providers reached this way may fuse several input elements into a single embedding instead of returning one per element. Measured through a LiteLLM gateway (v1.98.0) in front of Vertex AI:

model	3 text inputs	embeddings returned
vertex_ai/gemini-embedding-2-preview	3	1
vertex_ai/gemini-embedding-001	3	3
vertex_ai/text-embedding-004	3	3

Appending image data URIs to an existing batch would therefore return fewer embeddings than inputs, and _embed_in_batches() would shift every following vector or raise IndexError with no indication of the cause. Each image is sent on its own request, and the response is now verified to carry exactly one embedding rather than trusted.

This costs no extra round trips in practice: get_customizable_model_schema() in the SDK pins MAX_CHUNKS to 1, so the text path already sends one input per request.

Changes

• provider/openai_api_compatible.yaml: new multimodal_embedding_format select, shown only for text-embedding with vision_support: support
• models/text_embedding/text_embedding.py:
  • dispatch on the selected format in _embed_in_batches()
  • new _embed_multimodal_via_input_array()
  • request building and usage accounting shared with the text path factored into _post_embeddings() / _accumulate_usage() instead of duplicated
  • reject images whose format cannot be detected, instead of emitting data:image;base64,... — a data URI with no subtype, which strict providers reject with an unrelated-looking error (separate commit)
• tests/test_text_embedding.py: 5 new tests
• manifest.yaml: 0.0.67 → 0.0.68

_embed_multimodal_via_chat() is untouched.

Release Notes

Added a "Multimodal embedding request format" option so image embeddings can be sent as a standard input array for providers that reject the vLLM chat-embeddings payload.

Change Type

• Documentation / non-plugin change
• Non-LLM plugin (tools, extensions, datasource, etc.)
• LLM plugin

Areas affected by this change (check all that apply)

• Message flow (system messages, user ↔ assistant turn-taking)
• Tool interaction flow (multi-round usage, Agent App and Agent Node)
• Multimodal input (images, PDFs, audio, video, etc.)
• Multimodal output (images, audio, video, etc.)
• Structured output (JSON, XML, etc.)
• Token consumption metrics
• Other LLM functionality (reasoning, grounding, prompt caching, etc.)
• New models / model parameter fixes

Version

• Bumped top-level version in manifest.yaml (not the one under meta)
• dify_plugin>=0.3.0,<0.11.0 is declared in pyproject.toml and locked in uv.lock

pyproject.toml already declares dify_plugin>=0.9.0 and is unchanged by this PR.

Testing

• Local deployment — Dify Enterprise 3.12.1, self hosted on EKS
• SaaS (cloud.dify.ai)

uv run pytest tests -q → all green, including 5 new cases: image sent as a data URI in input, one request per image with order preserved across mixed text/image input, endpoint_model_name and encoding_format propagation, usage accumulation, a fused response rejected, the default staying on chat_embeddings for unset/explicit/unknown values, and an undetectable image format reported instead of sent.

End to end against a knowledge base, indexing the same PDF containing images, with gemini-embedding-2-preview served through a LiteLLM gateway in front of Vertex AI:

	before (0.0.67)	after (this PR)
multimodal_embedding_format	n/a	input_array
indexing status	error	completed
segments	0	576
error	/embeddings: Missing required parameter: 'input'.	none

The same dataset and document were reused, so the only difference is the wire format. With the credential left unset the run reproduces the original failure, confirming the default path is unchanged.

[crazywoola]

Thank you for your contribution, @nidcode! This PR currently has merge conflicts with main, so we are closing it for now. Please update your branch against the latest main and resolve the conflicts — note that the consolidated dependency refresh (#4106–#4111) has bumped plugin manifest.yaml versions on main, so please bump models/openai_api_compatible/manifest.yaml to the next unused version when you rebase. You are welcome to open a new PR with the updated changes and link back to this one. We appreciate your time and effort!

[nidcode]

Rebased on the latest main and reopened as #4175 (version bumped to 0.0.75).