[nidcode]

Summary

Fixes #3940.

Supersedes #3948, which was closed for merge conflicts after the dependency refresh (#4106–#4111). Rebased on the latest main; the code is unchanged apart from the version bump.

Image embedding only works against vLLM servers today. _invoke_multimodal() always sends the vLLM chat embeddings extension ({"messages": [...]}), and providers that implement the OpenAI embeddings spec reject the request before they ever look at the image:

• Gemini embedding endpoints: Invalid JSON payload received. Unknown name "messages": Cannot find field.
• OpenAI-compatible gateways such as LiteLLM: /embeddings: Missing required parameter: 'input'.

Those providers do accept images, but as data: URI elements of the standard input array.

This adds a multimodal_embedding_format credential to choose the wire format:

• chat_embeddings — the current vLLM payload, and the default, so existing deployments are untouched
• input_array — the standard {"input": [...]} payload carrying the image as a data URI

Why a credential rather than auto-detection

• Falling back on a 400 would add a failed round trip per document during indexing, and some gateways answer 200 while quietly dropping the image, so the response status is not a reliable signal.
• Absorbing gateway differences through an explicit credential is the existing pattern in this plugin (encoding_format, user_identity_support, compatibility_mode).

I kept chat_embeddings as the default to avoid silently changing behaviour for vLLM users, but I am happy to flip it if you would rather have the standard payload be the default.

Why one image per request

Providers reached this way may fuse several input elements into a single embedding instead of returning one per element. Measured through a LiteLLM gateway (v1.98.0) in front of Vertex AI:

model	3 text inputs	embeddings returned
vertex_ai/gemini-embedding-2-preview	3	1
vertex_ai/gemini-embedding-001	3	3
vertex_ai/text-embedding-004	3	3

Appending image data URIs to an existing batch would therefore return fewer embeddings than inputs, and _embed_in_batches() would shift every following vector or raise IndexError with no indication of the cause. Each image is sent on its own request, and the response is now verified to carry exactly one embedding rather than trusted.

This costs no extra round trips in practice: get_customizable_model_schema() in the SDK pins MAX_CHUNKS to 1, so the text path already sends one input per request.

Changes

• provider/openai_api_compatible.yaml: new multimodal_embedding_format select, shown only for text-embedding with vision_support: support
• models/text_embedding/text_embedding.py:
  • dispatch on the selected format in _embed_in_batches()
  • new _embed_multimodal_via_input_array()
  • request building and usage accounting shared with the text path factored into _post_embeddings() / _accumulate_usage() instead of duplicated
  • reject images whose format cannot be detected, instead of emitting data:image;base64,... — a data URI with no subtype, which strict providers reject with an unrelated-looking error (separate commit)
• tests/test_text_embedding.py: 5 new tests
• manifest.yaml: 0.0.73 → 0.0.75 (0.0.74 is already claimed by fix(openai_api_compatible): wire up the orphaned stream_function_calling credential #4068, fix(openai_api_compatible): accept gateways that report usage as input/output tokens #4131, fix(openai_api_compatible): carry video and audio attachments on the Responses path #4141, fix(openai_api_compatible): honor top_n=0 in text rerank #4142 and fix(openai_api_compatible): restore structured output capability #4146)

_embed_multimodal_via_chat() is untouched.

Release Notes

Added a "Multimodal embedding request format" option so image embeddings can be sent as a standard input array for providers that reject the vLLM chat-embeddings payload.

Change Type

• Documentation / non-plugin change
• Non-LLM plugin (tools, extensions, datasource, etc.)
• LLM plugin

Areas affected by this change (check all that apply)

• Message flow (system messages, user ↔ assistant turn-taking)
• Tool interaction flow (multi-round usage, Agent App and Agent Node)
• Multimodal input (images, PDFs, audio, video, etc.)
• Multimodal output (images, audio, video, etc.)
• Structured output (JSON, XML, etc.)
• Token consumption metrics
• Other LLM functionality (reasoning, grounding, prompt caching, etc.)
• New models / model parameter fixes

Screenshots / Videos

Indexing the same PDF with images into a knowledge base, gemini-embedding-2-preview behind a LiteLLM gateway:

Before	After
error: /embeddings: Missing required parameter: 'input'.	completed, 576 segments

Version

• Bumped top-level version in manifest.yaml (not the one under meta)
• dify_plugin>=0.3.0,<0.11.0 is declared in pyproject.toml and locked in uv.lock

pyproject.toml already declares dify_plugin>=0.9.0 and is unchanged by this PR.

Testing

• Local deployment — Dify Enterprise 3.12.1, self hosted on EKS
• SaaS (cloud.dify.ai)

uv run pytest tests -q → all green, including 5 new cases: image sent as a data URI in input, one request per image with order preserved across mixed text/image input, endpoint_model_name and encoding_format propagation, usage accumulation, a fused response rejected, the default staying on chat_embeddings for unset/explicit/unknown values, and an undetectable image format reported instead of sent.

End to end against a knowledge base, indexing the same PDF containing images, with gemini-embedding-2-preview served through a LiteLLM gateway in front of Vertex AI:

	before (0.0.67)	after (this PR)
multimodal_embedding_format	n/a	input_array
indexing status	error	completed
segments	0	576
error	/embeddings: Missing required parameter: 'input'.	none

The same dataset and document were reused, so the only difference is the wire format. With the credential left unset the run reproduces the original failure, confirming the default path is unchanged.