bug(openai-compatible): historical image serialization failures across chat roles and completion mode #387

Summary

Track the historical multimodal serialization failures investigated for COM-13644 and the remaining work needed to verify that community report.
Merged fixes: #386 for chat conversion and #411 for explicit completion-mode input handling.
SDK 0.11.0 publication is tracked in #413; #412 remains a draft response-handling follow-up.
This issue stays open for plugin delivery and verification of the original report in Dify 1.19.0.

The SDK's OpenAI-compatible chat converter historically converted TEXT/IMAGE/VIDEO content parts only for user messages.
For system, assistant, and configured tool messages it passed Python content objects into the HTTP payload, causing TypeError: Object of type ImagePromptMessageContent is not JSON serializable before the request reaches the server.
vLLM inherits this converter; the official OpenAI API Compatible plugin delegates non-user chat messages to it.

A separate existing path produces the same error: mode=completion places prompt_messages[0].content directly into prompt, bypassing the chat converter even when the image is in a user message.
The error text alone therefore does not identify which path caused the community report.

Historical evidence

• The user-only conversion is present in the original Dify Model Runtime implementation from January 2024 and the SDK implementation from September 2024.
• langgenius/dify-official-plugins#1118 contains a human analysis identifying system image variables as the trigger for the same exception.
• Dify 1.10.0 file-variable construction preserves the configured message role; ordinary Vision/sys_files inputs take a separate user-message path.
• Published SDK 0.7.4, 0.10.0, and 0.10.2 contain the affected code paths.
  Function-level comparisons covered 48 role/mode/stream combinations, using exact functions extracted from those wheels with current SDK entities/helpers; these were not full historical runtime environments.

Reproduction

The following reproduces both paths with released SDK 0.10.2 and requires no API key or running model server.
Save it as repro.py and run uv run --isolated --no-project --with dify-plugin==0.10.2 python repro.py.
The HTTP method is mocked in the completion check, so no request is sent.

import json
from unittest.mock import patch

from dify_plugin import OAICompatLargeLanguageModel
from dify_plugin.entities.model.message import (
    AssistantPromptMessage,
    ImagePromptMessageContent,
    SystemPromptMessage,
    ToolPromptMessage,
    UserPromptMessage,
)

llm = OAICompatLargeLanguageModel([])
image = ImagePromptMessageContent(
    url="https://example.com/image.png", format="png", mime_type="image/png"
)
credentials = {"function_calling_type": "tool_call"}
messages = [
    UserPromptMessage(content=[image]),
    SystemPromptMessage(content=[image]),
    AssistantPromptMessage(content=[image]),
    ToolPromptMessage(content=[image], tool_call_id="call-1"),
]
for message in messages:
    try:
        json.dumps(llm._convert_prompt_message_to_dict(message, credentials))
        print(message.role.value, "serialized")
    except TypeError as error:
        print(message.role.value, type(error).__name__, str(error))

with patch("dify_plugin.interfaces.model.openai_compatible.llm.requests.post") as post:
    try:
        llm._generate(
            "model",
            {"mode": "completion", "endpoint_url": "https://example.com/v1/"},
            [messages[0]],
            {},
            stream=False,
        )
    except TypeError as error:
        print("completion/user", type(error).__name__, str(error))
    post.assert_not_called()

Expected behavior: encode supported chat content into the provider's wire format while preserving roles and metadata.
If an input modality is unsupported by a configured API mode, handle that explicitly at the mode/capability boundary rather than leaking a Python JSON serialization exception.
This does not imply that the legacy completions API accepts Chat Completions image arrays, or that every chat endpoint/template supports images in every role.

Community report and unresolved attribution

Reported environment: Dify 1.10.0, vLLM server 0.26.0, vLLM plugin 0.2.4, OpenAI API Compatible 0.0.64, model reported as Qwen3.8-27B.
The reporter confirmed text calls work and image calls fail through both plugins.
We have not obtained the original attachment, full traceback, workflow DSL, configured mode, or actual message roles for either failing call.

The publicly verified vLLM plugin artifact is 0.2.3, which pins SDK 0.7.4; the reported 0.2.4 artifact and installation source remain unverified.
The official compatible 0.0.64 source specifies SDK >=0.10.0 and locks 0.10.2.
These inspected sources do not establish the exact dependencies inside the reporter's installed packages.

The pinned official Qwen3.8-27B chat template rejects system images/video.
The reporter's actual model revision, template, and launch configuration are unknown, and JSON serialization success alone cannot establish successful model inference.

Current fix and follow-up — 2026-11-02

#386 merged on September 11 and converts supported chat content after preserving message roles and metadata.
#411 merged on October 31 and handles completion input at the mode boundary: text blocks are joined in order, while media raises InvokeError through public invocation before HTTP.
It also fixes missing-usage estimates and forwards tools through response hooks.
Its Python 3.12/3.13 CI passed 1,750 tests on each version.

#412 remains a draft follow-up for complete tool-schema estimation, empty finish markers, and daemon-decodable tool-only response text.
Its current checks pass, including Python 3.12/3.13 CI; local validation reports 1,769 tests and a passing build.
These are implementation and controlled transport checks, with original deployment verification still outstanding.

• Merge the shared chat converter fix in fix(openai-compatible): serialize image content across message roles #386.
• Implement and test explicit completion-mode modality handling in fix(model)!: validate completion input and correct fallback usage #411.
• Review and merge fix(model): validate responses and stabilize token estimates #412 before the planned SDK release.
• Publish SDK 0.11.0 through chore(release): publish Plugin SDK v0.11.0 for Dify 1.19 #413; chore(release): prepare v0.11.0 #391 is merged release preparation and the current public version remains 0.10.2.
• Reconcile fix(agent)!: preserve model replay with complete tool calls dify-official-plugins#4034 and fix(models)!: align provider usage with SDK 0.11 dify-official-plugins#4156, replace review pins with the published SDK, and rebuild/verify the affected plugin archives.
• Obtain each plugin's original error/traceback, mode and Vision settings, minimal workflow or steps, and installed plugin source/version.
• If needed, compare a fresh Chat/user-image request through both plugins with a direct request to the same vLLM endpoint using the same model and image.
• Verify the original scenario with the updated packages and deployed model/template before declaring COM-13644 resolved.

Target product release: Dify 1.19.0.
Successful serialization does not establish that the selected model/template accepts an image in that role.
Keep API keys, authorization headers, private endpoints, and sensitive workflow/image data out of the public issue.