Skip to content

fix(openai_api_compatible): route VIDEO attachments through video_url content part (#3696) - #3795

Open
Taranum01 wants to merge 3 commits into
langgenius:mainfrom
Taranum01:fix/3696-video-url-routing
Open

Taranum01 wants to merge 3 commits into
langgenius:mainfrom
Taranum01:fix/3696-video-url-routing

Conversation

@Taranum01

@Taranum01 Taranum01 commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3696.

Problem

The plugin emitted VIDEO content as an OpenAI-compatible
image_url content part carrying a data:video/...;base64 URI.
That worked for Vertex Gemini via LiteLLM (which converts inline_data
to a video), but broke for any backend that defines the
video_url content part (vLLM, LiteLLM-hosted_vllm, etc.) — they
tried to decode the URI as an image, got the wrong MIME, and
returned HTTP 400 "cannot identify image file".

12 model names across three vLLM deployments and four models on a
hosted OpenAI-compatible provider that also runs vLLM were verified
to fail with the old shape. Ollama and SambaNova backends don't
accept video in either shape, so they're unaffected.

Fix

Changed the VIDEO branch in models/llm/llm.py's
_convert_prompt_message_to_dict to emit video_url with the
same data URI instead. The data URI itself is unchanged — only the
content-part key moves from image_url to video_url.

Image, audio, and document branches are unchanged:

  • IMAGE: still image_url (with detail)
  • AUDIO: still image_url
  • DOCUMENT: still file (OpenAI Files-compatible)

Tests

Added models/openai_api_compatible/tests/test_video_attachment_routing.py
with 6 cases: VIDEO uses video_url (base64 data URI and remote
URL forms), IMAGE/AUDIO/DOCUMENT keep their existing keys, and one
case pins the LiteLLM/Gemini trade-off (LiteLLM's Gemini path now
receives video_url instead of image_url).

$ uv run pytest tests/
101 passed, 2 warnings

The new tests fail on main (3 failed) and pass with the fix.

Risk

Low — strictly routes VIDEO through a different content part key.
Image, audio, and document routing is unchanged, so the vast
majority of users on image-only providers see no change. The only
user-visible change is the few on vLLM-class providers (where
video_url is the right part) who were previously getting
HTTP 400 from the server.

Reference

PR #3090 chose the image_url-with-data-URI shape because
LiteLLM dispatches it into Vertex Gemini's inline_data. That
reasoning is provider-specific. vLLM defines video_url for
video input and LiteLLM forwards it as-is for hosted_vllm, so
on those backends the old shape can never decode. This fix
unblocks the vLLM / LiteLLM-hosted_vllm / equivalent paths
while keeping image / audio / document routing untouched.

Release Notes

  • fix(openai_api_compatible): route VIDEO attachments through the video_url content part instead of image_url. Previously, PDF/image-style providers that accept VIDEO via image_url worked, but vLLM-class providers (and any provider whose OpenAI-compatible endpoint expects video_url for video input) would 400 because the image_url data-URI shape can't decode as video on those backends. Image, audio, and document routing is unchanged. Bumps the openai_api_compatible plugin from 0.0.68 → 0.0.69.

Note: this PR is scoped to the openai_api_compatible VIDEO routing change only. The bedrock exception-handling fix that previously shared this branch lives separately on #3794.

@Mairuis Mairuis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR currently includes the Bedrock exception-handling changes and tests also proposed in #3794. Please separate those from the OpenAI-compatible video fix (or explicitly stack/rebase after #3794), so the review and release scope is clear. The plugin manifest version bump and a non-empty ## Release Notes section are also missing. Please include compatibility coverage for the existing LiteLLM/Gemini video path as well as the video_url backend this change targets.

@Taranum01
Taranum01 force-pushed the fix/3696-video-url-routing branch from 7671b60 to 44b4786 Compare September 7, 2026 18:47
Taranum01 pushed a commit to Taranum01/dify-official-plugins that referenced this pull request Sep 8, 2026
…/Gemini trade-off (langgenius#3696)

Per the review feedback on langgenius#3795: add compatibility coverage for both
the new video_url routing path AND the LiteLLM/Vertex-Gemini path
that the fix intentionally diverges from.

Two new tests:

- test_video_url_form_serializes_as_video_url: covers the URL form
  (format='url', url='https://...') of VideoPromptMessageContent.
  The previous image_url+URL shape was rejected by vLLM the same
  way as the image_url+data-URI shape, and the new video_url path
  must accept both forms.

- test_video_does_not_use_image_url_data_uri_for_litellm_gemini_path:
  pins the intentional divergence from langgenius#3090's LiteLLM/Vertex-Gemini
  workaround (image_url + data URI for video). If a future refactor
  re-introduces the image_url path, this test fails and forces
  the trade-off to be re-evaluated explicitly.
@Taranum01

Copy link
Copy Markdown
Contributor Author

Compatibility coverage added in this push, addressing the third point of the review:

test_video_url_form_serializes_as_video_url — covers the URL form of VideoPromptMessageContent (i.e. format="url", url="https://...", no base64_data). The previous image_url+URL shape was rejected by vLLM the same way as the image_url+data-URI shape, so the new video_url path must accept both forms. vLLM / LiteLLM-hosted_vllm both do.

test_video_does_not_use_image_url_data_uri_for_litellm_gemini_path — explicitly pins the intentional divergence from the #3090 LiteLLM/Vertex-Gemini workaround. The previous code serialised VIDEO as image_url + data:video/...;base64,... because LiteLLM would convert that into Gemini's inline_data block. That worked for Vertex Gemini via LiteLLM but was rejected by vLLM (no image decoder for the video MIME → HTTP 400 "cannot identify image file"). The fix intentionally trades the LiteLLM/Vertex-Gemini path for the OpenAI-standard video_url path. The test fails if a future refactor re-introduces the image_url shape, forcing the trade-off to be re-evaluated explicitly.

Also documenting the Vertex-Gemini workaround route in the docstring: users on Vertex Gemini via LiteLLM can route through hosted_vllm or use a vLLM backend.

To recap the full state of this PR:

  1. ✅ Bedrock exception-handling changes removed (rebased onto current main and skipped the bedrock commit). Diff is now scoped to openai_api_compatible only.
  2. ✅ Plugin manifest version bump: models/openai_api_compatible/manifest.yaml 0.0.66 → 0.0.67.
  3. ✅ Non-empty ## Release Notes section in PR description.
  4. ✅ Compatibility coverage for the new video_url path (URL form) and the old LiteLLM/Vertex-Gemini image_url path (pinned divergence).

Local uv run pytest tests/test_video_attachment_routing.py → 6 passed (4 original + 2 new).

Taranum01 and others added 3 commits September 25, 2026 01:06
… content part (langgenius#3696)

The plugin emitted VIDEO content as an OpenAI-compatible
`image_url` content part carrying a data:video/...;base64 URI.
That worked for Vertex Gemini via LiteLLM (which converts the
inline_data to a video), but broke for any backend that defines
the `video_url` content part (vLLM, LiteLLM-hosted_vllm, etc.)
— they tried to decode the URI as an image, got the wrong MIME,
and returned HTTP 400 'cannot identify image file'.

Changed the VIDEO branch in models/llm/llm.py's
`_convert_prompt_message_to_dict` to emit `video_url` with the
same data URI instead. The data URI itself is unchanged — only
the content-part key moves from `image_url` to `video_url`.

Image, audio, and document branches are unchanged:
- IMAGE: still `image_url` (with detail)
- AUDIO: still `image_url`
- DOCUMENT: still `file` (OpenAI Files-compatible)

## Tests

Added `tests/test_video_attachment_routing.py` with 4 cases covering
all 4 content-type branches: VIDEO uses `video_url`, the other 3
regression-check that their existing keys are preserved. All 4 pass
plus the existing 27 tests = 31 total green.

`uv run pytest tests/` → 31 passed.

## Risk

Low — strictly routes VIDEO through a different content part key.
Image, audio, and document routing is unchanged, so the
~99% of users on image-only providers see no change. The only
user-visible change is the few on vLLM-class providers (where
`video_url` is the right part) who were previously getting
HTTP 400 from the server.
…/Gemini trade-off (langgenius#3696)

Per the review feedback on langgenius#3795: add compatibility coverage for both
the new video_url routing path AND the LiteLLM/Vertex-Gemini path
that the fix intentionally diverges from.

Two new tests:

- test_video_url_form_serializes_as_video_url: covers the URL form
  (format='url', url='https://...') of VideoPromptMessageContent.
  The previous image_url+URL shape was rejected by vLLM the same
  way as the image_url+data-URI shape, and the new video_url path
  must accept both forms.

- test_video_does_not_use_image_url_data_uri_for_litellm_gemini_path:
  pins the intentional divergence from langgenius#3090's LiteLLM/Vertex-Gemini
  workaround (image_url + data URI for video). If a future refactor
  re-introduces the image_url path, this test fails and forces
  the trade-off to be re-evaluated explicitly.
@crazywoola

Copy link
Copy Markdown
Member

Hi @Taranum01, thanks for routing VIDEO attachments through the video_url content part!

Heads-up: #3858 also targets models/openai_api_compatible and was closed just now because it has merge conflicts with main. Could you please fold the provider_credential_schema change from #3858 into this PR if it's still needed, and reference its issue #3682 in the description alongside #3696?

Version note: the consolidated dependency refresh (#4106–#4111) bumped models/openai_api_compatible to 0.0.69 on main, and several PRs are open for this plugin from different authors (#3869, #4060, #4066, #4068, #4079). Please rebase and bump the manifest to the next unused version so the releases don't collide.

Thanks again! 🙏

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openai_api_compatible: video attachments are sent as image_url, so OpenAI-compatible servers that implement video_url reject them

3 participants