Repository navigation
Conversation
Mairuis
left a comment
There was a problem hiding this comment.
This PR currently includes the Bedrock exception-handling changes and tests also proposed in #3794. Please separate those from the OpenAI-compatible video fix (or explicitly stack/rebase after #3794), so the review and release scope is clear. The plugin manifest version bump and a non-empty ## Release Notes section are also missing. Please include compatibility coverage for the existing LiteLLM/Gemini video path as well as the video_url backend this change targets.
7671b60 to
44b4786
Compare
…/Gemini trade-off (langgenius#3696) Per the review feedback on langgenius#3795: add compatibility coverage for both the new video_url routing path AND the LiteLLM/Vertex-Gemini path that the fix intentionally diverges from. Two new tests: - test_video_url_form_serializes_as_video_url: covers the URL form (format='url', url='https://...') of VideoPromptMessageContent. The previous image_url+URL shape was rejected by vLLM the same way as the image_url+data-URI shape, and the new video_url path must accept both forms. - test_video_does_not_use_image_url_data_uri_for_litellm_gemini_path: pins the intentional divergence from langgenius#3090's LiteLLM/Vertex-Gemini workaround (image_url + data URI for video). If a future refactor re-introduces the image_url path, this test fails and forces the trade-off to be re-evaluated explicitly.
|
Compatibility coverage added in this push, addressing the third point of the review:
Also documenting the Vertex-Gemini workaround route in the docstring: users on Vertex Gemini via LiteLLM can route through To recap the full state of this PR:
Local |
… content part (langgenius#3696) The plugin emitted VIDEO content as an OpenAI-compatible `image_url` content part carrying a data:video/...;base64 URI. That worked for Vertex Gemini via LiteLLM (which converts the inline_data to a video), but broke for any backend that defines the `video_url` content part (vLLM, LiteLLM-hosted_vllm, etc.) — they tried to decode the URI as an image, got the wrong MIME, and returned HTTP 400 'cannot identify image file'. Changed the VIDEO branch in models/llm/llm.py's `_convert_prompt_message_to_dict` to emit `video_url` with the same data URI instead. The data URI itself is unchanged — only the content-part key moves from `image_url` to `video_url`. Image, audio, and document branches are unchanged: - IMAGE: still `image_url` (with detail) - AUDIO: still `image_url` - DOCUMENT: still `file` (OpenAI Files-compatible) ## Tests Added `tests/test_video_attachment_routing.py` with 4 cases covering all 4 content-type branches: VIDEO uses `video_url`, the other 3 regression-check that their existing keys are preserved. All 4 pass plus the existing 27 tests = 31 total green. `uv run pytest tests/` → 31 passed. ## Risk Low — strictly routes VIDEO through a different content part key. Image, audio, and document routing is unchanged, so the ~99% of users on image-only providers see no change. The only user-visible change is the few on vLLM-class providers (where `video_url` is the right part) who were previously getting HTTP 400 from the server.
…/Gemini trade-off (langgenius#3696) Per the review feedback on langgenius#3795: add compatibility coverage for both the new video_url routing path AND the LiteLLM/Vertex-Gemini path that the fix intentionally diverges from. Two new tests: - test_video_url_form_serializes_as_video_url: covers the URL form (format='url', url='https://...') of VideoPromptMessageContent. The previous image_url+URL shape was rejected by vLLM the same way as the image_url+data-URI shape, and the new video_url path must accept both forms. - test_video_does_not_use_image_url_data_uri_for_litellm_gemini_path: pins the intentional divergence from langgenius#3090's LiteLLM/Vertex-Gemini workaround (image_url + data URI for video). If a future refactor re-introduces the image_url path, this test fails and forces the trade-off to be re-evaluated explicitly.
ffc5158 to
b62f8d7
Compare
|
Hi @Taranum01, thanks for routing VIDEO attachments through the Heads-up: #3858 also targets Version note: the consolidated dependency refresh (#4106–#4111) bumped Thanks again! 🙏 |
Fixes #3696.
Problem
The plugin emitted VIDEO content as an OpenAI-compatible
image_urlcontent part carrying a data:video/...;base64 URI.That worked for Vertex Gemini via LiteLLM (which converts inline_data
to a video), but broke for any backend that defines the
video_urlcontent part (vLLM, LiteLLM-hosted_vllm, etc.) — theytried to decode the URI as an image, got the wrong MIME, and
returned HTTP 400 "cannot identify image file".
12 model names across three vLLM deployments and four models on a
hosted OpenAI-compatible provider that also runs vLLM were verified
to fail with the old shape. Ollama and SambaNova backends don't
accept video in either shape, so they're unaffected.
Fix
Changed the VIDEO branch in
models/llm/llm.py's_convert_prompt_message_to_dictto emitvideo_urlwith thesame data URI instead. The data URI itself is unchanged — only the
content-part key moves from
image_urltovideo_url.Image, audio, and document branches are unchanged:
image_url(with detail)image_urlfile(OpenAI Files-compatible)Tests
Added
models/openai_api_compatible/tests/test_video_attachment_routing.pywith 6 cases: VIDEO uses
video_url(base64 data URI and remoteURL forms), IMAGE/AUDIO/DOCUMENT keep their existing keys, and one
case pins the LiteLLM/Gemini trade-off (LiteLLM's Gemini path now
receives
video_urlinstead ofimage_url).The new tests fail on main (3 failed) and pass with the fix.
Risk
Low — strictly routes VIDEO through a different content part key.
Image, audio, and document routing is unchanged, so the vast
majority of users on image-only providers see no change. The only
user-visible change is the few on vLLM-class providers (where
video_urlis the right part) who were previously gettingHTTP 400 from the server.
Reference
PR #3090 chose the
image_url-with-data-URI shape becauseLiteLLM dispatches it into Vertex Gemini's
inline_data. Thatreasoning is provider-specific. vLLM defines
video_urlforvideo input and LiteLLM forwards it as-is for
hosted_vllm, soon those backends the old shape can never decode. This fix
unblocks the vLLM / LiteLLM-hosted_vllm / equivalent paths
while keeping image / audio / document routing untouched.
Release Notes
video_urlcontent part instead ofimage_url. Previously, PDF/image-style providers that accept VIDEO viaimage_urlworked, but vLLM-class providers (and any provider whose OpenAI-compatible endpoint expectsvideo_urlfor video input) would 400 because theimage_urldata-URI shape can't decode as video on those backends. Image, audio, and document routing is unchanged. Bumps the openai_api_compatible plugin from 0.0.68 → 0.0.69.Note: this PR is scoped to the openai_api_compatible VIDEO routing change only. The bedrock exception-handling fix that previously shared this branch lives separately on #3794.