Repository navigation
AI providers: generic binary content blocks and capability check by media type #506
Copy link
Copy link
Closed
Copy link
Labels
enhancementNew feature or requestNew feature or requestsecurity-reviewTouches security-critical code: elevated reviewTouches security-critical code: elevated review
Milestone
Description
Goal
Providers send binary content blocks generically by media type instead of by attachment kind, and the model capability check is keyed by media class. Behaviour for images and PDFs stays the same (ADR 0141, decision 5; changes ADR 0138, decision 4).
Context
crates/ai-providers/src/content.rshasContentPart::{Text, Image, Pdf},message_parts/group_parts(matching onMessageContent::Attachments),check_attachmentsandAttachmentSupport { images, pdf }per provider (anthropic_support,openai_responses_support,openai_compatible_support; local OpenAI-compatible servers such as Ollama viaVISION_HINTS).AiError::AttachmentUnsupported(AttachmentKind)(crates/core/src/ai/types.rs:833), codeAI_ATTACHMENT_UNSUPPORTED, UI text in all locales.- The sealing issue introduced
MessageContent::Sealed(SealedContent)with sealed binary blocks{ mime, filename, bytes }.
Requirements
ContentPart::{Text, Binary { mime, filename, base64 }}replacesImage/Pdf. Serialisation by media class: Anthropicimage/document, Chat Completionsimage_url/file, Responsesinput_image/input_file.- Parts are built from
Sealedcontent (fenced text, then its binaries) and, until the old model is removed, fromAttachmentsthrough the same generic path. - Capability table keyed by media class (
InputSupport::accepts(MediaClass)); same results as today for every provider/model. AiError::InputUnsupported(MediaClass)replacesAttachmentUnsupported; codeAI_ATTACHMENT_UNSUPPORTEDand UI text unchanged (decided by maintainer). The check runs over the whole history before any request.- Request logging: only counts and bytes for sealed content and binaries, never label or content.
Acceptance Criteria
- Recorded request fixtures for image and PDF on Anthropic, Responses and Chat Completions are unchanged.
- Every case of the existing capability table tests gives the same result through
accepts(MediaClass)(incl. unknown models,api.openai.comvs. other hosts, vision hints). - An image in the history for a text-only model →
InputUnsupported, codeAI_ATTACHMENT_UNSUPPORTED, no request (test per provider). -
Sealedcontent and legacyAttachmentsproduce identical blocks for the same bytes (test). - Request logs contain neither labels nor content of sealed parts (test).
- ADR 0138 notes that decision 4 is changed by ADR 0141; Spec 0006 §8 and Spec 0110 §4 describe blocks by media type.
- Relevant automated tests pass
Technical Notes
- Binary bytes are never decoded as text; text parts are only the already redacted, fenced strings.
- No change to the error code or locales.
Spec impact: update 0006, update 0110
Related Work
- Blocked by Chat content extension point: seal extension content at send time (limits, redaction, fencing, binary checks) #505; builds on Chat content extension point: opaque extension part in chat messages, history display and removal callbacks #504
- Chat attachments: send image and PDF attachments as native content blocks, reject them for models that cannot read them #470, feat(ai-providers): send image and PDF attachments as native content blocks #477, ADR 0138, ADR 0141
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestsecurity-reviewTouches security-critical code: elevated reviewTouches security-critical code: elevated review
Type
Fields
Priority
P2 - Normal
Stage
Done
Role
None
Activity
smart-ssh-coder commentedon Oct 14, 2026
Implementation update
PR: #525
Implemented:
ContentPart::Binary, media-class capability check (InputSupport::accepts),AiError::InputUnsupported(MediaClass)(code/locales unchanged)Remaining / open:
Decisions:
smart-ssh-reviewer commentedon Oct 14, 2026
Review result
Approved — see review on #525.
Findings:
Recommended next step: merge (once CI is green)
refactor(ai-providers): generic binary content blocks, capability che…