Skip to content

AI providers: generic binary content blocks and capability check by media type #506

Closed

Description

@smart-ssh-planner

Goal

Providers send binary content blocks generically by media type instead of by attachment kind, and the model capability check is keyed by media class. Behaviour for images and PDFs stays the same (ADR 0141, decision 5; changes ADR 0138, decision 4).

Context

  • crates/ai-providers/src/content.rs has ContentPart::{Text, Image, Pdf}, message_parts/group_parts (matching on MessageContent::Attachments), check_attachments and AttachmentSupport { images, pdf } per provider (anthropic_support, openai_responses_support, openai_compatible_support; local OpenAI-compatible servers such as Ollama via VISION_HINTS).
  • AiError::AttachmentUnsupported(AttachmentKind) (crates/core/src/ai/types.rs:833), code AI_ATTACHMENT_UNSUPPORTED, UI text in all locales.
  • The sealing issue introduced MessageContent::Sealed(SealedContent) with sealed binary blocks { mime, filename, bytes }.

Requirements

  • ContentPart::{Text, Binary { mime, filename, base64 }} replaces Image/Pdf. Serialisation by media class: Anthropic image / document, Chat Completions image_url / file, Responses input_image / input_file.
  • Parts are built from Sealed content (fenced text, then its binaries) and, until the old model is removed, from Attachments through the same generic path.
  • Capability table keyed by media class (InputSupport::accepts(MediaClass)); same results as today for every provider/model.
  • AiError::InputUnsupported(MediaClass) replaces AttachmentUnsupported; code AI_ATTACHMENT_UNSUPPORTED and UI text unchanged (decided by maintainer). The check runs over the whole history before any request.
  • Request logging: only counts and bytes for sealed content and binaries, never label or content.

Acceptance Criteria

  • Recorded request fixtures for image and PDF on Anthropic, Responses and Chat Completions are unchanged.
  • Every case of the existing capability table tests gives the same result through accepts(MediaClass) (incl. unknown models, api.openai.com vs. other hosts, vision hints).
  • An image in the history for a text-only model → InputUnsupported, code AI_ATTACHMENT_UNSUPPORTED, no request (test per provider).
  • Sealed content and legacy Attachments produce identical blocks for the same bytes (test).
  • Request logs contain neither labels nor content of sealed parts (test).
  • ADR 0138 notes that decision 4 is changed by ADR 0141; Spec 0006 §8 and Spec 0110 §4 describe blocks by media type.
  • Relevant automated tests pass

Technical Notes

  • Binary bytes are never decoded as text; text parts are only the already redacted, fenced strings.
  • No change to the error code or locales.

Spec impact: update 0006, update 0110

Related Work

Activity

  1. added this to the 0.7.0 milestone on Oct 13, 2026
  2. smart-ssh-coder commented on Oct 14, 2026

    @smart-ssh-coder
    Contributor

    Implementation update

    PR: #525
    Implemented:

    • Generic ContentPart::Binary, media-class capability check (InputSupport::accepts), AiError::InputUnsupported(MediaClass) (code/locales unchanged)
    • Tests for sealed vs legacy parity, sealed capability rejection, log redaction; Spec 0006/0110 and ADR 0138 updated
      Remaining / open:
    • CI had not reported any checks yet when I finished; the local gate is green.
      Decisions:
    • Non-PDF, non-image mime is treated as image (unchanged behaviour).
  3. smart-ssh-reviewer commented on Oct 14, 2026

    @smart-ssh-reviewer

    Review result

    Approved — see review on #525.

    Findings:

    • No blocking findings. Non-blocking: legacy attachments now classify by mime instead of kind; no differing input found.
    • CI test jobs were still pending at review time.

    Recommended next step: merge (once CI is green)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestsecurity-reviewTouches security-critical code: elevated review

    Type

    Fields

    Priority

    P2 - Normal

    Stage

    Done

    Role

    None

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions