Skip to content

[Bug]: Text queries discard image candidates before invoking a vision-capable reranker #45227

Open

Description

@Zhi-Xiang-Guo

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.18.0; reproduced on source main@170dbaeb34789365131a7723f1f33063c4e58558. Initially investigated at 4c521a3.

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

A text query recalls an image from a multimodal knowledge base, but enabling a vision-capable reranker filters it out before scoring.

  1. Use a high-quality multimodal knowledge base with actual openai/clip-vit-base-patch32 text/image embeddings (512 dimensions) and Qdrant. Configure the official compatible plugin with vision_support=support for both the embedding and rerank models. The adapter below uses actual normalized CLIP cosine scores.
  2. Index three short text segments. Upload a generated 224x224 red-square PNG through /console/api/files/upload and attach its upload ID to two segments through the supported segment API's attachment_ids. Wait for indexing.
  3. Retrieve a red square with reranking disabled using the request below. Confirm an image is returned.
  4. Repeat with only reranking_enable changed to true. Compare the returned images and the candidate types received by the model endpoint. Do not remove attachments between requests.
POST /v1/datasets/{dataset_id}/retrieve
Authorization: Bearer <knowledge-api-key>
Content-Type: application/json
{
  "query": "a red square",
  "retrieval_model": {
    "search_method": "semantic_search",
    "reranking_enable": false,
    "reranking_model": {
      "reranking_provider_name": "langgenius/openai_api_compatible/openai_api_compatible",
      "reranking_model_name": "<registered-clip-reranker-name>"
    },
    "top_k": 10,
    "score_threshold_enabled": false
  }
}

Why this is a bug: the query modality and candidate modality are independent. Dify's retrieval settings documentation recommends selecting a multimodal reranker for multimodal embeddings and warns that otherwise retrieved images will be excluded. This supports the expectation that a compatible vision-capable reranker can score image candidates. The documentation does not explicitly specify the text-query branch; that expectation is an interpretation of the documented behavior, supported by the official compatible plugin's multimodal text-query handling.

Fewer images in final results alone would not demonstrate this bug. Here the threshold is disabled, top_k exceeds the candidate count, and the model never receives the image.

Executable reproduction: prerequisites and commands

Reproduction through Dify HTTP APIs

Use a disposable Self Hosted (Source) workspace. The setup changes that workspace's default embedding model and creates two model registrations, a knowledge base, one document, and a generated PNG; it does not modify production source or insert database records directly.

Tested baseline: main@170dbaeb34789365131a7723f1f33063c4e58558 (version 1.18.0), macOS arm64; API Python 3.13.15; PostgreSQL 15, Redis 6, Qdrant 1.12.6, plugin daemon 0.6.11-local. Actual CLIP inference uses Python 3.12.14, transformers 5.22.1, torch 2.15.0.

  1. Deploy the source API and worker following https://github.com/langgenius/dify/blob/main/api/README.md with Qdrant as the vector store. Run the worker queues dataset,priority_dataset. Initialize a disposable workspace and install official langgenius/openai_api_compatible plugin 0.0.73 (identifier: langgenius/openai_api_compatible:0.0.73@4cf123ffaa4556f079bed6c47f3a462cc825504b859b183239e74eb19136a4b5). Obtain a Knowledge API key from that same workspace.
  2. Save the supplied model_server.py and reproduce.py in one directory. Start the actual-model adapter:
uv venv .model-env --python 3.12
uv pip install --python .model-env/bin/python transformers==5.22.1 torch==2.15.0 pillow fastapi uvicorn requests
.model-env/bin/python -m uvicorn model_server:app --host 0.0.0.0 --port 15013

The model weights are downloaded on first startup. This is a local test service; choose a bind address reachable by your plugin daemon. On Docker Desktop the adapter URL below uses host.docker.internal; on Linux configure the equivalent host gateway or another reachable address. The adapter performs normalized CLIP cosine scoring, with no fixed or random responses. It is a protocol adapter, not a claim that CLIP is a production cross-encoder reranker.

  1. In another terminal set your own values. The login account and API key must belong to the same disposable workspace:
export DIFY_URL=http://127.0.0.1:15011
export MODEL_URL=http://host.docker.internal:15013/v1
export DIFY_EMAIL='your-disposable-workspace-owner-email'
read -s DIFY_PASSWORD
export DIFY_PASSWORD
read -s DIFY_DATASET_API_KEY
export DIFY_DATASET_API_KEY
.model-env/bin/python reproduce.py setup
.model-env/bin/python reproduce.py before

setup logs in through the Console API, registers vision-capable models, uploads a 224x224 red PNG, and creates three text segments. Two of those segments share the PNG. The state file contains only dataset/model identifiers, no credentials. Wait for the worker to finish indexing if needed; the retrieval assertion requires an image in the no-rerank control.

  1. On the original code, both retrieve requests return HTTP 200. Reranking disabled: one image. Reranking enabled: zero images. The model service logs three string candidates; no image inference occurs for the rerank request.
  2. Stop the source API, apply the fix, and restart the API with the same database/storage/model configuration. Keep fixture.json and the knowledge base unchanged. Run:
.model-env/bin/python reproduce.py after

Both requests now return one image. The rerank endpoint receives four structured candidates, and its logs include actual image inference. The image remains subject to normal relevance scores, threshold and top-k; this fixture disables thresholds and uses top_k=10, greater than the candidate count.

The fresh-fixture setup and before/after commands were actually run on the same local deployment. They are not merely syntax-checked examples. They do not claim a second clean deployment, every commercial provider, or improved final LLM answer quality.

model_server.py
"""Real CLIP inference behind the OpenAI-compatible embedding contract."""
import base64,io,logging
import torch
from PIL import Image
from transformers import CLIPModel,CLIPProcessor
from fastapi import FastAPI
model_id='openai/clip-vit-base-patch32'
model=CLIPModel.from_pretrained(model_id)
processor=CLIPProcessor.from_pretrained(model_id)
model.eval()
app=FastAPI()
@app.post('/v1/embeddings')
def embeddings(body:dict):
    if 'messages' in body:
        items=[c for m in body['messages'] if m['role']=='user' for c in m['content']]
        images=[Image.open(io.BytesIO(base64.b64decode(c['image_url']['url'].split(',',1)[1]))).convert('RGB') for c in items if c['type']=='image_url']
        with torch.inference_mode():
            result=model.get_image_features(**processor(images=images,return_tensors='pt'))
        kind='image'
    else:
        texts=body['input'];texts=[texts] if isinstance(texts,str) else texts
        with torch.inference_mode():
            result=model.get_text_features(**processor(text=texts,return_tensors='pt',padding=True,truncation=True,max_length=77))
        kind='text'
    if not isinstance(result,torch.Tensor):result=result.pooler_output
    result=result/result.norm(dim=-1,keepdim=True)
    logging.warning('Real CLIP inference kind=%s count=%d dimensions=%d',kind,result.shape[0],result.shape[1])
    return {'object':'list','model':model_id,'data':[{'object':'embedding','index':i,'embedding':v} for i,v in enumerate(result.tolist())],'usage':{'prompt_tokens':1,'total_tokens':1}}

@app.post('/v1/rerank')
def rerank(body:dict):
    def encode(value):
        if isinstance(value,str):return embeddings({'input':[value]})['data'][0]['embedding']
        content=value['content']
        images=[c for c in content if c['type']=='image_url']
        if images:return embeddings({'messages':[{'role':'user','content':images}]})['data'][0]['embedding']
        return encode(' '.join(c.get('text','') for c in content))
    query=torch.tensor(encode(body['query']))
    docs=body['documents']
    results=[{'index':i,'relevance_score':float(torch.dot(query,torch.tensor(encode(d)))), 'document':{'text':d if isinstance(d,str) else 'image'}} for i,d in enumerate(docs)]
    logging.warning('REAL_CLIP_RERANK query_type=%s document_types=%s',type(body['query']).__name__,[type(d).__name__ for d in docs])
    return {'results':sorted(results,key=lambda x:x['relevance_score'],reverse=True)[:body.get('top_n',len(docs))]}
reproduce.py
"""Create and replay a disposable Dify fixture through authenticated HTTP APIs."""
import argparse
import base64
import io
import json
import os
import time
from pathlib import Path

import requests
from PIL import Image

parser = argparse.ArgumentParser()
parser.add_argument('mode', choices=['setup', 'before', 'after'])
parser.add_argument('--state', default='fixture.json')
args = parser.parse_args()
base = os.environ.get('DIFY_URL', 'http://127.0.0.1:15011').rstrip('/')
endpoint = os.environ.get('MODEL_URL', 'http://host.docker.internal:15013/v1')
provider = 'langgenius/openai_api_compatible/openai_api_compatible'
session = requests.Session()
session.trust_env = False

def call(method, path, body=None, files=None, service=False):
    headers = {}
    if service:
        headers['Authorization'] = 'Bearer ' + os.environ['DIFY_DATASET_API_KEY']
    else:
        for cookie in session.cookies:
            if 'csrf' in cookie.name:
                headers['X-CSRF-Token'] = cookie.value
    response = session.request(method, base + path, json=body, files=files, headers=headers, timeout=120)
    if not response.ok:
        raise RuntimeError(f"{response.status_code} {path}: {response.text}")
    return response.json()

if args.mode == 'setup':
    call('POST', '/console/api/login', {
        'email': os.environ['DIFY_EMAIL'],
        'password': base64.b64encode(os.environ['DIFY_PASSWORD'].encode()).decode(),
        'language': 'en-US', 'remember_me': False,
    })
    suffix = str(time.time_ns())
    embedding = 'clip-embedding-' + suffix
    reranker = 'clip-reranker-' + suffix
    for name, kind in [(embedding, 'text-embedding'), (reranker, 'rerank')]:
        call('POST', '/console/api/workspaces/current/model-providers/' + provider + '/models/credentials', {
            'model': name, 'model_type': kind, 'name': 'Reproduction',
            'credentials': {'endpoint_url': endpoint, 'api_key': 'local-test',
                            'vision_support': 'support', 'context_size': '77', 'max_chunks': '16'},
        })
    call('POST', '/console/api/workspaces/current/default-model', {
        'model_settings': [{'model_type': 'text-embedding', 'provider': provider, 'model': embedding}],
    })
    dataset = call('POST', '/console/api/datasets', {
        'name': 'Vision rerank repro ' + suffix, 'indexing_technique': 'high_quality',
    })['id']
    call('PATCH', '/console/api/datasets/' + dataset, {
        'indexing_technique': 'high_quality', 'embedding_model': embedding,
        'embedding_model_provider': provider, 'is_multimodal': True,
        'retrieval_model': {'search_method': 'semantic_search', 'reranking_enable': False,
                            'top_k': 10, 'score_threshold_enabled': False},
    })
    document = call('POST', f'/v1/datasets/{dataset}/document/create-by-text', {
        'name': 'Red square', 'text': 'A red square diagram.', 'indexing_technique': 'high_quality',
        'embedding_model': embedding, 'embedding_model_provider': provider,
        'process_rule': {'mode': 'automatic'},
    }, service=True)['document']['id']
    for _ in range(120):
        segments = call('GET', f'/console/api/datasets/{dataset}/documents/{document}/segments')['data']
        if segments and all(s['status'] == 'completed' for s in segments):
            break
        time.sleep(1)
    else:
        raise RuntimeError('Indexing did not finish within 120 seconds')
    png = io.BytesIO()
    Image.new('RGB', (224, 224), (255, 0, 0)).save(png, format='PNG')
    upload = call('POST', '/console/api/files/upload', files={'file': ('red-square.png', png.getvalue(), 'image/png')})['id']
    for label in ['A', 'B']:
        call('POST', f'/console/api/datasets/{dataset}/documents/{document}/segment', {
            'content': f'Segment {label}: a red square diagram.', 'attachment_ids': [upload],
        })
    Path(args.state).write_text(json.dumps({'dataset': dataset, 'reranker': reranker}))
    print('Created fixture: three text segments; A and B share one PNG. Run before/after after indexing.')
else:
    state = json.loads(Path(args.state).read_text())
    evidence = []
    for enabled in [False, True]:
        body = {'query': 'a red square', 'retrieval_model': {
            'search_method': 'semantic_search', 'reranking_enable': enabled,
            'reranking_model': {'reranking_provider_name': provider, 'reranking_model_name': state['reranker']},
            'top_k': 10, 'score_threshold_enabled': False,
        }}
        result = call('POST', '/v1/datasets/' + state['dataset'] + '/retrieve', body, service=True)
        image_ids = sorted({f['id'] for r in result['records'] for f in r.get('files', [])})
        row = {'reranking_enable': enabled, 'status': 200, 'image_ids': image_ids,
               'records': [{'segment_id': r['segment']['id'], 'score': r['score'],
                            'file_ids': [f['id'] for f in r.get('files', [])]} for r in result['records']]}
        evidence.append(row)
        print(f'reranking_enable={enabled}: HTTP 200, returned images={len(image_ids)}')
    Path(args.mode + '.json').write_text(json.dumps(evidence, indent=2))
    assert [len(e['image_ids']) for e in evidence] == ([1, 0] if args.mode == 'before' else [1, 1])

✔️ Expected Behavior

For a compatible vision-capable reranker, text queries with mixed text/image candidates should give the images a chance to be scored. Final results can follow relevance, threshold and top-k. This does not require retaining every image or asking text-only models to process images.

❌ Actual Behavior

Both requests return HTTP 200 on the original code:

Reranking Text records returned Images returned
Disabled 3 1
Enabled, vision-capable 3 0

Model endpoint log for the failing request:

REAL_CLIP_RERANK query_type=str document_types=['str', 'str', 'str']

api/core/rag/rerank/rerank_model.py routes vision-capable models into fetch_multimodal_rerank and constructs multimodal candidates. Its TEXT_QUERY branch then calls fetch_text_rerank, which excludes internal image documents. The prepared image inputs are never used in this branch.

Related work: #41480 is still open and handles source/index alignment when UploadFile rows are missing; its diff leaves this text-query routing unchanged. All uploads exist in this reproduction. #37117 / #37165 concern attachment-only image-query routing. Targeted searches across open/closed issues and PRs found no same-root-cause fix; this is not a claim of exhaustive historical coverage.

The reproduction uses authenticated Dify HTTP APIs, real embedding/rerank inference, worker indexing and Qdrant. A fresh knowledge-base fixture was tested with the scripts above. AI assistance was used for investigation, implementation, verification and drafting. This does not establish every commercial provider's behavior or final LLM answer quality.

Related fix and before/after validation: #45229.

Activity

  1. dosubot commented on Nov 1, 2026

    @dosubot

    There is a useful regression-test gap here: test_fetch_multimodal_rerank_builds_docs_and_calls_text_rerank currently mocks fetch_text_rerank and checks that it receives three documents, including the image. It never exercises the filtering inside that method, so it does not catch the image being discarded before model invocation. [1]

    The compatible plugin already has a complementary test, test_raw_base64_document_image_gets_data_uri_prefix, that supplies a text query with an image candidate and asserts that the outgoing request contains the image data URI. [2]

    For regression coverage, I’d add a runner-level test with vision enabled and mixed text/image candidates, without mocking fetch_text_rerank. It should assert that invoke_multimodal_rerank receives both candidate types and that a returned image index maps back to the image’s metadata. That would bridge the gap between the existing runner and plugin tests. [3] [1] [2]

    To reply, just mention @dosu.


    Docs are dead. Just use Dosu.

    Leave Feedback Ask Dosu about dify

  2. added a commit that references this issue on Nov 1, 2026
    0ae37f3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions