OpenAI-compatible LLM: a stream that ends without finish_reason or [DONE] is returned as a complete answer #409

Delivery status — 2026-11-02

Implemented by #410, merged on October 29.
Record this issue as completed; #412 separately closes the empty-string finish-marker gap and remains a draft.
SDK 0.11.0 publication and plugin delivery are tracked in #413 and #366 for Dify 1.19.0.

Bug Description

When an OpenAI-compatible endpoint ends a streamed response early but cleanly, with no chunk carrying a finish_reason and no data: [DONE], OAICompatLargeLanguageModel returns the text received so far as a finished answer. Dify then saves and shows the cut-off text as a normal, complete answer, and a workflow's LLM node succeeds with it as its output. This is what a client sees when a gateway, relay or load balancer between Dify and the model ends the response after its upstream stream died or timed out.

In _handle_generate_stream_response (llm.py#L767-L938):

• finish_reason starts as None and is set only from a chunk that carries one (#L789, #L810-L811).
• _iter_stream_json stops the same way on [DONE] and at the end of the body, so the caller can't tell them apart (#L88-L131).
• After the loop, the final chunk is yielded unconditionally, with finish_reason=None if none arrived (#L929-L938). When no usage chunk arrived, tokens are estimated from the partial text.

An abrupt TCP close is handled correctly: requests raises ChunkedEncodingError and Dify shows an error. Only the clean early end slips through. Every plugin built on OAICompatLargeLanguageModel inherits this, not only openai_api_compatible.

Steps to reproduce

Seen with Dify 1.18.0 and the marketplace plugin langgenius/openai_api_compatible 0.0.73, which runs on dify_plugin 0.10.2. The stream parser was rewritten on main (a6c7642) but still ends this way; I read that from the code and didn't run main.

1. Add an OpenAI-compatible model (chat mode, streaming) pointing at an endpoint you control.
2. Create a Chatflow: Start → HTTP Request → LLM → Answer. Publish it.
3. Make the endpoint send the role chunk and 2 content chunks, then end the HTTP response normally (a proper end of the chunked body). No chunk with finish_reason, no [DONE].
4. POST /v1/chat-messages with "response_mode": "streaming".

The result, 3 of 3 runs on a Chatflow and 3 of 3 on a basic Chat app:

• The client gets only the first 24 characters of the answer, then message_end, and in the Chatflow workflow_finished with status succeeded. No error event.
• The saved message has the cut-off answer, status normal, no error.
• The LLM node execution is succeeded with finish_reason: null, and its text output is the cut-off text that later nodes would run on.

Without the fault (full stream, finish_reason: "stop", [DONE]): the full answer is saved, 1 of 1 control runs on each app. With an abrupt TCP close at the same point: an error event, the message saved as error, the run failed.

Recording of the reproduction above (Dify 1.18.0, dify_plugin 0.10.2):

Expected behavior

A stream that ends before the model finished is reported as an error (InvokeError), the same as an abrupt disconnect, so Dify marks the message and the run as failed instead of complete.

Technical inputs

• Have _iter_stream_json report whether it stopped on [DONE], and raise InvokeError("stream ended before the model finished") when the body ends with no [DONE] and no non-null finish_reason.
• Requiring [DONE] alone might break compatible providers that omit it but send finish_reason, so accept either.

Found by Opslane.