[gakugaku]

Summary

When thinking mode is disabled for a model, the plugin removes a leading <think>…</think> block from the response. In streaming mode this only works if <think> arrives at the start of a single chunk. If the tag is split across chunks (for example < followed by think>), _filter_thinking_stream sees that the buffer does not start with <think>, yields the chunk and drops the buffer, so the whole reasoning is passed through to the answer.

This happens with gateways that stream reasoning inline in content, e.g. LiteLLM in front of a model that emits <think> in its text. Models that return reasoning in reasoning_content are not affected, because the plugin builds the <think> tag in one chunk.

• A chunk whose buffered content is a prefix of <think> is held until the tag can be decided. If the stream ends first, the held content is yielded as is.
• Whitespace after </think> is now skipped even when it arrives in later chunks, matching the non-streaming filter (<think>.*?</think>\s*).
• Content held as a possible tag is yielded before chunks without content, so it is not emitted after the final chunk with finish_reason and usage.

As before, a <think> block is detected at any chunk boundary, not only at the start of the stream, which matches the non-streaming filter that removes every block.

Release Notes

• Fixed: with thinking mode disabled, reasoning was shown in streamed answers when the <think> tag was split across chunks.

Change Type

• Documentation / non-plugin change
• Non-LLM plugin (tools, extensions, datasource, etc.)
• LLM plugin

Screenshots / Videos

Streamed content chunks with thinking mode disabled:

Chunks	Before	After
<, think, >, reasoning, </think>, \n\nanswer	<think>reasoning</think>\n\nanswer	answer
<think>, reasoning, </think>, \n\nanswer	\n\nanswer	answer

LLM Plugin Checklist

Areas affected by this change (check all that apply)

• Message flow (system messages, user ↔ assistant turn-taking)
• Tool interaction flow (multi-round usage, Agent App and Agent Node)
• Multimodal input (images, PDFs, audio, video, etc.)
• Multimodal output (images, audio, video, etc.)
• Structured output (JSON, XML, etc.)
• Token consumption metrics
• Other LLM functionality (reasoning, grounding, prompt caching, etc.)
• New models / model parameter fixes

Version

• Bumped top-level version in manifest.yaml (not the one under meta)
• dify_plugin>=0.3.0,<0.11.0 is declared in pyproject.toml and locked in uv.lock (or kept in requirements.txt for legacy plugins without uv.lock) - SDK docs

Testing

Added tests/test_filter_thinking_stream.py for split opening and closing tags, whitespace-only chunks after </think>, consecutive thinking blocks, content that only looks like the tag (<b>), a partial tag before the final chunk and at the end of the stream, and an unclosed thinking block. All tests of the plugin pass locally (uv run --frozen --with pytest python -m pytest).

Also ran the plugin against a LiteLLM gateway (streaming, agent_thought_support: not_supported) with a model that returns reasoning inline in content. The first two chunks were <th and ink>The. The same stream gave:

• Before: <think>The user asks ... Probably just "2".</think>2
• After: 2

• Local deployment - Dify version:
• SaaS (cloud.dify.ai)

[crazywoola]

Hi @gakugaku, thanks for fixing think tags that are split across streaming chunks!

Heads-up about models/openai_api_compatible/manifest.yaml: the consolidated dependency refresh (#4106–#4111) bumped it to 0.0.69 on main, and several PRs are currently open for this plugin from different authors (#3795, #3869, #4060, #4068, #4079). Please rebase on latest main and bump the version to the next unused one so the releases don't collide. If there's a related issue for this fix, linking it in the description would help too.

Thanks again! 🙏

[gakugaku]

Rebased on main and bumped the version to 0.0.70. There is no related issue.