[koc-Z3]

Delivery status — 2026-11-02

Merged on 2026-10-12.
The fix is included in the planned SDK 0.11.0 release for Dify 1.19.0, tracked in #413.
The public SDK remains 0.10.2; plugin delivery follows publication.

Summary

OpenAI-compatible reasoning streams could split one thinking block at empty heartbeats and lose their final result at nearby stream boundaries.
Keep an active thinking block across absent, null, or empty reasoning fields until content, tool calls, or stream completion closes it.
For example, reasoning A, an empty delta, reasoning B, and answer text now produce one <think>AB</think> block.

The same response path now preserves the rest of the result:

• Retain the latest non-null finish reason and merge non-null usage fields, including explicit zero counts.
• Stop reading at [DONE] and close the HTTP response on completion, cancellation, parse failures, and read errors.
• Raise InvokeError for invalid JSON and provider error envelopes instead of returning a successful partial result.
• Associate parallel tool argument fragments with their provider index, including an initial frame that contains only the tool ID and a provider ID arriving after argument fragments.
• Send accumulated tool calls with the final usage and finish reason, after closing any active thinking block.
• Estimate missing usage from the complete chat history and tool output; preserve the completion API's single-prompt behavior and prefer provider counts.
• Apply model error mapping while consuming lazy streams and preserve existing SDK error types.

Refs #277, #366, and #391.

Validation

• just check: passed.
• just test: 276 passed, 1 skipped because the Dify plugin CLI is unavailable.
• just build: wheel and source distribution built successfully.
• Regression tests use real requests.Response iteration, stream parsing, final chunks, and usage calculation with in-memory network data.
  They cover heartbeats, terminal frames, invalid JSON, provider errors, response cleanup, partial/zero usage, parallel tool arguments, missing usage, and lazy stream errors.
• Additional local checks exercised real official plugin classes, the SDK executor and stdio/TCP/serverless writers, JSON/XML wrappers, backwards invocation aggregation, and Graphon reasoning/metadata consumption.
  These checks used local data and simulated sockets, not deployed model servers.

Call-chain audit

The audit followed _generate → _handle_generate_stream_response → thinking/tool aggregation → _create_final_llm_result_chunk → LargeLanguageModel.invoke → executor/writers → daemon/Dify/Graphon consumers.

• Audited 31 official OAICompatLargeLanguageModel subclasses at dify-official-plugins@7584b133.
  Twenty-nine inherit the SDK stream handler; OpenRouter and Stepfun implement their own.
  The heartbeat guard stays before provider wrapper dispatch so existing wrapper overrides receive the fix.
• Retained the _increase_tool_call interface because OpenRouter imports it directly.
  Finalizer signatures and wire schemas remain compatible with actual callers.
• Checked daemon 689f331f, Dify 1a817b99, and Graphon 11e2dee8 (0.7.0, the version pinned by Dify), as well as current Graphon main.
  Their consumers accept tool calls, usage, and finish reason in the same final chunk and do not deduplicate chunks by index.
  No consumer changes are required for these SDK fixes.
• GPUStack, CometAPI, DeerAPI, and OpenRouter override reasoning wrappers that can discard answer text when it arrives in the same delta as reasoning.
  OpenRouter's independent stream handler also needs matching terminal-state, tool-aggregation, and response-cleanup fixes.
  These plugin-specific changes require a companion PR in the official plugin repository.
  SDK publication also requires downstream lock updates and plugin repackaging before installed plugins receive the fixes.

Compatibility

Malformed response data now fails the invocation instead of being reported as a completed partial answer.
Empty reasoning fields alone no longer terminate an active thinking block.
Usage estimates remain local approximations when the provider omits counts.
No public schema or package version is changed by this PR.

General SSE framing expansion, transport keepalive/EOF policy, standalone JSON/XML fence parsing, and unrelated non-stream response behavior are outside this change.

Available checks

• Compatibility and downstream call sites checked.
• Regression tests added for the changed behavior.
• just check, just test, and just build completed.
• Release and plugin rollout requirements documented.

[koc-Z3]

Follow-up: deployment note for plugin authors

While validating this fix end-to-end with the langgenius/openai_api_compatible plugin, I discovered that each installed plugin ships with its own bundled copy of the dify-plugin SDK inside its isolated virtualenv at:

/app/storage/cwd/<plugin>@<hash>/.venv/lib/python<ver>/site-packages/dify_plugin/

When patching in-place, you must patch both:

1. The system-installed SDK (e.g. /usr/local/lib/python3.12/dist-packages/dify_plugin/ inside the plugin_daemon container) — which is what most casual debugging suggests.
2. The bundled SDK inside each plugin's own .venv — which is what the plugin actually imports at runtime.

For the bundled SDK, also delete the matching __pycache__/llm.cpython-*.pyc so the next request recompiles the patched source, and restart docker-plugin_daemon-1 so the plugin subprocess reloads the module.

This is a deployment-side caveat, not an SDK bug — the PR fix itself is correct. Filing here for visibility because the symptom would look like the patch didn't take if you only patch #1.

[WH-2099]

Reviewed the complete six-file diff from e6daca8 to 4674b90 with three independent reviews of correctness, stream handling, and caller compatibility.

No blocking regressions found. The changes preserve reasoning through heartbeats, keep terminal metadata and tool arguments, close responses on completion or cancellation, and map errors raised while consuming lazy streams. Checked the actual SDK and downstream consumers of the combined tool/usage final chunk, plus the 66 official LLM override signatures affected by keyword invocation.

Validation: 151 model tests and 8 subtests passed; lockfile, Ruff formatting/lint, and diff checks passed. Additional local checks used real requests.Response iteration and the tokenizer, public invoke/executor paths, synchronous and lazy errors, structured output, and cancellation cleanup.

Existing limits remain: fallback prompt usage omits tool definitions, and fragmented tool names can replace an earlier name fragment. Both were reproduced against the base revision as well; neither is introduced by this diff. Fallback usage is still an approximation.

Remote CI for this commit is awaiting workflow approval. No live provider calls or production load tests were performed.