Skip to content

fix(model-runtime): complete streaming invocation callbacks #375

Closed

Description

@WH-2099

The callbacks registered with LargeLanguageModel.invoke(stream=True) omit parts of the invocation outcome.

  1. Consume a runtime stream containing two valid tool calls to completion.
    The yielded chunks contain both calls, but on_after_invoke receives an LLMResult with an empty message.tool_calls list.
    The same problem occurs when a call's arguments arrive as fragments.
  2. Consume a runtime stream that raises before its first chunk or after one chunk.
    The caller receives InvokeError, but on_invoke_error is never called.
    Blocking consumption of the same stream calls the error callback once.

The completion callback should receive the assembled tool calls using the existing delta semantics.
The error callback should receive the original stream exception once before it is propagated to the caller, and failed streams should not trigger the completion callback.
These omissions affect custom callback consumers and the built-in debug logging callback.

Reproduced on a315092e0a84160aaf336091e6e547445208324e through the public invoke method with a local runtime generator.
Regression assertions in tests/model_runtime/test_model_dispatch.py currently produce four expected failures and 51 passing cases.
The failures are two missing tool-call results and two missing stream-error notifications.

Related context: #260 and #261 added provider-state preservation; #354 and #355 exposed result accumulation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions