[laipz8200]

Summary

Ops tracing ran four overlapping paths: the legacy queue, "unified" trace, the Agent trace ledger and enterprise telemetry. They shared one loosely typed payload. Configuration was decrypted on the request path, and every provider package reached into the database, Redis and process-global SDK state. This PR replaces them with one pipeline and gives every provider the same minimal interface.

Upgrade notice

• Existing saved Langfuse provider settings need no changes. Saved credentials, host, and project URL continue to be used. There is no need to re-enter keys, re-save the provider, or toggle tracing.
• Upgrade the API and workers and apply the normal database migrations. For rolling deployments, follow the queue transition below. New runs automatically include nested workflow-tool invocations, including repeated calls and failures, while retaining formatted prompts and responses, token usage, and project/trace/observation links.
• Existing Docker .env files remain valid and need no regeneration. The root .env.example now contains only inputs requiring manual configuration. Application and Docker defaults load automatically, and both environment-sync scripts preserve existing overrides, including settings absent from the template. Optional settings such as OPS_TRACE_QUEUE remain documented in the service templates.
• The richer hierarchy applies to new executions; historical traces are not backfilled with nested spans. Span names and IDs follow the shared format described below.
• Custom clients of GET /apps/<id>/trace-config must use configured_secrets instead of expecting secret values in the response. Blank or omitted secrets still retain their saved values.

Rolling deployment

New tracing tasks default to OPS_TRACE_QUEUE=ops_trace_v2, isolating their names and payloads from old workers. The API, workers, and Beat must use the same value. Default Docker and development worker subscriptions follow it; custom queue lists must include the new queue and exclude legacy ops_trace while old workers are draining.

1. Start new workers on ops_trace_v2 and roll out the new producers. Keep an old tracing worker pool subscribed only to ops_trace. Replace the old Beat scheduler without running duplicate schedules.
2. Stop old producers after their in-flight requests, workflow jobs, and buffered trace emissions finish. Let old workers drain broker messages, active tasks, reserved/prefetched tasks, scheduled/ETA tasks, and retries, then stop them gracefully. The new sweeper preserves legacy payloads during this transition.
3. Keeping ops_trace_v2 is supported. To restore ops_trace, first remove all old workers and have new-version workers consume both queues. Roll every new API, worker, and Beat process with OPS_TRACE_QUEUE=ops_trace. Keep both queues subscribed until the ops_trace_v2 backlog and all in-flight or delayed work finish; retries can retain their original queue.

This sequence isolates tracing tasks; unrelated queues and database migrations still require normal compatibility checks. See the detailed rollout guide.

Provider interface (api/providers/trace/trace-sdk, imported as dify_trace_sdk)

This is the lowest common denominator of the 10 supported providers. Each provider implements its own send:

class TraceProvider[ConfigT: ProviderConfig]:
    id: ClassVar[str]
    config_model: ClassVar[type[ProviderConfig]]
    capabilities: ClassVar[ProviderCapabilities]

    def verify(self, config: ConfigT) -> ConfigT: ...          # check credentials, fill derived fields
    def project_url(self, config: ConfigT) -> str | None: ...  # no I/O
    def send(self, config: ConfigT, trace: Trace) -> SendResult: ...

• send receives a provider-neutral Trace: a parent-first tree of spans with OpenTelemetry IDs and typed LLM usage and cost.
• Each failure is classified as retryable, rejected or outcome-unknown.
• All network access goes through an injected Egress, which applies Dify's existing outbound HTTP policy.
• Providers are discovered through dify.trace_providers entry points, so a deployment can install any subset.
• Ported providers: Langfuse (ingestion REST API), LangSmith, Opik, Weave, Arize, Phoenix, Aliyun, MLflow, Databricks and Tencent.
  • Langfuse, Opik, Weave, Arize and Phoenix no longer use vendor SDKs.
  • LangSmith and MLflow still use their SDKs, routed through the egress.
• IDs are deterministic, so a resend updates the same records instead of duplicating them.

Pipeline

• Producers emit typed events through TraceScope.emit after their records are committed. A scope exists only when the app traces or enterprise telemetry is on, and emitting never raises or blocks the request.

• tasks.trace_delivery_task runs on OPS_TRACE_QUEUE (default ops_trace_v2):

  • reloads the records through the event's workspace;
  • feeds enterprise telemetry and the Agent collector;
  • builds the trace for the app's current configuration and calls send.

  A retryable failure stores only the built trace and resends it, up to OPS_TRACE_EXPORT_MAX_ATTEMPTS attempts.

• Workflow tools: persisted child frames are exported under the Tool span that invoked them, including nested calls and repeated invocations. Reads remain scoped to the admitted tenant, root run, and caller; predecessor and loop/iteration links are resolved within each invocation. This uses the existing Graphon version.

• The Agent trace ledger keeps its durable path and exports through send. It still reads runs captured before the upgrade.

• Removed:

  • the per-process queue and timer;
  • Redis parent-context polling;
  • the coordination between separate workflow runs, which remain separate traces linked by attributes.

Configuration

• Secrets are write-only. GET /apps/<id>/trace-config returns the non-secret fields plus configured_secrets.
  • A blank or omitted secret keeps its stored value.
  • Changing a destination field (host, endpoint or URL) requires entering the secrets again.
  • The console form follows this.
• Migration 53c10126d551 keeps one trace_app_config row per (app, provider) and adds a unique index.
• Settings:
  • Removed: OPS_TRACE_UNIFIED_ENABLED, OPS_TRACE_RETRYABLE_DISPATCH_MAX_RETRIES, OPS_TRACE_RETRYABLE_DISPATCH_DELAY_SECONDS, OPS_TRACE_PARENT_CONTEXT_TTL_SECONDS.
  • Added: OPS_TRACE_QUEUE (ops_trace_v2), OPS_TRACE_EXPORT_MAX_ATTEMPTS (6), and ENABLE_TRACE_PAYLOAD_SWEEP_TASK (daily cleanup of orphaned new-pipeline payloads).

Removed

• api/core/ops/**;
• services/app_tracing_config_gateway.py;
• the legacy provider dispatch and transports;
• the per-provider *_trace.py and agent_trace.py exporters;
• the opik, weave, langfuse and arize-phoenix-otel dependencies.

Dropping those SDKs exposed runtime dependencies the API never declared. They are now direct dependencies at their already-locked versions:

• pydantic-settings, which configs uses;
• sentry-sdk;
• the OTLP gRPC and HTTP exporters.

Architecture

The tracing slice follows the backend architecture guide linked from api/AGENTS.md, including the domain boundary described in #44957:

• Domain (api/domain/tracing): an Agent trace run is the aggregate TraceRunAgg, with its invocations, batches and receipts as child entities.
  • Its methods own the run's transitions: freezing the destination and capture policy, collecting batches, capturing the terminal, claiming, accepting, deferring and retrying exports.
  • Callers pass in the current time, IDs and export tokens; the package performs no I/O.
  • Recovering a dropped terminal is a domain service of named functions over the run and its source's committed state.
  • Domain errors are separate from the application's and are translated at the boundary, so error codes are unchanged.
• Application services (TraceScopes, TraceDeliveryService, the provider configuration gateway and the Agent trace collector) receive ports and plain settings at construction. They no longer open sessions, read configuration, or reach Celery, storage or the encrypter themselves. Their errors share one base in services/tracing/errors.py with no HTTP status, and the handlers map them.
• Records are immutable values. Repositories return plain records through their application ports, and database failures surface as TracingStoreUnavailableError. Trace record assembly moved to repositories/trace_record_repository.py, with configured workflow readers injected from the composition root. Offloaded payloads are read through their execution's tenant after the SQL session closes.
• Agent trace runs are loaded and saved whole by ApplicationTraceRepository, which holds the run's row lock from load to save. The unique source key settles concurrent starts.
• Adapters in services/tracing/adapters.py and registry.py implement the remaining ports: the Celery emitter and retry queue, the entry-point provider catalog, workspace secret encryption, enterprise telemetry and Agent export scheduling.
• Composition root: extensions/application_services/tracing.py builds every tracing service from configuration. Celery tasks and handlers get them from application_services().tracing, and the Agent hooks are bound only when Agent tracing is on.
• Import contracts:
  • domain-boundary keeps api/domain free of application, persistence, framework, configuration, queue, transport and legacy kernel code.
  • tracing-application-boundary, tracing-controller-boundary and tracing-repository-boundary keep the services free of Flask, SQLAlchemy, models, configs and Celery, keep the handlers off persistence and the domain, and keep the repositories on the contracts they implement.

Behaviour changes worth reviewing

• Trace and span IDs are now deterministic per Dify record, so traces sent before and after the upgrade won't merge in vendor UIs.
• Span names and structure come from one shared builder. Some vendor-specific attributes and the Tencent metrics are gone.
• MLflow OSS export needs an MLflow server at version 3.4 or later with a database-backed store.
• Only old workers can export legacy process_trace_tasks payloads. The new version retains a retired handler that discards unsupported messages; it must not be used to drain the old queue.
• With a caller-supplied trace ID, suggested-question spans still form their own trace, as before. That request does not know the caller's trace ID.

Screenshots

Langfuse nested workflows: workflow-tool hierarchy with formatted LLM prompts and responses.

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've updated the documentation accordingly.

[github-actions]

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	81.50%	82.51%	+1.01%
Strict coverage	81.15%	82.17%	+1.02%
Typed symbols	85,534	85,009	-525
Untyped symbols	19,495	18,092	-1,403
Modules	4396	4371	-25

[github-actions]

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-10-31 15:20:51.710680491 +0000
+++ /tmp/pyrefly_pr.txt	2026-10-31 15:20:33.154536401 +0000
@@ -332,8 +332,6 @@
    --> tests/e2e_tests/containers/models/test_types_enum_text.py:147:20
 ERROR Object of class `NoneType` has no attribute `user_type_nullable` [missing-attribute]
    --> tests/e2e_tests/containers/models/test_types_enum_text.py:148:20
-ERROR Argument `Literal['workflow-run']` is not assignable to parameter `triggered_from` with type `SQLCoreOperations[WorkflowNodeExecutionTriggeredFrom] | WorkflowNodeExecutionTriggeredFrom` in function `models.workflow.WorkflowNodeExecutionModel.__init__` [bad-argument-type]
-  --> tests/e2e_tests/containers/repositories/test_sqlalchemy_api_workflow_node_execution_repository.py:39:24
 ERROR Generator function should return `Generator` [bad-return]
    --> tests/e2e_tests/containers/repositories/test_sqlalchemy_api_workflow_run_repository.py:150:56
 ERROR Argument `Literal['chat']` is not assignable to parameter `mode` with type `AppMode | SQLCoreOperations[AppMode]` in function `models.model.App.__init__` [bad-argument-type]
@@ -1788,14 +1786,10 @@
    --> tests/e2e_tests/contracts/core/workflow/nodes/test_if_else.py:453:19
 ERROR `Literal['{invalid-json-schema']` is not assignable to attribute `json_schema` with type `dict[str, Any] | None` [bad-assignment]
    --> tests/e2e_tests/contracts/core/workflow/nodes/test_start_node_json_object.py:209:28
-ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
-  --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:73:9
-ERROR Object of class `ModuleType` has no attribute `TraceTask` [missing-attribute]
-  --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:74:9
 ERROR Argument `_StubToolRuntime` is not assignable to parameter `runtime` with type `ToolNodeRuntimeProtocol` in function `graphon.nodes.tool.tool_node.ToolNode.__init__` [bad-argument-type]
-   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:125:17
+   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:112:17
 ERROR Argument `Iterator[ToolRuntimeMessage]` is not assignable to parameter `messages` with type `Generator[ToolRuntimeMessage]` in function `graphon.nodes.tool.tool_node.ToolNode._transform_message` [bad-argument-type]
-   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:141:18
+   --> tests/e2e_tests/contracts/core/workflow/nodes/tool/test_tool_node.py:128:18
 ERROR `-` is not supported between `None` and `datetime` [unsupported-operation]
   --> tests/e2e_tests/contracts/libs/_human_input/test_models.py:64:21
 ERROR `-` is not supported between `None` and `datetime` [unsupported-operation]
@@ -2859,87 +2853,87 @@
 ERROR No matching overload found for function `str.join` called with arguments: (list[AgentScratchpadUnit.Action | str]) [no-matching-overload]
    --> tests/unit_tests/core/agent/output_parser/test_cot_output_parser.py:286:25
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:121:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:143:29
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:124:5
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:146:5
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:129:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:151:9
 ERROR Missing required key `action` for TypedDict `ActionDict` [bad-typed-dict-key]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:157:44
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:179:44
 ERROR Missing required key `action_input` for TypedDict `ActionDict` [bad-typed-dict-key]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:157:44
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:179:44
 ERROR Key `invalid` is not defined in TypedDict `ActionDict` [bad-typed-dict-key]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:157:45
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:179:45
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:211:55
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:233:55
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:224:55
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:246:55
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:248:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:270:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:250:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:272:33
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:251:61
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:273:61
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:269:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:291:33
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:272:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:294:29
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:275:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:297:9
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:291:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:313:33
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:294:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:316:29
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:305:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:327:35
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:309:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:331:9
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:321:29
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:343:29
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:326:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:348:9
 ERROR `Literal[1]` is not assignable to attribute `prompt_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:332:32
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:354:32
 ERROR `Literal[1]` is not assignable to attribute `completion_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:333:36
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:355:36
 ERROR `Literal[2]` is not assignable to attribute `total_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:334:31
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:356:31
 ERROR `Literal[1]` is not assignable to attribute `prompt_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:340:32
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:362:32
 ERROR `Literal[1]` is not assignable to attribute `completion_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:341:36
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:363:36
 ERROR `Literal[2]` is not assignable to attribute `total_price` with type `Decimal` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:342:31
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:364:31
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:370:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:392:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:382:19
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:404:19
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:402:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:424:9
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:410:16
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:432:16
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:428:25
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:450:25
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:448:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:472:9
 ERROR `None` is not assignable to attribute `agent_callback` with type `DifyAgentCallbackHandler` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:456:33
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:480:33
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:458:25
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:482:25
 ERROR `list[str]` is not assignable to attribute `_agent_scratchpad` with type `list[AgentScratchpadUnit]` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:488:36
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:513:36
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:510:55
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:536:13
 ERROR Object of class `FunctionType` has no attribute `assert_called` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:514:9
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:541:9
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:580:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:607:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:594:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:621:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:608:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:635:35
 ERROR Object of class `DummyRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:623:35
+   --> tests/unit_tests/core/agent/test_cot_agent_runner.py:650:35
 ERROR `list[DummyTool]` is not assignable to attribute `_prompt_messages_tools` with type `Sequence[PromptMessageTool]` [bad-assignment]
   --> tests/unit_tests/core/agent/test_cot_chat_agent_runner.py:49:37
 ERROR `DummyGenerateEntity` is not assignable to attribute `application_generate_entity` with type `AgentChatAppGenerateEntity` [bad-assignment]
@@ -2957,81 +2951,81 @@
 ERROR `in` is not supported between `Literal['Question: What is Python?']` and `None` [not-iterable]
    --> tests/unit_tests/core/agent/test_cot_completion_agent_runner.py:180:16
 ERROR `Literal[0]` is not assignable to attribute `prompt_price` with type `Decimal` [bad-assignment]
-  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:51:26
+  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:53:26
 ERROR `Literal[0]` is not assignable to attribute `completion_price` with type `Decimal` [bad-assignment]
-  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:52:30
+  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:54:30
 ERROR `Literal[0]` is not assignable to attribute `total_price` with type `Decimal` [bad-assignment]
-  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:53:25
+  --> tests/unit_tests/core/agent/test_fc_agent_runner.py:55:25
 ERROR Missing argument `session` in function `core.agent.base_agent_runner.BaseAgentRunner.__init__` [missing-argument]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:194:37
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:216:37
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:227:5
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:249:5
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:238:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:260:9
 ERROR Argument `None` is not assignable to parameter `query` with type `str` in function `core.agent.fc_agent_runner.FunctionCallAgentRunner._organize_user_query` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:339:46
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:361:46
 ERROR `list[str]` is not assignable to attribute `files` with type `Sequence[File] | list[File]` [bad-assignment]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:351:24
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:373:24
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:407:58
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:429:58
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:455:19
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:477:19
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:512:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:534:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:514:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:536:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:516:33
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:538:33
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:517:33
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:539:33
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:518:16
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:540:16
 ERROR Object of class `FunctionType` has no attribute `assert_called` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:523:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:545:9
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:525:23
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:547:23
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:531:19
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:553:19
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:548:57
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:570:57
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:554:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:576:9
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:570:57
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:592:57
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:578:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:600:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:580:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:602:35
 ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:586:46
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:608:46
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:589:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:611:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:591:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:613:35
 ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:593:16
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:615:16
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:609:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:631:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:620:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:642:35
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:636:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:658:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:638:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:660:35
 ERROR Object of class `FunctionType` has no attribute `side_effect` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:653:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:677:9
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:658:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:682:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:667:35
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:691:35
 ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:673:25
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:698:25
 ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:677:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:702:9
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:689:9
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:714:9
 ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
-   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:692:29
+   --> tests/unit_tests/core/agent/test_fc_agent_runner.py:717:29
 ERROR Missing argument `description` in function `core.agent.plugin_entities.AgentStrategyEntity.__init__` [missing-argument]
    --> tests/unit_tests/core/agent/test_plugin_entities.py:292:32
 ERROR Argument `dict[str, dict[str, str | dict[str, float | list[str]]]]` is not assignable to parameter `config` with type `AppModelConfigDict` in function `core.app.app_config.easy_ui_based_app.model_config.manager.ModelConfigManager.convert` [bad-argument-type]
@@ -3047,111 +3041,105 @@
 ERROR Expected a mapping, got object | None [invalid-argument]
   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_config_manager.py:41:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:225:22
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:261:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:236:22
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:272:22
 ERROR Object of class `object` has no attribute `files` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:337:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:368:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:380:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:424:33
 ERROR Object of type `Generator[Mapping[str, Any] | str] | Mapping[str, Any]` has no attribute `resumed` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:384:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:428:16
 ERROR Object of class `object` has no attribute `single_iteration_run` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:449:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:493:16
 ERROR Object of class `object` has no attribute `extras` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:450:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:494:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:502:18
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:546:18
 ERROR Object of class `object` has no attribute `single_loop_run` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:512:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:556:16
 ERROR Object of class `object` has no attribute `extras` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:513:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:557:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:612:32
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:656:32
 ERROR Cannot index into `Generator[Mapping[str, Any] | str]` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:615:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:659:16
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:619:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:663:16
 ERROR Argument `str | None` is not assignable to parameter `s` with type `bytearray | bytes | str` in function `json.loads` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:624:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:668:27
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:749:13
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:793:13
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:791:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:835:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:796:25
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:840:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:842:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:886:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:847:25
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:891:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:904:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:948:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:909:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:953:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState | None` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:913:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:957:33
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:977:23
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1021:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:982:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1026:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1045:27
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1089:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1050:25
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1094:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1101:31
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1145:31
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1111:41
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1155:41
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1149:35
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1193:35
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1159:45
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1203:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1223:23
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1267:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1228:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1272:21
 ERROR Object of class `NoneType` has no attribute `show_retrieve_source` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1309:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1340:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1450:33
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1459:33
 ERROR Object of type `Generator[Mapping[str, Any] | str] | Mapping[str, Any]` has no attribute `ok` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1453:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1462:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `RuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1498:33
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1516:33
 ERROR Object of type `Generator[Mapping[str, Any] | str] | Mapping[str, Any]` has no attribute `ok` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1501:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1519:16
 ERROR Argument `Literal['advanced-chat', 'agent', 'agent-chat', 'channel', 'chat', 'completion', 'rag-pipeline', 'workflow']` is not assignable to parameter `mode` with type `AppMode | SQLCoreOperations[AppMode]` in function `models.model.Conversation.__init__` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1595:14
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1614:14
 ERROR Argument `Literal['normal']` is not assignable to parameter `status` with type `ConversationStatus | SQLCoreOperations[ConversationStatus]` in function `models.model.Conversation.__init__` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1601:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1620:16
 ERROR Argument `Literal['web-app']` is not assignable to parameter `invoke_from` with type `InvokeFrom | SQLCoreOperations[InvokeFrom | None] | None` in function `models.model.Conversation.__init__` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1602:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1621:21
 WARN Identity comparison `True is False` is always False [unnecessary-comparison]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1721:42
-ERROR Object of class `ModuleType` has no attribute `TraceQueueManager` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1731:5
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1761:42
 ERROR Class member `_StubToolNode._get_error_strategy` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1763:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1792:9
 ERROR Class member `_StubToolNode._get_retry_config` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1766:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1795:9
 ERROR Class member `_StubToolNode._get_title` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1769:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1798:9
 ERROR Class member `_StubToolNode._get_description` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1772:9
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1801:9
 ERROR Class member `_StubToolNode._get_default_value_dict` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1775:9
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `WorkflowAppGenerateEntity` in function `core.app.apps.workflow.app_generator.WorkflowAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1983:37
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1804:9
 ERROR `+` is not supported between `list[str]` and `Generator[Mapping[str, Any] | str]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1993:12
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2053:12
 ERROR `+` is not supported between `list[str]` and `Mapping[str, Any]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1993:12
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2030:37
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2053:12
 ERROR `+` is not supported between `list[str]` and `Generator[Mapping[str, Any] | str]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2040:12
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2096:12
 ERROR `+` is not supported between `list[str]` and `Mapping[str, Any]` [unsupported-operation]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2040:12
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:2096:12
 ERROR Argument `list[object]` is not assignable to parameter `value` with type `Sequence[VariableBase]` in function `models.workflow.Workflow.conversation_variables` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_runner.py:62:39
 ERROR Argument `object` is not assignable to parameter `variable` with type `VariableBase` in function `models.workflow.ConversationVariable.from_variable` [bad-argument-type]
@@ -3166,158 +3154,136 @@
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:130:16
 ERROR Cannot index into `str` [bad-index]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:131:16
-ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:106:45
+ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:130:23
+ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:142:33
 ERROR `SimpleNamespace` is not assignable to attribute `_base_task_pipeline` with type `BasedGenerateTaskPipeline[Unknown]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:108:36
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:161:36
 ERROR Argument `Literal[core.app.entities.app_invoke_entities.InvokeFrom.WEB_APP]` is not assignable to parameter `invoke_from` with type `models.enums.InvokeFrom | SQLCoreOperations[models.enums.InvokeFrom | None] | None` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:125:21
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:244:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:246:23
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:298:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:300:23
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:375:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:377:23
-ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:442:37
-ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:444:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:178:21
 ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:482:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:330:45
 ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:504:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:352:45
 ERROR Argument `Literal['session']` is not assignable to parameter `type` with type `EndUserType | SQLCoreOperations[EndUserType]` in function `models.model.EndUser.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:606:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:490:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:611:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:495:23
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:617:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:501:33
 ERROR Argument `Literal[0]` is not assignable to parameter `message_unit_price` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:634:28
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:518:28
 ERROR Argument `Literal[0]` is not assignable to parameter `message_price_unit` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:635:28
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:519:28
 ERROR Argument `Literal[0]` is not assignable to parameter `answer_unit_price` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:638:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:522:27
 ERROR Argument `Literal[0]` is not assignable to parameter `answer_price_unit` with type `Decimal | SQLCoreOperations[Decimal]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:639:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:523:27
 ERROR Argument `Literal[0]` is not assignable to parameter `total_price` with type `Decimal | SQLCoreOperations[Decimal | None] | None` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:642:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:526:21
 ERROR Argument `Literal[core.app.entities.app_invoke_entities.InvokeFrom.WEB_APP]` is not assignable to parameter `invoke_from` with type `models.enums.InvokeFrom | SQLCoreOperations[models.enums.InvokeFrom | None] | None` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:644:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:528:21
 ERROR Argument `Literal['api']` is not assignable to parameter `from_source` with type `ConversationFromSource | SQLCoreOperations[ConversationFromSource]` in function `models.model.Message.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:645:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:529:21
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:734:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:618:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:742:60
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:626:60
 ERROR Argument `SimpleNamespace` is not assignable to parameter `publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._listen_audio_msg` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:787:57
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:671:57
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(e: Exception) -> ErrorStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:802:65
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:686:65
 ERROR `(**kwargs: Sequence[NodeExecutionSnapshot] | str) -> Literal['started']` is not assignable to attribute `workflow_start_to_stream_response` with type `(*, task_id: str, workflow_run_id: str, workflow_id: str, reason: WorkflowStartReason, node_execution_snapshots: Sequence[NodeExecutionSnapshot] = ...) -> WorkflowStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:817:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:701:83
 ERROR `TestAdvancedChatGenerateTaskPipeline.test_handle_output_moderation_chunk_publishes_stop._Moderation` is not assignable to attribute `output_moderation_handler` with type `OutputModeration | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:851:66
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:735:66
 ERROR `SimpleNamespace` is not assignable to attribute `queue_manager` with type `AppQueueManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:852:54
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:736:54
 ERROR `(outputs: Unknown) -> list[dict[str, str]]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:865:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:749:79
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:868:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:752:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:878:64
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:762:64
 ERROR `(outputs: Unknown) -> list[@_]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:885:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:769:79
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:886:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:770:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:896:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:780:52
 ERROR `(outputs: Unknown) -> list[@_]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:902:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:786:79
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:903:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:787:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:915:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:799:52
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:916:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:800:52
 ERROR `(**kwargs: QueueIterationStartEvent | str) -> Literal['iter_start']` is not assignable to attribute `workflow_iteration_start_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueIterationStartEvent) -> IterationNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:923:93
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:807:93
 ERROR `(**kwargs: QueueIterationNextEvent | str) -> Literal['iter_next']` is not assignable to attribute `workflow_iteration_next_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueIterationNextEvent) -> IterationNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:926:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:810:92
 ERROR `(**kwargs: QueueIterationCompletedEvent | str) -> Literal['iter_done']` is not assignable to attribute `workflow_iteration_completed_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueIterationCompletedEvent) -> IterationNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:927:97
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:811:97
 ERROR `(**kwargs: QueueLoopStartEvent | str) -> Literal['loop_start']` is not assignable to attribute `workflow_loop_start_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueLoopStartEvent) -> LoopNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:930:88
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:814:88
 ERROR `(**kwargs: QueueLoopNextEvent | str) -> Literal['loop_next']` is not assignable to attribute `workflow_loop_next_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueLoopNextEvent) -> LoopNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:931:87
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:815:87
 ERROR `(**kwargs: QueueLoopCompletedEvent | str) -> Literal['loop_done']` is not assignable to attribute `workflow_loop_completed_to_stream_response` with type `(*, task_id: str, workflow_execution_id: str, event: QueueLoopCompletedEvent) -> LoopNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:932:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:816:92
 ERROR `(**kwargs: RuntimeState | int | str | None) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(*, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: RuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:994:84
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:878:84
 ERROR `(**kwargs: QueueWorkflowPausedEvent | RuntimeState | str) -> list[str]` is not assignable to attribute `workflow_pause_to_stream_response` with type `(*, event: QueueWorkflowPausedEvent, task_id: str, graph_runtime_state: RuntimeState) -> list[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:995:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:879:83
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(e: Exception) -> ErrorStreamResponse` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1000:65
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:884:65
 ERROR `(**kwargs: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent | str) -> Literal['node_finish']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(*, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1026:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:910:89
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1061:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:945:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1067:47
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:951:47
 ERROR Argument `TestAdvancedChatGenerateTaskPipeline.test_handle_text_chunk_event_tracks_streaming_metrics._Publisher` is not assignable to parameter `tts_publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1067:68
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:951:68
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_message` with type `MessageQueueMessage | WorkflowQueueMessage | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1067:96
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:951:96
 ERROR `TestAdvancedChatGenerateTaskPipeline.test_handle_output_moderation_chunk_appends_token._Moderation` is not assignable to attribute `output_moderation_handler` with type `OutputModeration | None` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1088:66
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:972:66
 ERROR `(**kwargs: Unknown) -> Literal['replace']` is not assignable to attribute `message_replace_to_stream_response` with type `(answer: str, reason: str = '') -> MessageReplaceStreamResponse` [bad-assignment]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1119:78
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:1003:78
 ERROR `(**kwargs: QueueHumanInputFormFilledEvent | str) -> Literal['filled']` is not assignable to attribute `human_input_form_filled_to_stream_response` with type `(*, event: QueueHumanInputFormFilledEvent, task_id: str) -> HumanInputFormFilledResponse` [bad-assignment]
-    --> tests/unit\n\n... (truncated) ...

[codecov]

Codecov Report

❌ Patch coverage is 97.64293% with 47 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.80%. Comparing base (4c521a3) to head (edeef21).

Files with missing lines	Patch %	Lines
...nix/src/dify_trace_arize_phoenix/_arize_project.py	91.57%	5 Missing and 3 partials ⚠️
.../unit_tests/src/dify_trace_aliyun/test_provider.py	95.39%	6 Missing and 1 partial ⚠️
...ests/src/dify_trace_arize_phoenix/test_provider.py	97.96%	6 Missing and 1 partial ⚠️
api/domain/tracing/normalization.py	92.10%	3 Missing and 3 partials ⚠️
api/extensions/tracing.py	92.59%	2 Missing and 2 partials ⚠️
...e/app/apps/advanced_chat/generate_task_pipeline.py	72.72%	3 Missing ⚠️
...e-phoenix/src/dify_trace_arize_phoenix/provider.py	98.08%	2 Missing and 1 partial ⚠️
api/controllers/console/agent/tracing.py	71.42%	2 Missing ⚠️
api/core/telemetry/gateway.py	85.71%	1 Missing and 1 partial ⚠️
...logstore_api_workflow_node_execution_repository.py	77.77%	2 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #44673      +/-   ##
==========================================
- Coverage   89.85%   89.80%   -0.05%     
==========================================
  Files        6249     6240       -9     
  Lines      398125   391464    -6661     
  Branches    80223    79499     -724     
==========================================
- Hits       357737   351563    -6174     
+ Misses      34078    33809     -269     
+ Partials     6310     6092     -218     

Flag	Coverage Δ
api	89.63% <97.64%> (-0.09%)	⬇️
cli	94.26% <ø> (ø)
dify-ui	92.64% <ø> (ø)
web	89.83% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.