From d072acabce1c2ed2e60f26d2a91056bc70010296 Mon Sep 17 00:00:00 2001 From: Thierry Schellenbach Date: Thu, 19 Nov 2026 14:03:59 -0700 Subject: [PATCH] feat(tools): show what a tool call is doing with display_title A tool call's state (tool_call.started, .progress, .finished) carries display_title, the title its tool was declared with, or "Searching the web" for the router's own search, the default a Chat reply's tool step already used (conversation.ToolTitle). A worker's host_tools frame takes display_title per tool. Every SDK can declare one: Go's tools.DisplayTitle, and the Python, JavaScript, Kotlin, Swift, Dart, .NET, Ruby, Rust and PHP registries, which also get lighter ways to describe parameters. The generated clients are regenerated; Swift's generates no ToolCallState, so it is unchanged. Co-authored-by: Cursor --- .claude/skills/sdk/SKILL.md | 1 + acceleration/api/openapi.yaml | 5 +- acceleration/internal/api/dispatch_test.go | 15 ++ acceleration/internal/api/dispatchws.go | 8 +- acceleration/internal/api/handwritten.go | 3 +- acceleration/internal/api/sessionws.go | 25 +- acceleration/internal/conversation/parts.go | 17 +- acceleration/internal/dispatch/tools.go | 2 + acceleration/internal/pricing/list.go | 4 +- acceleration/internal/session/hosted_tools.go | 2 +- acceleration/internal/session/manager.go | 1 + .../internal/session/tool_call_state.go | 21 ++ .../internal/session/tool_call_state_test.go | 17 ++ .../_generated/api/default/watch_session.py | 28 ++- .../_generated/models/tool_call_state.py | 11 + sdks/dart/stream_agents/CHANGELOG.md | 1 + sdks/dart/stream_agents/README.md | 19 ++ .../stream_agents/lib/src/core/tools.dart | 32 +++ .../stream_agents/test/core/client_test.dart | 38 +++ .../stream_agents/test/core/session_test.dart | 77 ++++++ .../Generated/Models.cs | 232 +++++++++++++++--- .../src/GetStream.VisionAgents/Tools.cs | 25 +- .../ToolCallTests.cs | 57 +++++ sdks/go/acceleration/generated.go | 3 + sdks/go/stream/accelerated.go | 4 + sdks/go/stream/dispatch.go | 3 + sdks/go/stream/dispatch_test.go | 23 ++ sdks/go/tools/options.go | 13 + sdks/go/tools/registry.go | 3 + sdks/js/README.md | 23 ++ sdks/js/src/generated/api.ts | 4 +- sdks/js/src/react.ts | 54 +++- sdks/js/tests/use-tool.test.ts | 164 +++++++++++++ sdks/kotlin/README.md | 14 ++ .../agents/android/core/AgentTool.kt | 102 ++++++++ .../agents/android/core/AgentSessionTest.kt | 24 ++ .../agents/android/core/AgentToolTest.kt | 66 +++++ sdks/php/README.md | 15 ++ sdks/php/src/Generated/ToolCallState.php | 6 + sdks/php/src/Tools.php | 20 ++ sdks/php/tests/Unit/ToolCallTest.php | 46 ++++ .../core/llm/function_registry.py | 64 ++++- .../vision_agents/core/llm/tool_arguments.py | 20 +- sdks/ruby/README.md | 11 + .../getstream/vision_agents/generated/spec.rb | 2 +- .../ruby/lib/getstream/vision_agents/tools.rb | 39 ++- sdks/ruby/test/tool_call_test.rb | 45 ++++ sdks/rust/Cargo.lock | 65 ++++- sdks/rust/Cargo.toml | 1 + sdks/rust/src/lib.rs | 2 + sdks/rust/src/tools.rs | 40 +++ sdks/rust/tests/session.rs | 99 ++++++++ .../core/Sources/StreamAgents/AgentTool.swift | 73 ++++++ .../StreamAgentsTests/AgentSessionTests.swift | 45 +++- tests/test_function_calling.py | 133 +++++++++- 55 files changed, 1771 insertions(+), 96 deletions(-) create mode 100644 sdks/js/tests/use-tool.test.ts create mode 100644 sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentToolTest.kt diff --git a/.claude/skills/sdk/SKILL.md b/.claude/skills/sdk/SKILL.md index 9bac4e05ae..ee3fbb64be 100644 --- a/.claude/skills/sdk/SKILL.md +++ b/.claude/skills/sdk/SKILL.md @@ -928,3 +928,4 @@ Connections (AI-3376): Go has `client.Connections()` for the accounts the app an The session socket sends `reasoning_delta` (`turn_id`, `text`): the next piece of the model's thinking on a written reply, which a kept conversation also streams into the reply's Chat message and an incognito one has nowhere else to show. JavaScript folds it, with `tool_started`, `tool_updated` (not `verbatim`) and `tool_ran`, into each agent turn's `steps` in `sdks/js/src/conversation.ts`: a round of thinking `r1`, `r2`... opens on a delta after anything else and completes when the turn writes text or UI, calls a tool, answers or is cut off; a call runs until `tool_ran`, failed when it carries an `error`, its summary the last non-verbatim update. A turn is `done` on `responded` with no `pending_work`, or `interrupted`. `AgentChat` shows a session with no `conversationId` from these turns (`sdks/js/src/turn-messages.ts`): the agent's turns between two of the person's messages are one reply, shaped as the channel's messages (top level in stream-chat 9, under `custom` in 10) with `ai_reasoning` and `ai_tool_call` steps, rounds numbered across the reply, its whole thinking given to `reasoning`, and a call waiting in `approvals` shown `awaiting_approval` with its question and the session's `user_id` as `target_user_id`. The clients that carry the socket's description (JavaScript types, Python `plugins/stream`) are regenerated. Swift does the same: `Conversation` folds the frames into `Turn.steps` and `Turn.isDone`, and `TurnMessage.messages(of:approvals:userID:isConnected:)` (`session.turnMessages`) builds the replies, each step an attachment type and JSON payload for `AIMessagePart.parts(from:)`, the session's user now `Session.userID`; `swift_assistant`'s `AgentChat` shows an incognito session from them. Other SDKs need to update: Kotlin and Dart, whose AI components read the same steps, should fold the same frames into their `Conversation` turns and show a session with no channel through their transports; the server SDKs need nothing. The session socket sends `reasoning_delta` (`turn_id`, `text`): the next piece of the model's thinking on a written reply, which a kept conversation also streams into the reply's Chat message and an incognito one has nowhere else to show. JavaScript folds it, with `tool_started`, `tool_updated` (not `verbatim`) and `tool_ran`, into each agent turn's `steps` in `sdks/js/src/conversation.ts`: a round of thinking `r1`, `r2`... opens on a delta after anything else and completes when the turn writes text or UI, calls a tool, answers or is cut off; a call runs until `tool_ran`, failed when it carries an `error`, its summary the last non-verbatim update. A turn is `done` on `responded` with no `pending_work`, or `interrupted`. `AgentChat` shows a session with no `conversationId` from these turns (`sdks/js/src/turn-messages.ts`): the agent's turns between two of the person's messages are one reply, shaped as the channel's messages (top level in stream-chat 9, under `custom` in 10) with `ai_reasoning` and `ai_tool_call` steps, rounds numbered across the reply, its whole thinking given to `reasoning`, and a call waiting in `approvals` shown `awaiting_approval` with its question and the session's `user_id` as `target_user_id`. The clients that carry the socket's description (JavaScript types, Python `plugins/stream`) are regenerated. Other SDKs need to update: Swift, Kotlin and Dart, whose AI components read the same steps, should fold the same frames into their `Conversation` turns and show a session with no channel through their transports; the server SDKs need nothing. An agent's UI instructions are kept apart from its instructions, so a session can turn UI off: `AgentConfig`, `AgentConfigRequest`, `AgentConfigPatch` and `SyncAgentRequest` have `ui_instructions` (an additive migration), and `CreateSessionRequest` has `ui`, a boolean a device may send. The router tells the model `ui_instructions` after the instructions (a backend's own session instructions included) unless the session was opened with `ui: false`, which reopening or forking it keeps (`agent_sessions.no_ui`). A sync sends the directory whole, so a sync without `ui_instructions` removes them; a PUT that leaves them out keeps them, so a client that does not know the field does not take the UI away. Go's `Load` compiles agent.yaml's `ui:` into `Folder.UIInstructions` and `syncFolder` sends it as `ui_instructions`; the folder's hash takes it in only when there is UI, so a directory without `ui:` is not synced again, and one with it is synced once more, which moves the UI out of the stored instructions. A router older than the field takes the sync and drops it, so `syncFolder` does not trust a stamp covering UI that the stored config lacks, and syncs again, and the router writes a sync of the same hash that declares UI the config lacks rather than finding it unchanged. Deploy the router before releasing an SDK that sends the field. JavaScript does the same: `loadFolder` puts the compiled `ui:` in `Folder.uiInstructions`, `Agent.sync` sends it as `ui_instructions` (left out without `ui:`, so such a directory keeps its fingerprint), and a stamp covering UI the stored config lacks is synced again. When a sync's answer has no `ui_instructions` though it sent them (a router older than the field), JavaScript syncs once more with the UI joined onto `instructions`, as before, under that body's own fingerprint and without writing the stamp, so the agent keeps its UI and the next sync moves it apart once the router knows the field. That fallback cannot honour `ui: false`: the UI is in `instructions` then, which every session is told, so the Interactive UI switch needs a router that knows the field and a sync it stored apart. Go only syncs again there, so an older router leaves a Go-synced agent with no UI, and Go should fall back the same way. `Agent`'s `instructions` option replaces instructions.md only: the folder's `ui:` is still sent as `ui_instructions`, as in Go. `react_assistant`'s settings have the same Interactive UI switch as `swift_assistant`'s, which opens new conversations with `ui: false`. The shared fixtures are unchanged: they are what the model is told, the instructions and the UI joined by a blank line, as the router joins them. Every generated client is regenerated (Go, JavaScript types, Python `plugins/stream`, Swift, Rust's `types.rs`); Rust compiling and sending `ui:` is still to do, below. Other SDKs need to update: Python, .NET, Ruby, Rust and PHP, when they read `ui:`, should send it as `ui_instructions` the same way; Swift has `SessionOptions.ui`, which `swift_assistant`'s Settings switch sets, and JavaScript's `sessions.create` passes `ui` through with the regenerated types; Python, Kotlin, Dart and the other server SDKs should offer `ui` on their create-session options, so an app that shows no UI can say so. +A tool call's state (`ToolCallState`, the `tool_call.started`, `tool_call.progress` and `tool_call.finished` frames) carries `display_title`: what the call is doing, as its tool was declared with (a session's own tool's `display_title`, or a hosted tool's), or "Searching the web" for the router's own search, the same default a Chat reply's tool step uses (`conversation.ToolTitle`). A worker's `host_tools` frame takes `display_title` on each tool, beside `interruptible` and `timeout_ms`. Go has `tools.DisplayTitle(title)`, sent on `host_tools` and on a session's own tools. The generated clients are regenerated (Go, JavaScript types, Python `plugins/stream`, Ruby, PHP, .NET); Swift's generates no `ToolCallState`, so it is unchanged. JavaScript needs nothing: its `host_tools` sends `tools.declared()`, `displayTitle` included, which the router now keeps. Other SDKs need to update: Python, .NET, Ruby, Rust and PHP should send their tool's display title on `host_tools`, their registries taking one where they do not yet; Swift, Kotlin and Dart read `display_title` off the tool call state once regenerated. diff --git a/acceleration/api/openapi.yaml b/acceleration/api/openapi.yaml index 0ab8b445bd..c0d151664c 100644 --- a/acceleration/api/openapi.yaml +++ b/acceleration/api/openapi.yaml @@ -11013,6 +11013,9 @@ components: description: When a person allowed the call. format: date-time type: string + display_title: + description: 'What the call is doing, in words for the people in the conversation, such as "Checking the weather": the display_title its tool was declared with, or "Searching the web" for the router''s own search. Absent when the tool declared none.' + type: string duration_ms: description: From started_at to this change; on tool_call.finished, how long the call took. format: int64 @@ -18950,7 +18953,7 @@ paths: `connector_unavailable` names an optional connector binding the session opened without: name (its alias), connector_id and reason, one of no_selection, shared_session, caller_unverified, connection_unavailable, provider_mismatch, needs_reauthorization, credential_rejected (the provider rejected the token or key a bearer or api_key connection holds, or the connection reads a connector revision marked broken; only saving its credentials again fixes it, so no login is offered), not_connected, open_failed, tool_unavailable and selection_dropped (a fork's or a reopened chat's selection for an alias its config no longer declares). Every watcher is sent each one when it attaches. `connector_scope_required` says a connector tool call was refused because the caller's own connection lacks access the provider asked for (insufficient_scope or a claims challenge), and a step-up consent was begun for it: name (the binding's alias), connector_id, connection_id, scopes (what the provider asked for, empty for a claims challenge), authorization_id, launch_url, handoff_token and expires_at. A page of the session's return_url's origin, or the dashboard for a session created without one, opens launch_url in a popup and posts it handoff_token, as for a createAuthorization with that return_url: the launch page takes the handoff from that origin only, and the consent goes back to the session's return_url, or else to the dashboard. The old grant keeps working until the step-up succeeds, and the same call works afterwards in the same session. While that step-up is open, calls refused for the same access send no second event. Persistent text sessions also emit `conversation_updated` with conversation_id and a complete message snapshot: id, request_id, question_id, role, text, state, response_started_at, state_started_at, finished_at, duration_ms, saved, persistence_error, refused (true on a reply the prompt injection screen refused) and attachments. Each tool_calling attachment has tool_call_id, name, title, status, phase, summary, immutable started_at, execution_started_at, finished_at and duration_ms. A connector_authorization attachment asks the end user to connect a connector binding the reply needed with their own account, with name (the binding's alias), connector_id, connection_id, authorization_id, title, launch_url, handoff_token and expires_at: a page of the session's return_url's origin, or the dashboard for a session created without one, opens launch_url in a popup and posts it handoff_token, as for a createAuthorization with that return_url, and the consent goes back to the session's return_url, or else to the dashboard. Once the user finishes that login the message is sent again with status connected and no handoff_token, and the agent carries on by itself. The call that needed the login waits for it on its own ai_tool_call step, in parts: status awaiting_authorization, target_user_id the caller, and an authorization of type oauth with provider (the connector id), provider_name, title, message, url (the launch page), handoff_token and expires_at. A page of the return_url's origin opens url and posts it handoff_token; a native app opens url#handoff_token= in its web authentication sheet. Connected, the step is running again with authorization.status connected and no handoff_token, and the call runs in the same reply. A tool_approval naming its tool_call_id with allowed false, or the person refusing at the provider, declines the login (declined); a consent that ended otherwise without connecting, such as with another account, is failed, and one not finished by expires_at is expired. Each way the step is cancelled, authorization.status says how, and the call runs nothing. Activity states are thinking, queued, tools, writing, completed, failed and cancelled. tool_started includes tool_call_id, tool, turn_id and started_at, and pre_speech when the tool's connector binding sets one in its policy; tool_ran also includes tool_call_id, and files when the tool was run_code, as on task_settled. - `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, waiting for a person's approval, success, failed or cancelled), progress (the last `tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id. + `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, waiting for a person's approval, success, failed or cancelled), display_title (what the call is doing, as its tool was declared with), progress (the last `tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id. A respond command carrying request_id emits command_accepted with a nested command receipt (request_id, user_message_id, assistant_message_id, state, duplicate). Personal persistent text sessions require this ID. A retry with the same text and images returns the existing IDs without invoking the model again; reuse with different content emits an error. Its images are data URIs, at most 8, uploaded to the conversation's channel and shown on the person's message. On a session a dispatch worker opened for a message handed out from a channel, that message is already there: nothing is written or uploaded for it, user_message_id is empty, and a reused id is answered as a duplicate even with other text or images. After restart an interrupted command is reported, not rerun. An `interrupt` command carrying `request_id` stops that command and emits `command_stopped` with its terminal receipt. A stop arriving after its command finished replays that command's receipt and leaves the command running now alone; an unknown command is reported as an error. Without `request_id` the frame stops whichever reply is current, which is what a caller with no command to name means by it. A `decision` frame is one judgement the conversation made, carrying the same fields as a CallEvent. Together they are why the call went the way it did, and they are also written down, so a finished call replays them from `/v1/agents/calls/{id}/events`. diff --git a/acceleration/internal/api/dispatch_test.go b/acceleration/internal/api/dispatch_test.go index 54c3d0b9c8..d750d9aafa 100644 --- a/acceleration/internal/api/dispatch_test.go +++ b/acceleration/internal/api/dispatch_test.go @@ -321,6 +321,21 @@ func (s *DispatchSuite) TestHostedToolsAreKeptForTheAgentTheWorkerNames() { s.Equal(30*time.Second, timeout, "a worker that names no timeout takes the session's default") } +func (s *DispatchSuite) TestAHostedToolKeepsWhatItsCallsAreShownDoing() { + s.serverClient.agentNamed("stream-support") + worker := s.worker() + + s.Require().NoError(worker.WriteJSON(frame{ + "type": "host_tools", "agent_id": "stream-support", + "tools": []frame{{"name": "weather_lookup", "description": "Look up the weather", "display_title": " Checking the weather "}}, + })) + + s.Require().Equal("hosting", s.next(worker)["type"]) + tools, _ := s.dispatch.HostedTools(s.customerID(), "stream-support") + s.Require().Len(tools, 1) + s.Equal("Checking the weather", tools[0].DisplayTitle) +} + func (s *DispatchSuite) TestAHostedToolsOwnTimeoutReplacesTheWorkers() { s.serverClient.agentNamed("stream-support") worker := s.worker() diff --git a/acceleration/internal/api/dispatchws.go b/acceleration/internal/api/dispatchws.go index 582d3274b0..a0101461a5 100644 --- a/acceleration/internal/api/dispatchws.go +++ b/acceleration/internal/api/dispatchws.go @@ -380,9 +380,10 @@ type hostedTool struct { Name string `json:"name"` Description string `json:"description"` Parameters map[string]any `json:"parameters"` - // Interruptible and TimeoutMs mean what they do on a session's own tool. - Interruptible bool `json:"interruptible"` - TimeoutMs int `json:"timeout_ms"` + // DisplayTitle, Interruptible and TimeoutMs mean what they do on a session's own tool. + DisplayTitle string `json:"display_title"` + Interruptible bool `json:"interruptible"` + TimeoutMs int `json:"timeout_ms"` // Approval is the question a voice caller is asked before each call runs, as on a // session's own tool. Only its title is asked. Approval *hostedApproval `json:"approval"` @@ -423,6 +424,7 @@ func (s *Server) hostTools(worker *dispatch.Worker, agentID string, everyAgent b } hosted := dispatch.Tool{ Name: tool.Name, Description: tool.Description, Parameters: tool.Parameters, + DisplayTitle: strings.TrimSpace(tool.DisplayTitle), Interruptible: tool.Interruptible, Timeout: time.Duration(tool.TimeoutMs) * time.Millisecond, } if tool.Approval != nil { diff --git a/acceleration/internal/api/handwritten.go b/acceleration/internal/api/handwritten.go index 81e5a1e6e2..b49bd5263e 100644 --- a/acceleration/internal/api/handwritten.go +++ b/acceleration/internal/api/handwritten.go @@ -126,7 +126,8 @@ func documentHandWritten(api huma.API) { "tool_ran also includes tool_call_id, and files when the tool was run_code, as on task_settled.\n" + "`tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a " + "call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status " + - "(in_progress, waiting for a person's approval, success, failed or cancelled), progress (the last " + + "(in_progress, waiting for a person's approval, success, failed or cancelled), display_title (what " + + "the call is doing, as its tool was declared with), progress (the last " + "`tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, " + "approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id.\n" + "A respond command carrying request_id emits command_accepted with a nested command " + diff --git a/acceleration/internal/api/sessionws.go b/acceleration/internal/api/sessionws.go index 4937d38bdc..1a3044d527 100644 --- a/acceleration/internal/api/sessionws.go +++ b/acceleration/internal/api/sessionws.go @@ -1015,7 +1015,7 @@ func toolStateFrame(changed session.ToolCallChanged) frame { "started_at": state.StartedAt, "duration_ms": state.Duration.Milliseconds(), } - for key, value := range map[string]string{"progress": state.Progress, "error": state.Error} { + for key, value := range map[string]string{"display_title": state.DisplayTitle, "progress": state.Progress, "error": state.Error} { if value != "" { rendered[key] = value } @@ -1030,17 +1030,18 @@ func toolStateFrame(changed session.ToolCallChanged) frame { // ToolCallState is one change to a tool call's state, sent to every watcher of the session. type ToolCallState struct { - Type ToolCallFrameType `json:"type"` - ToolCallID string `json:"tool_call_id" doc:"The call, as tool_call, tool_started and tool_ran name it."` - Tool string `json:"tool"` - TurnID string `json:"turn_id"` - Status ToolCallStatus `json:"status"` - Progress string `json:"progress,omitempty" doc:"The last thing said about the call while it runs, such as \"Searching for flights\": a tool_update, the connector binding's pre_speech, or the question it waits on."` - StartedAt time.Time `json:"started_at"` - DurationMS int64 `json:"duration_ms" doc:"From started_at to this change; on tool_call.finished, how long the call took."` - ApprovedAt *time.Time `json:"approved_at,omitempty" doc:"When a person allowed the call."` - RejectedAt *time.Time `json:"rejected_at,omitempty" doc:"When a person declined the call, or its wait for them passed."` - Error string `json:"error,omitempty" doc:"Why a failed call failed, or what a declined one shows, such as \"Not allowed in time\"."` + Type ToolCallFrameType `json:"type"` + ToolCallID string `json:"tool_call_id" doc:"The call, as tool_call, tool_started and tool_ran name it."` + Tool string `json:"tool"` + TurnID string `json:"turn_id"` + Status ToolCallStatus `json:"status"` + DisplayTitle string `json:"display_title,omitempty" doc:"What the call is doing, in words for the people in the conversation, such as \"Checking the weather\": the display_title its tool was declared with, or \"Searching the web\" for the router's own search. Absent when the tool declared none."` + Progress string `json:"progress,omitempty" doc:"The last thing said about the call while it runs, such as \"Searching for flights\": a tool_update, the connector binding's pre_speech, or the question it waits on."` + StartedAt time.Time `json:"started_at"` + DurationMS int64 `json:"duration_ms" doc:"From started_at to this change; on tool_call.finished, how long the call took."` + ApprovedAt *time.Time `json:"approved_at,omitempty" doc:"When a person allowed the call."` + RejectedAt *time.Time `json:"rejected_at,omitempty" doc:"When a person declined the call, or its wait for them passed."` + Error string `json:"error,omitempty" doc:"Why a failed call failed, or what a declined one shows, such as \"Not allowed in time\"."` } func (*ToolCallState) TransformSchema(_ huma.Registry, schema *huma.Schema) *huma.Schema { diff --git a/acceleration/internal/conversation/parts.go b/acceleration/internal/conversation/parts.go index 44b4cb3596..409bbfc0f7 100644 --- a/acceleration/internal/conversation/parts.go +++ b/acceleration/internal/conversation/parts.go @@ -133,6 +133,16 @@ type ToolApproval struct { DeclineTitle string } +// ToolTitle is what a call of the tool called name is shown doing: title, the one its tool +// declared, or for the runtime's own search, which has no caller to title it, "Searching the +// web". Other untitled calls are left to the clients, which name them from the tool. +func ToolTitle(name, title string) string { + if title == "" && (name == "search" || name == "web_search") { + return "Searching the web" + } + return title +} + // DescribeTools records the caller's tools for this session. func (c *Conversation) DescribeTools(tools map[string]ToolDisplay) { c.mu.Lock() @@ -191,12 +201,7 @@ func (c *Conversation) called(m *Message, e toolCall) { } started := e.startedAt.UTC() part := Part{Type: partToolCall, V: partVersion, ID: e.id, Status: "running", Name: e.name, - DisplayTitle: display.Title, Executor: "server", StartedAt: &started} - // The runtime's own search has no caller to title it. Other untitled calls are left to - // the clients, which name them from the tool. - if part.DisplayTitle == "" && (e.name == "search" || e.name == "web_search") { - part.DisplayTitle = "Searching the web" - } + DisplayTitle: ToolTitle(e.name, display.Title), Executor: "server", StartedAt: &started} if display.Client { part.Executor = "client" record := c.data.Commands[m.CommandID] diff --git a/acceleration/internal/dispatch/tools.go b/acceleration/internal/dispatch/tools.go index ded603f7d5..679feea815 100644 --- a/acceleration/internal/dispatch/tools.go +++ b/acceleration/internal/dispatch/tools.go @@ -30,6 +30,8 @@ type Tool struct { Description string // Parameters is a JSON Schema object describing the arguments. Parameters map[string]any + // DisplayTitle is what a call is doing, in words for the people in the conversation. + DisplayTitle string `json:",omitempty"` // Interruptible says the caller interrupting cancels a call in flight. Interruptible bool `json:",omitempty"` // Timeout is how long one call may take, in place of what the worker gave all its diff --git a/acceleration/internal/pricing/list.go b/acceleration/internal/pricing/list.go index e30f868b7e..217b72e390 100644 --- a/acceleration/internal/pricing/list.go +++ b/acceleration/internal/pricing/list.go @@ -19,7 +19,9 @@ type key struct { byok bool } -func keyOf(item Item) key { return key{item.Modality, item.Provider, item.Model, item.Region, item.BYOK} } +func keyOf(item Item) key { + return key{item.Modality, item.Provider, item.Model, item.Region, item.BYOK} +} func keyOfEntry(entry Entry) key { return key{entry.Modality, entry.Provider, entry.Model, entry.Region, entry.BYOK} diff --git a/acceleration/internal/session/hosted_tools.go b/acceleration/internal/session/hosted_tools.go index f246ad5977..217d8eadc6 100644 --- a/acceleration/internal/session/hosted_tools.go +++ b/acceleration/internal/session/hosted_tools.go @@ -61,7 +61,7 @@ func (m *Manager) hostedTools(spec Spec, loading toolLoading, sessionID string, hosted[tool.Name] = agentID offered := harness.Tool{ Name: tool.Name, Description: tool.Description, Parameters: tool.Parameters, - Interruptible: tool.Interruptible, Timeout: tool.Timeout, + DisplayTitle: tool.DisplayTitle, Interruptible: tool.Interruptible, Timeout: tool.Timeout, } if tool.Approval != "" { offered.Approval = &harness.ToolApproval{Title: tool.Approval} diff --git a/acceleration/internal/session/manager.go b/acceleration/internal/session/manager.go index 9ed3b9eebd..68e4bfd5ca 100644 --- a/acceleration/internal/session/manager.go +++ b/acceleration/internal/session/manager.go @@ -708,6 +708,7 @@ func (m *Manager) Create(ctx context.Context, spec Spec) (_ *Session, failure er created.Close() } created.toolCalls.approves = created.approves + created.toolCalls.titles = created.toolTitle created.tools = newBridge( time.Duration(spec.ToolTimeoutMs)*time.Millisecond, spec.Tools, diff --git a/acceleration/internal/session/tool_call_state.go b/acceleration/internal/session/tool_call_state.go index 61264eaf38..dec02d7280 100644 --- a/acceleration/internal/session/tool_call_state.go +++ b/acceleration/internal/session/tool_call_state.go @@ -4,10 +4,12 @@ import ( "context" "encoding/json" "errors" + "slices" "sync" "time" "github.com/GetStream/Vision-Agents/acceleration/internal/agent" + persistent "github.com/GetStream/Vision-Agents/acceleration/internal/conversation" ) // ToolCallStatus is where one tool call stands. @@ -38,6 +40,9 @@ type ToolCallState struct { Tool string TurnID string Status ToolCallStatus + // DisplayTitle is what the call is doing, in words for the people in the conversation, + // as its tool declared it. Empty when the tool declared none. + DisplayTitle string // Progress is the last thing said about the call while it runs, such as "Searching for // flights", or the question it waits on. Progress string @@ -61,6 +66,8 @@ type ToolCallChanged struct { type toolCalls struct { // approves reports whether a call to a tool waits for a person before it runs. approves func(tool string) bool + // titles is what a call to a tool is shown doing, or empty for nothing. + titles func(tool string) string mu sync.Mutex calls map[string]*ToolCallState @@ -87,6 +94,9 @@ func (t *toolCalls) observe(event agent.Event) (ToolCallChanged, bool) { if t.approves != nil && t.approves(typed.Tool) { state.Status = ToolCallWaiting } + if t.titles != nil { + state.DisplayTitle = t.titles(typed.Tool) + } if waiting, ok := t.early[typed.ID]; ok { delete(t.early, typed.ID) state.Status, state.Progress = ToolCallWaiting, waiting.Question @@ -194,3 +204,14 @@ func (s *Session) approves(tool string) bool { } return false } + +// toolTitle is what a call to the tool is shown doing, as whoever declared it said: the +// caller, or the worker hosting it. +func (s *Session) toolTitle(tool string) string { + all := slices.Concat(s.declaredTools(), s.otherTools) + title := "" + if at := slices.IndexFunc(all, named(tool)); at >= 0 { + title = all[at].DisplayTitle + } + return persistent.ToolTitle(tool, title) +} diff --git a/acceleration/internal/session/tool_call_state_test.go b/acceleration/internal/session/tool_call_state_test.go index 1ccb5081e1..5842ff0ae4 100644 --- a/acceleration/internal/session/tool_call_state_test.go +++ b/acceleration/internal/session/tool_call_state_test.go @@ -10,6 +10,7 @@ import ( "github.com/stretchr/testify/suite" "github.com/GetStream/Vision-Agents/acceleration/internal/agent" + "github.com/GetStream/Vision-Agents/acceleration/internal/harness" ) type ToolCallStateSuite struct { @@ -149,3 +150,19 @@ func (s *ToolCallStateSuite) TestAConnectorCallHeldBeforeItsStartCameThroughStar s.Equal(ToolCallWaiting, started.State.Status) s.Equal("Allow github to run merge?", started.State.Progress) } + +func (s *ToolCallStateSuite) TestACallIsShownDoingWhatItsToolWasDeclaredWith() { + session := &Session{ + spec: Spec{Tools: []harness.Tool{{Name: "get_location", DisplayTitle: "Checking where you are"}}}, + otherTools: []harness.Tool{{Name: "weather_lookup", DisplayTitle: "Checking the weather"}, {Name: "lookup_order"}}, + } + s.calls.titles = session.toolTitle + + s.Equal("Checking where you are", s.start("call-1", "get_location").State.DisplayTitle) + s.Equal("Checking the weather", s.start("call-2", "weather_lookup").State.DisplayTitle, "a hosted tool's title") + s.Equal("Searching the web", s.start("call-3", "web_search").State.DisplayTitle) + s.Empty(s.start("call-4", "lookup_order").State.DisplayTitle, "a tool that declared no title") + finished, ok := s.apply(agent.ToolRan{ID: "call-2", Tool: "weather_lookup", Result: "Sunny"}) + s.Require().True(ok) + s.Equal("Checking the weather", finished.State.DisplayTitle, "the title stays on the finished call") +} diff --git a/plugins/stream/vision_agents/plugins/stream/_generated/api/default/watch_session.py b/plugins/stream/vision_agents/plugins/stream/_generated/api/default/watch_session.py index 083b9d8efb..499fa04442 100644 --- a/plugins/stream/vision_agents/plugins/stream/_generated/api/default/watch_session.py +++ b/plugins/stream/vision_agents/plugins/stream/_generated/api/default/watch_session.py @@ -179,9 +179,10 @@ def sync_detailed( task_settled. `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, - waiting for a person's approval, success, failed or cancelled), progress (the last `tool_update`, - the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, - rejected_at and error. Each carries the whole state: keep the last per tool_call_id. + waiting for a person's approval, success, failed or cancelled), display_title (what the call is + doing, as its tool was declared with), progress (the last `tool_update`, the binding's pre_speech or + the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries + the whole state: keep the last per tool_call_id. A respond command carrying request_id emits command_accepted with a nested command receipt (request_id, user_message_id, assistant_message_id, state, duplicate). Personal persistent text sessions require this ID. A retry with the same text and images returns the existing IDs without @@ -399,9 +400,10 @@ def sync( task_settled. `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, - waiting for a person's approval, success, failed or cancelled), progress (the last `tool_update`, - the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, - rejected_at and error. Each carries the whole state: keep the last per tool_call_id. + waiting for a person's approval, success, failed or cancelled), display_title (what the call is + doing, as its tool was declared with), progress (the last `tool_update`, the binding's pre_speech or + the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries + the whole state: keep the last per tool_call_id. A respond command carrying request_id emits command_accepted with a nested command receipt (request_id, user_message_id, assistant_message_id, state, duplicate). Personal persistent text sessions require this ID. A retry with the same text and images returns the existing IDs without @@ -614,9 +616,10 @@ async def asyncio_detailed( task_settled. `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, - waiting for a person's approval, success, failed or cancelled), progress (the last `tool_update`, - the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, - rejected_at and error. Each carries the whole state: keep the last per tool_call_id. + waiting for a person's approval, success, failed or cancelled), display_title (what the call is + doing, as its tool was declared with), progress (the last `tool_update`, the binding's pre_speech or + the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries + the whole state: keep the last per tool_call_id. A respond command carrying request_id emits command_accepted with a nested command receipt (request_id, user_message_id, assistant_message_id, state, duplicate). Personal persistent text sessions require this ID. A retry with the same text and images returns the existing IDs without @@ -832,9 +835,10 @@ async def asyncio( task_settled. `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, - waiting for a person's approval, success, failed or cancelled), progress (the last `tool_update`, - the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, - rejected_at and error. Each carries the whole state: keep the last per tool_call_id. + waiting for a person's approval, success, failed or cancelled), display_title (what the call is + doing, as its tool was declared with), progress (the last `tool_update`, the binding's pre_speech or + the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries + the whole state: keep the last per tool_call_id. A respond command carrying request_id emits command_accepted with a nested command receipt (request_id, user_message_id, assistant_message_id, state, duplicate). Personal persistent text sessions require this ID. A retry with the same text and images returns the existing IDs without diff --git a/plugins/stream/vision_agents/plugins/stream/_generated/models/tool_call_state.py b/plugins/stream/vision_agents/plugins/stream/_generated/models/tool_call_state.py index 799f393ccd..be6fa68582 100644 --- a/plugins/stream/vision_agents/plugins/stream/_generated/models/tool_call_state.py +++ b/plugins/stream/vision_agents/plugins/stream/_generated/models/tool_call_state.py @@ -37,6 +37,9 @@ class ToolCallState: progress (str | Unset): The last thing said about the call while it runs, such as "Searching for flights": a tool_update, the connector binding's pre_speech, or the question it waits on. rejected_at (datetime.datetime | Unset): When a person declined the call, or its wait for them passed. + display_title (str | Unset): What the call is doing, in words for the people in the conversation, such as + "Checking the weather": the display_title its tool was declared with, or "Searching the web" for the router's + own search. Absent when the tool declared none. """ duration_ms: int @@ -50,6 +53,7 @@ class ToolCallState: error: str | Unset = UNSET progress: str | Unset = UNSET rejected_at: datetime.datetime | Unset = UNSET + display_title: str | Unset = UNSET additional_properties: dict[str, Any] = _attrs_field(init=False, factory=dict) def to_dict(self) -> dict[str, Any]: @@ -79,6 +83,8 @@ def to_dict(self) -> dict[str, Any]: if not isinstance(self.rejected_at, Unset): rejected_at = self.rejected_at.isoformat() + display_title = self.display_title + field_dict: dict[str, Any] = {} field_dict.update(self.additional_properties) field_dict.update( @@ -100,6 +106,8 @@ def to_dict(self) -> dict[str, Any]: field_dict["progress"] = progress if rejected_at is not UNSET: field_dict["rejected_at"] = rejected_at + if display_title is not UNSET: + field_dict["display_title"] = display_title return field_dict @@ -138,6 +146,8 @@ def from_dict(cls, src_dict: Mapping[str, Any]) -> Self: else: rejected_at = datetime.datetime.fromisoformat(_rejected_at) + display_title = d.pop("display_title", UNSET) + tool_call_state = cls( duration_ms=duration_ms, started_at=started_at, @@ -150,6 +160,7 @@ def from_dict(cls, src_dict: Mapping[str, Any]) -> Self: error=error, progress=progress, rejected_at=rejected_at, + display_title=display_title, ) tool_call_state.additional_properties = d diff --git a/sdks/dart/stream_agents/CHANGELOG.md b/sdks/dart/stream_agents/CHANGELOG.md index d71235b080..f3b9d3b4d2 100644 --- a/sdks/dart/stream_agents/CHANGELOG.md +++ b/sdks/dart/stream_agents/CHANGELOG.md @@ -6,6 +6,7 @@ - `session.interrupt(requestId:)` - stops only the reply answering that request. - `session.say(text, interrupt: true)` - stops what the agent is saying first. - `session.addTools` and `session.removeTool` - change a running session's tools. +- `AgentTool.typed` - a tool whose arguments its `fromJson` reads into a class of yours. ## 0.7.0 diff --git a/sdks/dart/stream_agents/README.md b/sdks/dart/stream_agents/README.md index 79c011be3a..13f527a027 100644 --- a/sdks/dart/stream_agents/README.md +++ b/sdks/dart/stream_agents/README.md @@ -116,6 +116,25 @@ final session = await agents.agent('support').chat(SessionOptions(tools: [lookup The agent only sees what the tool returns. +Dart has no reflection, so to read the arguments into a class of your own, give the tool its +`fromJson`. The model is still shown `parameters`; arguments that do not read, such as a missing +one, tell the model the tool failed, as a `run` that throws does: + +```dart +class WeatherLookup { + WeatherLookup.fromJson(Map json) : location = json['location']! as String; + final String location; +} + +final weatherLookup = AgentTool.typed( + name: 'weather_lookup', + description: 'Look up the current weather for a location', + parameters: AgentTool.strings({'location': 'a city or region'}, required: ['location']), + fromJson: WeatherLookup.fromJson, + run: (lookup) => weather.today(lookup.location), +); +``` + A running session takes more tools, such as the ones a card shown in it brings, and gives them back. The model is offered the change from the next turn, once the reply in progress and its tool calls have finished, so never wait for either from inside one of the session's own tools: diff --git a/sdks/dart/stream_agents/lib/src/core/tools.dart b/sdks/dart/stream_agents/lib/src/core/tools.dart index 724ce700d8..86afef95d2 100644 --- a/sdks/dart/stream_agents/lib/src/core/tools.dart +++ b/sdks/dart/stream_agents/lib/src/core/tools.dart @@ -77,6 +77,38 @@ final class AgentTool { /// A call that asks waits in `AgentSession.approvals` until `AgentSession.decide` answers it. final ToolApproval? approval; + /// A tool whose arguments [fromJson] reads into a [T] before [run] gets them. + /// + /// [parameters] is still what the model is shown; [fromJson] only reads what it sends. One + /// that throws, such as on a missing argument, tells the model the tool failed, as [run] + /// throwing does. + /// + /// AgentTool.typed( + /// name: 'weather_lookup', + /// description: 'Look up the current weather for a location', + /// parameters: AgentTool.strings({'location': 'a city or region'}, required: ['location']), + /// fromJson: WeatherLookup.fromJson, + /// run: (lookup) => weather.today(lookup.location), + /// ) + static AgentTool typed({ + required String name, + required String description, + required T Function(Map json) fromJson, + required Future Function(T arguments) run, + Map? parameters, + String? displayTitle, + ToolExecutor? executor, + ToolApproval? approval, + }) => AgentTool( + name: name, + description: description, + run: (arguments) async => run(fromJson(arguments)), + parameters: parameters, + displayTitle: displayTitle, + executor: executor, + approval: approval, + ); + /// A JSON Schema object for a tool whose arguments are all strings. /// /// A convenience for the common shape; for anything else, write the object yourself. diff --git a/sdks/dart/stream_agents/test/core/client_test.dart b/sdks/dart/stream_agents/test/core/client_test.dart index a37cf89d99..f894fd8c28 100644 --- a/sdks/dart/stream_agents/test/core/client_test.dart +++ b/sdks/dart/stream_agents/test/core/client_test.dart @@ -445,6 +445,44 @@ void main() { ]); }); + test( + 'declares a typed tool with the schema it was given, not one read from its type', + () async { + router.answer('POST /v1/agents/sessions', Answer(201, sessionJson())); + + await agents.sessions.create( + SessionOptions( + tools: [ + AgentTool.typed( + name: 'weather_lookup', + description: 'Look up the weather.', + parameters: AgentTool.strings({'location': 'a city'}, required: ['location']), + fromJson: (json) => json['location']! as String, + displayTitle: 'Checking the weather', + executor: ToolExecutor.client, + approval: const ToolApproval(title: 'Share your city?'), + run: (location) async => location, + ), + ], + ), + ); + + final [tool as Map] = + (router.last('POST /v1/agents/sessions').json as Map)['tools'] as List; + expect(tool['name'], 'weather_lookup'); + expect(tool['display_title'], 'Checking the weather'); + expect(tool['executor'], 'client'); + expect(tool['approval'], {'title': 'Share your city?'}); + expect(tool['parameters'], { + 'type': 'object', + 'properties': { + 'location': {'type': 'string', 'description': 'a city'}, + }, + 'required': ['location'], + }); + }, + ); + test('naming a config by id does not also send the agent it is scoped to', () async { router.answer('POST /v1/agents/sessions', Answer(201, sessionJson())); diff --git a/sdks/dart/stream_agents/test/core/session_test.dart b/sdks/dart/stream_agents/test/core/session_test.dart index f298157901..5e9df97473 100644 --- a/sdks/dart/stream_agents/test/core/session_test.dart +++ b/sdks/dart/stream_agents/test/core/session_test.dart @@ -238,6 +238,74 @@ void main() { await chat.close(); }); + test('a typed tool runs on the arguments its fromJson read', () async { + final chat = await agents.chat( + SessionOptions( + tools: [ + AgentTool.typed<_OrderLookup>( + name: 'lookup_order', + description: 'Look up an order.', + fromJson: _OrderLookup.fromJson, + run: (lookup) async => 'Order ${lookup.orderId}: ${lookup.items} items.', + ), + ], + ), + ); + final socket = await router.socket(); + + socket.send( + '{"type":"tool_call","id":"c1","name":"lookup_order",' + r'"arguments":"{\"order_id\":\"A-1042\",\"items\":2}"}', + ); + + final result = await socket.next(); + expect(result['output'], 'Order A-1042: 2 items.'); + expect(result['error'], ''); + await chat.close(); + }); + + test('tells the model a typed tool failed when its arguments do not read', () async { + final ran = <_OrderLookup>[]; + final chat = await agents.chat( + SessionOptions( + tools: [ + AgentTool.typed<_OrderLookup>( + name: 'lookup_order', + description: 'Look up an order.', + fromJson: _OrderLookup.fromJson, + run: (lookup) async { + ran.add(lookup); + return 'found'; + }, + ), + ], + ), + ); + final socket = await router.socket(); + + socket + ..send( + '{"type":"tool_call","id":"missing","name":"lookup_order",' + r'"arguments":"{\"items\":2}"}', + ) + ..send( + '{"type":"tool_call","id":"wrong_type","name":"lookup_order",' + r'"arguments":"{\"order_id\":\"A-1042\",\"items\":\"two\"}"}', + ); + + final results = { + for (final result in [await socket.next(), await socket.next()]) + result['tool_call_id']: result, + }; + for (final result in results.values) { + expect(result['output'], ''); + expect(result['error'], isNotEmpty); + } + expect(results.keys, unorderedEquals(['missing', 'wrong_type'])); + expect(ran, isEmpty); + await chat.close(); + }); + test('answers with pictures as parts beside the words', () async { final chat = await agents.chat( SessionOptions( @@ -1695,3 +1763,12 @@ void main() { }); }); } + +class _OrderLookup { + _OrderLookup.fromJson(Map json) + : orderId = json['order_id']! as String, + items = json['items']! as int; + + final String orderId; + final int items; +} diff --git a/sdks/dotnet/src/GetStream.VisionAgents/Generated/Models.cs b/sdks/dotnet/src/GetStream.VisionAgents/Generated/Models.cs index bd1eea1454..7c1b8b4a70 100644 --- a/sdks/dotnet/src/GetStream.VisionAgents/Generated/Models.cs +++ b/sdks/dotnet/src/GetStream.VisionAgents/Generated/Models.cs @@ -888,8 +888,11 @@ public partial class AgentConfig [System.Text.Json.Serialization.JsonPropertyName("tags")] public System.Collections.Generic.Dictionary? Tags { get; set; } = default!; + /// + /// The functions an SDK provides that the agent may offer, in named groups. Empty when there are none, and then the agent offers no function. + /// [System.Text.Json.Serialization.JsonPropertyName("tools")] - public AgentTools? Tools { get; set; } = default!; + public System.Collections.Generic.List? Tools { get; set; } = default!; [System.Text.Json.Serialization.JsonPropertyName("tts")] public string? Tts { get; set; } = default!; @@ -1083,8 +1086,11 @@ public partial class AgentConfigPatch [System.Text.Json.Serialization.JsonPropertyName("tags")] public System.Collections.Generic.Dictionary? Tags { get; set; } = default!; + /// + /// Replaces the tool groups whole; an empty list removes them all. + /// [System.Text.Json.Serialization.JsonPropertyName("tools")] - public AgentTools? Tools { get; set; } = default!; + public System.Collections.Generic.List? Tools { get; set; } = default!; [System.Text.Json.Serialization.JsonPropertyName("tts")] public string? Tts { get; set; } = default!; @@ -1306,10 +1312,10 @@ public partial class AgentConfigRequest public System.Collections.Generic.Dictionary? Tags { get; set; } = default!; /// - /// How the agent is offered its connector tools. Left out on an update, the stored settings stay. + /// The functions an SDK provides that the agent may offer, in named groups, each with how it is loaded. A function an SDK registers, declares on a session or hosts in a worker is offered only when a group names it: a session declaring one that none names is refused (400 tool_not_configured), and a worker's is left out. The router's own tools, connector tools and other agents are not named here. Omitted or null on an update, the groups stored stay; an empty list removes them all. The mapping this field once was, with progressive and output_retention, is refused: loading summary and output_retention on a group or a connector binding replaced them. /// [System.Text.Json.Serialization.JsonPropertyName("tools")] - public AgentTools? Tools { get; set; } = default!; + public System.Collections.Generic.List? Tools { get; set; } = default!; [System.Text.Json.Serialization.JsonPropertyName("tts")] public string? Tts { get; set; } = default!; @@ -1368,12 +1374,24 @@ public partial class AgentConnectorBinding [System.Text.Json.Serialization.JsonPropertyName("events")] public System.Collections.Generic.List? Events { get; set; } = default!; + /// + /// How the binding's tools are offered to the model. Omitted is full. + /// + [System.Text.Json.Serialization.JsonPropertyName("loading")] + public string? Loading { get; set; } = default!; + /// /// The alias, unique within the config: a lowercase letter, then up to 62 lowercase letters, digits, - or _, never __ and not ending in _. The model is offered each tool as <name>__<tool>, split back at the first __, so a __ inside the alias or a _ at its end would split it in the wrong place. /// [System.Text.Json.Serialization.JsonPropertyName("name")] public string Name { get; set; } = default!; + /// + /// How long the agent's sessions keep what the binding's tools were given and gave back on the turns they record, as a tool group's output_retention. Omitted or empty, it is kept as long as the session. + /// + [System.Text.Json.Serialization.JsonPropertyName("output_retention")] + public string? OutputRetention { get; set; } = default!; + /// /// How the binding's tool calls behave around speech and interruptions. Omitted, a call is cancelled at the provider when the turn is interrupted, and the agent picks its own words while it runs. /// @@ -1805,6 +1823,12 @@ public System.Collections.Generic.IDictionary AdditionalProperti public partial class AgentStats { + /// + /// Turns a session took in the window, on average. A turn is a reply to the person, not a rewound one or a delegated task's. Null when no session took one. + /// + [System.Text.Json.Serialization.JsonPropertyName("avg_turns")] + public double? AvgTurns { get; set; } = default!; + /// /// When that peak was first reached. Null when no session was open in the window. /// @@ -1817,6 +1841,12 @@ public partial class AgentStats [System.Text.Json.Serialization.JsonPropertyName("peak_concurrent_sessions")] public long PeakConcurrentSessions { get; set; } = default!; + /// + /// Sessions that took a turn in the window: what a tool's session_share is out of. + /// + [System.Text.Json.Serialization.JsonPropertyName("sessions")] + public long Sessions { get; set; } = default!; + /// /// Every tool call in the window, as one. Its tool is empty. /// @@ -1840,36 +1870,6 @@ public System.Collections.Generic.IDictionary AdditionalProperti } - /// - /// How an agent is offered its tools, and how long what they were given and gave back is kept. - /// - [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.7.1.0 (NJsonSchema v11.6.1.0 (Newtonsoft.Json v13.0.0.0))")] - public partial class AgentTools - { - - /// - /// How long this agent's sessions keep what any of its tools were given and gave back on the turns they record: a tool's arguments, its result, a search's query and the wording of a tool's error. none records none of it; a duration such as 12h or 30d (at least 1h, at most 3650d) keeps it that long after the call and then takes it out, within minutes. The item stays, with its tool name, its time, its call id and whether it failed. Left out on a create, or empty, it is kept as long as the session; left out on an update, the stored setting stays, and an empty string removes it. The app's and the organization's policies (tool_output_retention) can shorten it and never lengthen it, and the shortest applies. It applies to tool calls recorded after it is set. - /// - [System.Text.Json.Serialization.JsonPropertyName("output_retention")] - public string? OutputRetention { get; set; } = default!; - - /// - /// Offer each tool by the first line of its description, with its arguments' descriptions left out, and have the first call to a tool return its full description and input schema instead of running it. It saves context on an agent with many tools, at the cost of one more model turn for each tool a conversation uses. Off by default. Left out on an update, the stored setting stays. - /// - [System.Text.Json.Serialization.JsonPropertyName("progressive")] - public bool? Progressive { get; set; } = default!; - - private System.Collections.Generic.IDictionary? _additionalProperties; - - [System.Text.Json.Serialization.JsonExtensionData] - public System.Collections.Generic.IDictionary AdditionalProperties - { - get { return _additionalProperties ?? (_additionalProperties = new System.Collections.Generic.Dictionary()); } - set { _additionalProperties = value; } - } - - } - [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.7.1.0 (NJsonSchema v11.6.1.0 (Newtonsoft.Json v13.0.0.0))")] public partial class AgentUser { @@ -7507,7 +7507,7 @@ public partial class ErrorDetail { /// - /// What went wrong, for a program to branch on. Every type has a code of its own name (invalid_request, unauthenticated, forbidden, payment_required, not_found, method_not_allowed, not_acceptable, conflict, gone, payload_too_large, unsupported_media_type, rate_limited, internal_error, unavailable, timeout) that a failure has when nothing names it better. The others are validation_failed, missing_customer, missing_organization, server_side_only, not_configured (this deployment does not offer the feature), modality_not_routed, unsynced_changes (a sync asked to check would write over somebody's edits), name_taken (a 409: another agent config, router config or voice, or another skill of the same agent config, already has the name, so another name will do), channel_connection_taken (a 409: another live agent config binds the channel connection as fixed, and one agent answers a channel connection's messages, so the message names that config), call_unanswered (a 409: the call placed for a phone session ended with nobody answering, and the message names how: busy, no_answer, failed or canceled. Opening the session is refused, or, when the status came while the agent was joining the call, the session is ended at once and kept as ended), no_oauth_client (a 400: a consent found no OAuth client for the connector, and the message names each way the connector lets the app give it one), invalid_config (a 400: an agent config a run names does not start a session), insufficient_credits (a 402: the organization's credit is spent, so work that costs money is refused until credit is added; a model that costs nothing is still served), system_limit_reached (a 402: the organization's spend hard limit for the day or the month is reached, whatever its credit, and the message names the interval, the amount and when it resets), spend_limit_reached (a 402: one of the organization's own spend limits is spent, on the organization, an app, a key, a usage category or a team, and the message says which and when it resets, so retrying before then is refused again), user_limit_reached (a 429: a spend limit on the one end user the request is for is spent, which stops that user and no one else, and Retry-After says when its window ends), spend_check_unavailable (a 503: the router could not share what the organization is spending with the other routers for a while, so work that costs money is refused until it can; a model that costs nothing is still served, and Retry-After says to try again in a few seconds), budget_moved (a 400: a policy sent a budget, which spend limits replaced), limit_above_parent (a 400: a spend limit is above a limit of its parent in the same interval, and the message names the parent's limit), limit_exists (a 409: the subject already has a limit at that level and interval, and the message names its id), no_limit (a 404: the subject has no spend limit to reset or grant past), use_team (a 400: a category starting team: was sent, where the team field names a team), invalid_expiry (a 400: a grant's expires_at is not in the future, or is more than 366 days away), too_many_grants (a 409: the organization has 20 live grants, as many as it may), tier_limit_reached (the organization's tier limits refuse the work, and limit says which: a 403 for a call or a transfer, to a country the tier does not reach, past its minutes or recipients of the day, or past its calls at once; a 400 for an agent config whose sandbox is bigger or more open than the organization's sandbox limits allow. The message names the tier and the limit), account_suspended (a 403: the organization is suspended, and every request is refused, one for a model that costs nothing included, until it is restored; the message says to contact support), concurrency_limited (a 429: the app has as many runs in flight as it may, and Retry-After says when to try again), model_failed (a 503: the model failed or gave no answer), prompt_injection (a 400: the prompt injection screen refused the input of a run or a tool result it read, which retrying an unchanged input does not fix), image_unavailable (a 400: an image a run was handed could not be fetched or is not a JPEG, PNG, WebP or GIF, which retrying the same image does not fix), document_too_large (a 413: a file sent with a response has more pages or text than one may, alone or with the files the conversation already holds, and the message names the limit), document_unreadable (a 400: a file sent with a response could not be read, such as one with a password, one that is damaged or one with no text in it, which retrying the same file does not fix), invalid_usage_categories (a 400: the usage categories the request names, in the token's usage_categories and team claims or in X-Stream-Usage-Categories, are more than 10, one is not 1 to 64 characters of A-Z, a-z, 0-9, _ . : -, more than one names a team, or the team is over 59 characters), memory_unavailable (a 503: the memory store could not delete what was asked, so nothing that waits on it was deleted either, and the same request later can succeed), billing_not_enabled (a 404: this deployment bills nobody, or the caller's app is in no organization it bills, so there is no account to read), provider_takes_no_key (a 400: a key was put for a provider whose work is never routed on a customer's own key here, and the message names the ones that are), campaign_locked (a 409: the campaign's state does not allow the change, such as a launched campaign's sequence, or resuming one that is running), campaign_not_ready (a 400: a launch found the campaign missing what it needs to ring anybody, and the message says what), consent_required (a 400: a launch did not say the people on the campaign agreed to be called), branch_limit_reached (a 409: the conversation has as many edit and regenerate points as Stream Chat keeps for it, 100, so a regenerate or an edit is refused and the conversation is continued in a new chat; asking and reading are unaffected), branch_too_long (a 409: the branch an activation would bring back is more than 100 messages to send again, so it is continued in a new chat), branch_unavailable (a 410: the copies of a hidden branch were deleted, as a memory truncate deletes them, so it can be neither read nor brought back), switch_unfinished (a 409: a regenerate, an edit or an activation of the session failed after Stream Chat may have rewound, and finishing it still fails because Stream Chat does not answer, so the session's questions, regenerates, edits and activations are refused meanwhile; try again: each one tries to finish it first, so it clears once Stream Chat answers), and <resource>_not_found for agent_config, call, campaign, channel_account, command, connection, grant, knowledge_document, knowledge_url, limit, provider_key, router_config, scheduled_task, session, simulation, simulation_run, skill and voice. A scheduled task also has invalid_schedule (a schedule that does not read: at and in both or neither, a date, time or duration that does not parse, or until without repeat), schedule_in_past, schedule_too_far (a one-shot, or an in, more than a year ahead), invalid_timezone, timezone_unknown (no zone was given and neither the user nor the config has one), invalid_repeat, interval_too_short (a run moved to less than 15 minutes from now and earlier than the run it replaces) and too_many_tasks (a 409: the user is at their active-task limit). More may be added, so a client should expect one it does not know. + /// What went wrong, for a program to branch on. Every type has a code of its own name (invalid_request, unauthenticated, forbidden, payment_required, not_found, method_not_allowed, not_acceptable, conflict, gone, payload_too_large, unsupported_media_type, rate_limited, internal_error, unavailable, timeout) that a failure has when nothing names it better. The others are validation_failed, missing_customer, missing_organization, server_side_only, not_configured (this deployment does not offer the feature), modality_not_routed, unsynced_changes (a sync asked to check would write over somebody's edits), name_taken (a 409: another agent config, router config or voice, or another skill of the same agent config, already has the name, so another name will do), channel_connection_taken (a 409: another live agent config binds the channel connection as fixed, and one agent answers a channel connection's messages, so the message names that config), call_unanswered (a 409: the call placed for a phone session ended with nobody answering, and the message names how: busy, no_answer, failed or canceled. Opening the session is refused, or, when the status came while the agent was joining the call, the session is ended at once and kept as ended), no_oauth_client (a 400: a consent found no OAuth client for the connector, and the message names each way the connector lets the app give it one), invalid_config (a 400: an agent config a run names does not start a session), tool_not_configured (a 400: a session under an agent config was declared or given a function that none of the config's tool groups names, and the message names it), insufficient_credits (a 402: the organization's credit is spent, so work that costs money is refused until credit is added; a model that costs nothing is still served), system_limit_reached (a 402: the organization's spend hard limit for the day or the month is reached, whatever its credit, and the message names the interval, the amount and when it resets), spend_limit_reached (a 402: one of the organization's own spend limits is spent, on the organization, an app, a key, a usage category or a team, and the message says which and when it resets, so retrying before then is refused again), user_limit_reached (a 429: a spend limit on the one end user the request is for is spent, which stops that user and no one else, and Retry-After says when its window ends), spend_check_unavailable (a 503: the router could not share what the organization is spending with the other routers for a while, so work that costs money is refused until it can; a model that costs nothing is still served, and Retry-After says to try again in a few seconds), budget_moved (a 400: a policy sent a budget, which spend limits replaced), limit_above_parent (a 400: a spend limit is above a limit of its parent in the same interval, and the message names the parent's limit), limit_exists (a 409: the subject already has a limit at that level and interval, and the message names its id), no_limit (a 404: the subject has no spend limit to reset or grant past), use_team (a 400: a category starting team: was sent, where the team field names a team), invalid_expiry (a 400: a grant's expires_at is not in the future, or is more than 366 days away), too_many_grants (a 409: the organization has 20 live grants, as many as it may), tier_limit_reached (the organization's tier limits refuse the work, and limit says which: a 403 for a call or a transfer, to a country the tier does not reach, past its minutes or recipients of the day, or past its calls at once; a 400 for an agent config whose sandbox is bigger or more open than the organization's sandbox limits allow. The message names the tier and the limit), account_suspended (a 403: the organization is suspended, and every request is refused, one for a model that costs nothing included, until it is restored; the message says to contact support), concurrency_limited (a 429: the app has as many runs in flight as it may, and Retry-After says when to try again), model_failed (a 503: the model failed or gave no answer), prompt_injection (a 400: the prompt injection screen refused the input of a run or a tool result it read, which retrying an unchanged input does not fix), image_unavailable (a 400: an image a run was handed could not be fetched or is not a JPEG, PNG, WebP or GIF, which retrying the same image does not fix), document_too_large (a 413: a file sent with a response has more pages or text than one may, alone or with the files the conversation already holds, and the message names the limit), document_unreadable (a 400: a file sent with a response could not be read, such as one with a password, one that is damaged or one with no text in it, which retrying the same file does not fix), invalid_usage_categories (a 400: the usage categories the request names, in the token's usage_categories and team claims or in X-Stream-Usage-Categories, are more than 10, one is not 1 to 64 characters of A-Z, a-z, 0-9, _ . : -, more than one names a team, or the team is over 59 characters), memory_unavailable (a 503: the memory store could not delete what was asked, so nothing that waits on it was deleted either, and the same request later can succeed), billing_not_enabled (a 404: this deployment bills nobody, or the caller's app is in no organization it bills, so there is no account to read), provider_takes_no_key (a 400: a key was put for a provider whose work is never routed on a customer's own key here, and the message names the ones that are), campaign_locked (a 409: the campaign's state does not allow the change, such as a launched campaign's sequence, or resuming one that is running), campaign_not_ready (a 400: a launch found the campaign missing what it needs to ring anybody, and the message says what), consent_required (a 400: a launch did not say the people on the campaign agreed to be called), branch_limit_reached (a 409: the conversation has as many edit and regenerate points as Stream Chat keeps for it, 100, so a regenerate or an edit is refused and the conversation is continued in a new chat; asking and reading are unaffected), branch_too_long (a 409: the branch an activation would bring back is more than 100 messages to send again, so it is continued in a new chat), branch_unavailable (a 410: the copies of a hidden branch were deleted, as a memory truncate deletes them, so it can be neither read nor brought back), switch_unfinished (a 409: a regenerate, an edit or an activation of the session failed after Stream Chat may have rewound, and finishing it still fails because Stream Chat does not answer, so the session's questions, regenerates, edits and activations are refused meanwhile; try again: each one tries to finish it first, so it clears once Stream Chat answers), and <resource>_not_found for agent_config, call, campaign, channel_account, command, connection, grant, knowledge_document, knowledge_url, limit, provider_key, router_config, scheduled_task, session, simulation, simulation_run, skill and voice. A scheduled task also has invalid_schedule (a schedule that does not read: at and in both or neither, a date, time or duration that does not parse, or until without repeat), schedule_in_past, schedule_too_far (a one-shot, or an in, more than a year ahead), invalid_timezone, timezone_unknown (no zone was given and neither the user nor the config has one), invalid_repeat, interval_too_short (a run moved to less than 15 minutes from now and earlier than the run it replaces) and too_many_tasks (a 409: the user is at their active-task limit). More may be added, so a client should expect one it does not know. /// [System.Text.Json.Serialization.JsonPropertyName("code")] public string Code { get; set; } = default!; @@ -10029,6 +10029,12 @@ public partial class OfferedTool [System.Text.Json.Serialization.JsonPropertyName("description")] public string Description { get; set; } = default!; + /// + /// How the tool is offered, as its tool group or connector binding says: full for the router's own. A deferred tool is listed once the model has loaded it. Absent for a session that has ended. + /// + [System.Text.Json.Serialization.JsonPropertyName("loading")] + public string? Loading { get; set; } = default!; + /// /// How the model asks for it. /// @@ -10552,7 +10558,7 @@ public partial class Policy public System.Collections.Generic.Dictionary? Tags { get; set; } = default!; /// - /// How long a session keeps what its tools were given and gave back on the turns it records: a tool's arguments, its result, a search's query and the wording of a tool's error. none records none of it; a duration such as 12h or 30d (at least 1h, at most 3650d) keeps it that long after the call and then takes it out, within minutes. The item stays, with its tool name, its time, its call id and whether it failed, so a session's items read in the same order and tool usage still counts the call. Left out, or empty, it is kept as long as the session. The shortest of the organization's, the app's and the agent config's tools.output_retention applies, so an agent can shorten it and never lengthen it. It applies to tool calls recorded after it is set. Deleting a session deletes all of it at once. + /// How long a session keeps what its tools were given and gave back on the turns it records: a tool's arguments, its result, a search's query and the wording of a tool's error. none records none of it; a duration such as 12h or 30d (at least 1h, at most 3650d) keeps it that long after the call and then takes it out, within minutes. The item stays, with its tool name, its time, its call id and whether it failed, so a session's items read in the same order and tool usage still counts the call. Left out, or empty, it is kept as long as the session. The shortest of the organization's, the app's and the output_retention of the agent config's tool group or connector binding the tool is in applies, so an agent can shorten it and never lengthen it. It applies to tool calls recorded after it is set. Deleting a session deletes all of it at once. /// [System.Text.Json.Serialization.JsonPropertyName("tool_output_retention")] public string? ToolOutputRetention { get; set; } = default!; @@ -15681,10 +15687,10 @@ public partial class SyncAgentRequest public System.Collections.Generic.Dictionary? Tags { get; set; } = default!; /// - /// How connector tools are offered. A setting left out keeps what is stored. + /// agent.yaml's tools: the functions an SDK provides that the agent may offer, in named groups, as AgentConfigRequest's tools names them, with the same rules. Sent, they replace the stored groups whole, an empty list removing them. Left out, the stored ones stay. /// [System.Text.Json.Serialization.JsonPropertyName("tools")] - public AgentTools? Tools { get; set; } = default!; + public System.Collections.Generic.List? Tools { get; set; } = default!; [System.Text.Json.Serialization.JsonPropertyName("tts")] public string? Tts { get; set; } = default!; @@ -16201,6 +16207,120 @@ public System.Collections.Generic.IDictionary AdditionalProperti } + /// + /// One change to a tool call's state. Every watcher is sent tool_call.started once, tool_call.progress for each change while it runs or waits, and tool_call.finished once, carrying the call's whole state each time, so a client keeps the last one it got per tool_call_id. + /// + [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.7.1.0 (NJsonSchema v11.6.1.0 (Newtonsoft.Json v13.0.0.0))")] + public partial class ToolCallState + { + + /// + /// When a person allowed the call. + /// + [System.Text.Json.Serialization.JsonPropertyName("approved_at")] + public System.DateTimeOffset? ApprovedAt { get; set; } = default!; + + /// + /// What the call is doing, in words for the people in the conversation, such as "Checking the weather": the display_title its tool was declared with, or "Searching the web" for the router's own search. Absent when the tool declared none. + /// + [System.Text.Json.Serialization.JsonPropertyName("display_title")] + public string? DisplayTitle { get; set; } = default!; + + /// + /// From started_at to this change; on tool_call.finished, how long the call took. + /// + [System.Text.Json.Serialization.JsonPropertyName("duration_ms")] + public long DurationMs { get; set; } = default!; + + /// + /// Why a failed call failed, or what a declined one shows, such as "Not allowed in time". + /// + [System.Text.Json.Serialization.JsonPropertyName("error")] + public string? Error { get; set; } = default!; + + /// + /// The last thing said about the call while it runs, such as "Searching for flights": a tool_update, the connector binding's pre_speech, or the question it waits on. + /// + [System.Text.Json.Serialization.JsonPropertyName("progress")] + public string? Progress { get; set; } = default!; + + /// + /// When a person declined the call, or its wait for them passed. + /// + [System.Text.Json.Serialization.JsonPropertyName("rejected_at")] + public System.DateTimeOffset? RejectedAt { get; set; } = default!; + + [System.Text.Json.Serialization.JsonPropertyName("started_at")] + public System.DateTimeOffset StartedAt { get; set; } = default!; + + [System.Text.Json.Serialization.JsonPropertyName("status")] + public string Status { get; set; } = default!; + + [System.Text.Json.Serialization.JsonPropertyName("tool")] + public string Tool { get; set; } = default!; + + /// + /// The call, as tool_call, tool_started and tool_ran name it. + /// + [System.Text.Json.Serialization.JsonPropertyName("tool_call_id")] + public string ToolCallId { get; set; } = default!; + + [System.Text.Json.Serialization.JsonPropertyName("turn_id")] + public string TurnId { get; set; } = default!; + + [System.Text.Json.Serialization.JsonPropertyName("type")] + public string Type { get; set; } = default!; + + private System.Collections.Generic.IDictionary? _additionalProperties; + + [System.Text.Json.Serialization.JsonExtensionData] + public System.Collections.Generic.IDictionary AdditionalProperties + { + get { return _additionalProperties ?? (_additionalProperties = new System.Collections.Generic.Dictionary()); } + set { _additionalProperties = value; } + } + + } + + /// + /// A named set of functions an SDK provides that the agent may offer, how they are loaded and how long what they were given and gave back is kept. A function an SDK registers, declares on a session or hosts in a worker is offered only when a group names it. + /// + [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.7.1.0 (NJsonSchema v11.6.1.0 (Newtonsoft.Json v13.0.0.0))")] + public partial class ToolGroup + { + + [System.Text.Json.Serialization.JsonPropertyName("loading")] + public string? Loading { get; set; } = default!; + + /// + /// What the group is called, unique within the config. + /// + [System.Text.Json.Serialization.JsonPropertyName("name")] + public string Name { get; set; } = default!; + + /// + /// How long the agent's sessions keep what the group's tools were given and gave back on the turns they record: a tool's arguments, its result and the wording of its error. none records none of it; a duration such as 12h or 30d (at least 1h, at most 3650d) keeps it that long after the call and then takes it out, within minutes. The item stays, with its tool name, its time, its call id and whether it failed. Left out or empty, it is kept as long as the session. The app's and the organization's policies (tool_output_retention) can shorten it and never lengthen it. It applies to tool calls recorded after it is set. + /// + [System.Text.Json.Serialization.JsonPropertyName("output_retention")] + public string? OutputRetention { get; set; } = default!; + + /// + /// The functions' names, as the SDK registers them. A name sits in one group at most. + /// + [System.Text.Json.Serialization.JsonPropertyName("tools")] + public System.Collections.Generic.List Tools { get; set; } = new System.Collections.Generic.List(); + + private System.Collections.Generic.IDictionary? _additionalProperties; + + [System.Text.Json.Serialization.JsonExtensionData] + public System.Collections.Generic.IDictionary AdditionalProperties + { + get { return _additionalProperties ?? (_additionalProperties = new System.Collections.Generic.Dictionary()); } + set { _additionalProperties = value; } + } + + } + [System.CodeDom.Compiler.GeneratedCode("NJsonSchema", "14.7.1.0 (NJsonSchema v11.6.1.0 (Newtonsoft.Json v13.0.0.0))")] public partial class ToolResultCommand { @@ -16253,18 +16373,54 @@ public partial class ToolStats [System.Text.Json.Serialization.JsonPropertyName("avg_tokens")] public double AvgTokens { get; set; } = default!; + /// + /// Turns a session took before its first call, on average: 0 is a call on the first turn. Null on the row over every tool. + /// + [System.Text.Json.Serialization.JsonPropertyName("avg_turns_before_use")] + public double? AvgTurnsBeforeUse { get; set; } = default!; + [System.Text.Json.Serialization.JsonPropertyName("calls")] public long Calls { get; set; } = default!; + /// + /// What the tool's whole definition, its name, description and schema, costs every turn that carries it, estimated at four characters a token. Null when no call recorded it. + /// + [System.Text.Json.Serialization.JsonPropertyName("definition_tokens")] + public long? DefinitionTokens { get; set; } = default!; + /// /// Calls that ended in an error. /// [System.Text.Json.Serialization.JsonPropertyName("failed")] public long Failed { get; set; } = default!; + /// + /// How the latest call had the tool offered: full, summary or deferred. Empty when no call recorded it, as calls made before it was recorded and built-in tools. + /// + [System.Text.Json.Serialization.JsonPropertyName("loading")] + public string Loading { get; set; } = default!; + [System.Text.Json.Serialization.JsonPropertyName("p99_ms")] public double? P99Ms { get; set; } = default!; + /// + /// Sessions that called it, out of the window's sessions, from 0 to 1. Null when no session took a turn. + /// + [System.Text.Json.Serialization.JsonPropertyName("session_share")] + public double? SessionShare { get; set; } = default!; + + /// + /// Sessions that called it at least once. + /// + [System.Text.Json.Serialization.JsonPropertyName("sessions")] + public long Sessions { get; set; } = default!; + + /// + /// What its definition costs offered as a summary. Null unless its loading is summary. + /// + [System.Text.Json.Serialization.JsonPropertyName("summary_tokens")] + public long? SummaryTokens { get; set; } = default!; + [System.Text.Json.Serialization.JsonPropertyName("tool")] public string Tool { get; set; } = default!; diff --git a/sdks/dotnet/src/GetStream.VisionAgents/Tools.cs b/sdks/dotnet/src/GetStream.VisionAgents/Tools.cs index b5f1c36ccc..0a57d26bfb 100644 --- a/sdks/dotnet/src/GetStream.VisionAgents/Tools.cs +++ b/sdks/dotnet/src/GetStream.VisionAgents/Tools.cs @@ -1,3 +1,5 @@ +using System.ComponentModel; +using System.Reflection; using System.Text.Json; using System.Text.Json.Nodes; using System.Text.Json.Schema; @@ -55,7 +57,7 @@ public Tools Register( TimeSpan? timeout = null, string? approval = null) { - var schema = Json.Options.GetJsonSchemaAsNode(typeof(TArguments)); + var schema = Json.Options.GetJsonSchemaAsNode(typeof(TArguments), Described); return Register(name, description, schema as JsonObject ?? [], async (toolCallContext, arguments, cancellationToken) => { var read = arguments.Deserialize(Json.Options) @@ -268,6 +270,27 @@ private static void CheckApproval(string name, string? approval, JsonObject para private static readonly Regex Placeholder = new(@"\{([A-Za-z_][A-Za-z0-9_]*)\}", RegexOptions.Compiled); + /// + /// Writes a into the schema as description: a property's + /// own, else its record parameter's, else its type's. + /// + private static readonly JsonSchemaExporterOptions Described = new() + { + TransformSchemaNode = (context, node) => + { + var attribute = context.PropertyInfo?.AttributeProvider?.GetCustomAttributes(typeof(DescriptionAttribute), inherit: true) + .OfType().FirstOrDefault() + ?? context.PropertyInfo?.AssociatedParameter?.AttributeProvider?.GetCustomAttributes(typeof(DescriptionAttribute), inherit: true) + .OfType().FirstOrDefault() + ?? context.TypeInfo.Type.GetCustomAttribute(); + if (attribute is not null && node is JsonObject schema) + { + schema["description"] = attribute.Description; + } + return node; + }, + }; + private sealed record Registered( string Description, JsonObject Parameters, diff --git a/sdks/dotnet/tests/GetStream.VisionAgents.Tests/ToolCallTests.cs b/sdks/dotnet/tests/GetStream.VisionAgents.Tests/ToolCallTests.cs index d32778f28b..6612774d3b 100644 --- a/sdks/dotnet/tests/GetStream.VisionAgents.Tests/ToolCallTests.cs +++ b/sdks/dotnet/tests/GetStream.VisionAgents.Tests/ToolCallTests.cs @@ -1,4 +1,5 @@ using System.Collections.Concurrent; +using System.ComponentModel; using System.Text.Json; using System.Text.Json.Nodes; @@ -210,6 +211,62 @@ public void APolicyThatCannotWorkIsRefusedWhenItIsRegistered() Assert.Equal(1, tools.Count); } + [Fact] + public void ARecordParametersDescriptionIsItsArgumentsDescription() + { + var schema = Schema(); + + Assert.Equal("""{"type":"string","description":"a city or region"}""", schema["properties"]!["location"]!.ToJsonString()); + } + + [Fact] + public void APropertysDescriptionIsItsArgumentsDescription() + { + var schema = Schema(); + + Assert.Equal("how many people are coming", schema["properties"]!["guests"]!.Text("description")); + } + + [Fact] + public void ATypesDescriptionDescribesTheArgumentsAndAnyObjectOfThatType() + { + var schema = Schema(); + + Assert.Equal("a journey of one leg", schema.Text("description")); + Assert.Equal("one hop between two places", schema["properties"]!["leg"]!.Text("description")); + } + + [Fact] + public void ArgumentsWithNoDescriptionAreDeclaredAsTheyAre() + { + var schema = Schema(); + + Assert.Equal("""{"type":["object","null"],"properties":{"location":{"type":"string"}},"required":["location"]}""", + schema.ToJsonString()); + } + + private static JsonObject Schema() + { + var tools = new Tools().Register("tool", "A tool", (_, _) => Task.FromResult("")); + return JsonSerializer.SerializeToNode(Assert.Single(tools.Declared()).Parameters)!.AsObject(); + } + + private sealed record WeatherLookup([Description("a city or region")] string Location); + + private sealed record Booking + { + [Description("how many people are coming")] + public int Guests { get; init; } + } + + [Description("a journey of one leg")] + private sealed record Trip(Leg Leg); + + [Description("one hop between two places")] + private sealed record Leg(string From); + + private sealed record Undescribed(string Location); + [Fact] public async Task AHostedToolIsDeclaredWithItsPoliciesAndAnsweredByCallId() { diff --git a/sdks/go/acceleration/generated.go b/sdks/go/acceleration/generated.go index 51d6ea794e..9d91f04611 100644 --- a/sdks/go/acceleration/generated.go +++ b/sdks/go/acceleration/generated.go @@ -10559,6 +10559,9 @@ type ToolCallState struct { // ApprovedAt When a person allowed the call. ApprovedAt *time.Time `json:"approved_at,omitempty"` + // DisplayTitle What the call is doing, in words for the people in the conversation, such as "Checking the weather": the display_title its tool was declared with, or "Searching the web" for the router's own search. Absent when the tool declared none. + DisplayTitle *string `json:"display_title,omitempty"` + // DurationMs From started_at to this change; on tool_call.finished, how long the call took. DurationMs int64 `json:"duration_ms"` diff --git a/sdks/go/stream/accelerated.go b/sdks/go/stream/accelerated.go index c8b703645f..b6e2271bf0 100644 --- a/sdks/go/stream/accelerated.go +++ b/sdks/go/stream/accelerated.go @@ -893,6 +893,10 @@ func declarationsOf(registered []*tools.Function) []acceleration.SessionTool { parameters := function.Parameters tool.Parameters = ¶meters } + if function.DisplayTitle != "" { + title := function.DisplayTitle + tool.DisplayTitle = &title + } if function.Interruptible { interruptible := true tool.Interruptible = &interruptible diff --git a/sdks/go/stream/dispatch.go b/sdks/go/stream/dispatch.go index e487605229..6802bde847 100644 --- a/sdks/go/stream/dispatch.go +++ b/sdks/go/stream/dispatch.go @@ -246,6 +246,9 @@ func (d *Dispatch) host() { declared := []Frame{} for _, function := range offer.functions.List() { tool := Frame{"name": function.Name, "description": function.Description, "parameters": function.Parameters} + if function.DisplayTitle != "" { + tool["display_title"] = function.DisplayTitle + } if function.Interruptible { tool["interruptible"] = true } diff --git a/sdks/go/stream/dispatch_test.go b/sdks/go/stream/dispatch_test.go index dbf85ddb2e..c19a8cb5ad 100644 --- a/sdks/go/stream/dispatch_test.go +++ b/sdks/go/stream/dispatch_test.go @@ -557,6 +557,29 @@ func TestAHostedFunctionIsDeclaredAndAnsweredOverTheDispatchSocket(t *testing.T) stopped() } +func TestAHostedFunctionTellsTheRouterWhatItsCallsAreShownDoing(t *testing.T) { + router := newPool(t, func(*websocket.Conn) {}) + functions := tools.NewRegistry() + if err := tools.Register(functions, "weather_lookup", "Look up the weather", func(context.Context, struct{}) (any, error) { + return "sunny", nil + }, tools.DisplayTitle("Checking the weather")); err != nil { + t.Fatal(err) + } + worker := waiting(t, router, DispatchOptions{}) + worker.Host(toolHost{functions}, time.Minute) + + ctx, cancel := context.WithCancel(t.Context()) + stopped := run(t, ctx, worker) + + declared := router.told(t, "host_tools") + hosted, _ := declared["tools"].([]any) + if len(hosted) != 1 || hosted[0].(map[string]any)["display_title"] != "Checking the weather" { + t.Errorf("the router was told %v", declared["tools"]) + } + cancel() + stopped() +} + func TestAWorkerTheRouterDropsReconnectsAndHostsAgain(t *testing.T) { // The first connection is cut without a close frame, the way a router pod being // replaced ends it, and the second goes away with one. A worker that stopped at diff --git a/sdks/go/tools/options.go b/sdks/go/tools/options.go index 76e9ce2a30..b5bafadad1 100644 --- a/sdks/go/tools/options.go +++ b/sdks/go/tools/options.go @@ -19,6 +19,19 @@ func Interruptible(interruptible bool) Option { } } +// DisplayTitle is what a call is doing, in words for the people in the conversation, such as +// "Checking the weather". Apps show it while the call runs. +func DisplayTitle(title string) Option { + return func(function *Function) error { + title = strings.TrimSpace(title) + if title == "" { + return fmt.Errorf("a display title cannot be empty") + } + function.DisplayTitle = title + return nil + } +} + // Timeout bounds one call, in place of the session's default. func Timeout(timeout time.Duration) Option { return func(function *Function) error { diff --git a/sdks/go/tools/registry.go b/sdks/go/tools/registry.go index dc2b5c6fe3..5db7a59efb 100644 --- a/sdks/go/tools/registry.go +++ b/sdks/go/tools/registry.go @@ -21,6 +21,9 @@ type Function struct { // Parameters is the JSON Schema object describing the arguments. Every call's arguments // are held to it before the function runs. Parameters map[string]any + // DisplayTitle is what a call is doing, in words for the people in the conversation. + // Empty shows none. + DisplayTitle string // Interruptible is whether the caller talking over the agent cancels a call. Interruptible bool // Timeout bounds one call. Zero leaves it to the session. diff --git a/sdks/js/README.md b/sdks/js/README.md index c04b538c6f..a3cdb2eac9 100644 --- a/sdks/js/README.md +++ b/sdks/js/README.md @@ -409,6 +409,29 @@ and an add or a removal whose answer was lost, or was a failure other than a 4xx the function running until the name is added or removed again. `session.fork()` runs the functions the session runs when it forks, which are the tools the router copies to the fork. +In React, `useTool` from `@stream-io/stream-agents/react` offers a component's own function +while it is mounted, and withdraws it when it unmounts or the session or name changes: + +```tsx +import { useTool } from "@stream-io/stream-agents/react"; + +function CheckoutPage({ session }: { session: Session | undefined }) { + const [cart] = useCart(); + const failure = useTool(session, { + name: "read_cart", + description: "What is in the person's cart right now", + parameters: z.object({}), + run: async () => JSON.stringify(cart), + }); + // ... +} +``` + +What the model is told of the tool is sent once per session and name; each call runs the latest +`run`, so a re-render with new state sends nothing. A removal waits for the add before it, and +an add that was refused, such as the 409 for a name the session already has, is not followed by +one. The hook answers the last add or removal that failed, or undefined. + ## Agent dispatch A caller reached a Stream call over SIP, or somebody wrote in a channel, and the router diff --git a/sdks/js/src/generated/api.ts b/sdks/js/src/generated/api.ts index c6d1b96e3d..6d2e294cc0 100644 --- a/sdks/js/src/generated/api.ts +++ b/sdks/js/src/generated/api.ts @@ -2097,7 +2097,7 @@ export type paths = { * `connector_unavailable` names an optional connector binding the session opened without: name (its alias), connector_id and reason, one of no_selection, shared_session, caller_unverified, connection_unavailable, provider_mismatch, needs_reauthorization, credential_rejected (the provider rejected the token or key a bearer or api_key connection holds, or the connection reads a connector revision marked broken; only saving its credentials again fixes it, so no login is offered), not_connected, open_failed, tool_unavailable and selection_dropped (a fork's or a reopened chat's selection for an alias its config no longer declares). Every watcher is sent each one when it attaches. * `connector_scope_required` says a connector tool call was refused because the caller's own connection lacks access the provider asked for (insufficient_scope or a claims challenge), and a step-up consent was begun for it: name (the binding's alias), connector_id, connection_id, scopes (what the provider asked for, empty for a claims challenge), authorization_id, launch_url, handoff_token and expires_at. A page of the session's return_url's origin, or the dashboard for a session created without one, opens launch_url in a popup and posts it handoff_token, as for a createAuthorization with that return_url: the launch page takes the handoff from that origin only, and the consent goes back to the session's return_url, or else to the dashboard. The old grant keeps working until the step-up succeeds, and the same call works afterwards in the same session. While that step-up is open, calls refused for the same access send no second event. * Persistent text sessions also emit `conversation_updated` with conversation_id and a complete message snapshot: id, request_id, question_id, role, text, state, response_started_at, state_started_at, finished_at, duration_ms, saved, persistence_error, refused (true on a reply the prompt injection screen refused) and attachments. Each tool_calling attachment has tool_call_id, name, title, status, phase, summary, immutable started_at, execution_started_at, finished_at and duration_ms. A connector_authorization attachment asks the end user to connect a connector binding the reply needed with their own account, with name (the binding's alias), connector_id, connection_id, authorization_id, title, launch_url, handoff_token and expires_at: a page of the session's return_url's origin, or the dashboard for a session created without one, opens launch_url in a popup and posts it handoff_token, as for a createAuthorization with that return_url, and the consent goes back to the session's return_url, or else to the dashboard. Once the user finishes that login the message is sent again with status connected and no handoff_token, and the agent carries on by itself. The call that needed the login waits for it on its own ai_tool_call step, in parts: status awaiting_authorization, target_user_id the caller, and an authorization of type oauth with provider (the connector id), provider_name, title, message, url (the launch page), handoff_token and expires_at. A page of the return_url's origin opens url and posts it handoff_token; a native app opens url#handoff_token= in its web authentication sheet. Connected, the step is running again with authorization.status connected and no handoff_token, and the call runs in the same reply. A tool_approval naming its tool_call_id with allowed false, or the person refusing at the provider, declines the login (declined); a consent that ended otherwise without connecting, such as with another account, is failed, and one not finished by expires_at is expired. Each way the step is cancelled, authorization.status says how, and the call runs nothing. Activity states are thinking, queued, tools, writing, completed, failed and cancelled. tool_started includes tool_call_id, tool, turn_id and started_at, and pre_speech when the tool's connector binding sets one in its policy; tool_ran also includes tool_call_id, and files when the tool was run_code, as on task_settled. - * `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, waiting for a person's approval, success, failed or cancelled), progress (the last `tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id. + * `tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, waiting for a person's approval, success, failed or cancelled), display_title (what the call is doing, as its tool was declared with), progress (the last `tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id. * A respond command carrying request_id emits command_accepted with a nested command receipt (request_id, user_message_id, assistant_message_id, state, duplicate). Personal persistent text sessions require this ID. A retry with the same text and images returns the existing IDs without invoking the model again; reuse with different content emits an error. Its images are data URIs, at most 8, uploaded to the conversation's channel and shown on the person's message. On a session a dispatch worker opened for a message handed out from a channel, that message is already there: nothing is written or uploaded for it, user_message_id is empty, and a reused id is answered as a duplicate even with other text or images. After restart an interrupted command is reported, not rerun. * An `interrupt` command carrying `request_id` stops that command and emits `command_stopped` with its terminal receipt. A stop arriving after its command finished replays that command's receipt and leaves the command running now alone; an unknown command is reported as an error. Without `request_id` the frame stops whichever reply is current, which is what a caller with no command to name means by it. * A `decision` frame is one judgement the conversation made, carrying the same fields as a CallEvent. Together they are why the call went the way it did, and they are also written down, so a finished call replays them from `/v1/agents/calls/{id}/events`. @@ -10942,6 +10942,8 @@ export type components = { * @description When a person allowed the call. */ readonly approved_at?: string; + /** @description What the call is doing, in words for the people in the conversation, such as "Checking the weather": the display_title its tool was declared with, or "Searching the web" for the router's own search. Absent when the tool declared none. */ + readonly display_title?: string; /** * Format: int64 * @description From started_at to this change; on tool_call.finished, how long the call took. diff --git a/sdks/js/src/react.ts b/sdks/js/src/react.ts index 502d833f7b..6babf2047f 100644 --- a/sdks/js/src/react.ts +++ b/sdks/js/src/react.ts @@ -8,10 +8,11 @@ * Its own entry point so `react` is resolved only for an app that asked for it. */ -import { useCallback, useEffect, useRef, useSyncExternalStore } from "react"; +import { useCallback, useEffect, useRef, useState, useSyncExternalStore } from "react"; import type { Session, SessionEvent } from "./session.js"; import type { ToolCallState } from "./tool-calls.js"; +import { Tools, type Tool } from "./tools.js"; /** Every tool call of the session by `tool_call_id`, in the order they started, re-rendering as one changes. */ export function useToolCalls(session: Session): ReadonlyMap { @@ -33,3 +34,54 @@ export function useSessionEvent(session: Session, kind: string, handler: (event: latest.current = handler; useEffect(() => session.on(kind, (event) => latest.current(event)), [session, kind]); } + +/** + * Offers `tool` to the session's agent while the component is mounted, with `addTools`, and + * withdraws it with `removeTool` on unmount or when `session` or `tool.name` changes. What the + * model is told of it is read once per session and name; the latest `run` is the one called, + * so a re-render with new state does not offer it again. Answers the last add or removal that + * failed, such as the 409 for a name the session already has, or undefined. + */ +export function useTool(session: Session | undefined, tool: Tool): Error | undefined { + const latest = useRef(tool); + latest.current = tool; + const queue = useRef(Promise.resolve()); + const [failure, setFailure] = useState(); + useEffect( + () => (session ? offerTool(session, latest, queue, setFailure) : undefined), + [session, tool.name], + ); + return failure; +} + +/** + * @internal What `useTool` does when it mounts, returning what it does when it unmounts. Each + * change waits in `queue` for the one before, so a tool is withdrawn only once it was added: + * withdrawn after a refused add, it would be the session's own of the same name that went. + */ +export function offerTool( + session: Session, + latest: { readonly current: Tool }, + queue: { current: Promise }, + failed: (cause: Error) => void, +): () => void { + const { name } = latest.current; + const report = (cause: unknown): void => failed(cause instanceof Error ? cause : new Error(String(cause))); + const added = queue.current + .then(() => + session.addTools( + new Tools().register({ ...latest.current, run: (input, context) => latest.current.run(input, context) }), + ), + ) + .then( + () => true, + (cause: unknown) => { + report(cause); + return false; + }, + ); + queue.current = added.then(() => undefined); + return () => { + queue.current = added.then((offered) => (offered ? session.removeTool(name) : undefined)).catch(report); + }; +} diff --git a/sdks/js/tests/use-tool.test.ts b/sdks/js/tests/use-tool.test.ts new file mode 100644 index 0000000000..11fa2b52ce --- /dev/null +++ b/sdks/js/tests/use-tool.test.ts @@ -0,0 +1,164 @@ +import assert from "node:assert/strict"; +import { afterEach, beforeEach, describe, it } from "node:test"; + +import { Client, Session, Tools, type Tool } from "../src/index.js"; +import { offerTool } from "../src/react.js"; +import { TestRouter, type Connection, type Received } from "./router.js"; +import { until } from "./stream-chat.js"; + +const TOOLS = "/v1/agents/sessions/sess_1/tools"; + +/** + * What `useTool` does, driven the way React drives its effect: `mount` runs it, the function it + * answers is the unmount, and a re-render only points `latest` at the new tool. The hook itself + * is not rendered, since running an effect needs a DOM this suite has none of. + */ +describe("useTool", () => { + let router: TestRouter; + let session: Session; + let connection: Connection; + /** The names the router offers the model, kept the way it keeps them. */ + let offered: Set; + let failures: Error[]; + let queue: { current: Promise }; + + beforeEach(async () => { + router = await TestRouter.start(); + router.serve("POST", "/v1/agents/sessions", { + status: 201, + body: { + id: "sess_1", + call_id: "sess_1", + call_type: "agent", + user_id: "john", + agent_id: "john", + state: "live", + created_at: "2026-01-01T00:00:00Z", + }, + }); + offered = new Set(); + failures = []; + queue = { current: Promise.resolve() }; + router.serve("POST", TOOLS, ({ body }) => { + const names = (body as { tools: { name: string }[] }).tools.map(({ name }) => name); + if (names.some((name) => offered.has(name))) { + return taken; + } + names.forEach((name) => offered.add(name)); + return { status: 204 }; + }); + router.serve("DELETE", `${TOOLS}/read_cart`, () => (offered.delete("read_cart") ? { status: 204 } : missing)); + }); + + afterEach(async () => { + await session.close(); + await router.stop(); + }); + + async function open(tools?: Tools): Promise { + const opening = Session.open(new Client({ url: router.url, customerId: "local" }), { start_voice: true }, tools ? { tools } : {}); + connection = await router.socket(); + session = await opening; + } + + function mount(latest: { current: Tool> }): () => void { + return offerTool(session, latest, queue, (cause) => failures.push(cause)); + } + + async function call(id: string): Promise { + connection.send({ type: "tool_call", id, name: "read_cart", arguments: "{}" }); + const answered = await connection.next(); + assert.equal(answered["tool_call_id"], id); + return answered["output"] ?? answered["error"]; + } + + function cart(items: string[]): Tool> { + return { name: "read_cart", description: "What is in the cart", run: () => items.join(", ") }; + } + + it("offers the tool once and runs the latest closure after a re-render, so new state needs no re-add", async () => { + await open(); + const latest = { current: cart(["socks"]) }; + mount(latest); + await queue.current; + + latest.current = cart(["socks", "shoes"]); + + assert.equal(await call("call-1"), "socks, shoes"); + assert.deepEqual([...offered], ["read_cart"]); + assert.equal(router.requestsTo("POST", TOOLS).length, 1); + assert.deepEqual(failures, []); + }); + + it("withdraws the tool on unmount, so the model is no longer offered it", async () => { + await open(); + const unmount = mount({ current: cart(["socks"]) }); + await queue.current; + + unmount(); + await queue.current; + + assert.deepEqual([...offered], []); + assert.deepEqual(failures, []); + }); + + it("still withdraws a tool unmounted while its add was in flight", async () => { + await open(); + let answer: (reply: { status: number }) => void = () => undefined; + router.serve("POST", TOOLS, () => { + offered.add("read_cart"); + return new Promise((resolve) => (answer = resolve)); + }); + const unmount = mount({ current: cart(["socks"]) }); + await until(() => router.requestsTo("POST", TOOLS).length === 1); + + unmount(); + answer({ status: 204 }); + await queue.current; + + assert.deepEqual([...offered], []); + assert.deepEqual(failures, []); + }); + + it("ends StrictMode's mount, unmount and mount with exactly one offer", async () => { + await open(); + const latest = { current: cart(["socks"]) }; + mount(latest)(); + mount(latest); + await queue.current; + + assert.deepEqual([...offered], ["read_cart"]); + assert.deepEqual( + router.received.filter(({ path }) => path.startsWith(TOOLS)).map(({ method }: Received) => method), + ["POST", "DELETE", "POST"], + ); + assert.equal(await call("call-1"), "socks"); + assert.deepEqual(failures, []); + }); + + it("reports a refused add and leaves the session's own tool of that name alone on unmount", async () => { + await open(new Tools().register(cart(["the session's own"]))); + offered.add("read_cart"); + const unmount = mount({ current: cart(["the component's"]) }); + await queue.current; + + unmount(); + await queue.current; + + assert.equal(failures.length, 1); + assert.match(failures[0]!.message, /taken/); + assert.deepEqual([...offered], ["read_cart"]); + assert.equal(router.requestsTo("DELETE", `${TOOLS}/read_cart`).length, 0); + assert.equal(await call("call-1"), "the session's own"); + }); +}); + +const taken = { + status: 409, + body: { error: { message: "read_cart is taken", type: "conflict", code: "tool_name_taken", doc_url: "" } }, +}; + +const missing = { + status: 404, + body: { error: { message: "no tool read_cart", type: "not_found", code: "tool_not_found", doc_url: "" } }, +}; diff --git a/sdks/kotlin/README.md b/sdks/kotlin/README.md index a49eb90441..fc77cd9e30 100644 --- a/sdks/kotlin/README.md +++ b/sdks/kotlin/README.md @@ -137,6 +137,20 @@ val lookup = AgentTool( val chat = agents.agent("docs").chat(lookup) ``` +Or let a `@Serializable` class be the arguments and their schema. A property with no default +that is not nullable is required, `@SerialName` names it, and `@AgentTool.Describe` tells the +model what it is. Arguments that do not decode tell the model why, as a tool that throws does: + +```kotlin +@Serializable +data class WeatherLookup(@AgentTool.Describe("a city or region") val location: String) + +val weatherLookup = AgentTool.typed( + name = "weather_lookup", + description = "Look up the current weather for a location", +) { args -> weather.today(args.location) } +``` + Tools are answered inside `AgentSession` whether or not anybody collects `events()`, and a `tool_cancel` cancels the coroutine running one. `displayTitle` is what the people in the conversation see while it runs, and `executor = AgentTool.Executor.Client` shows the call as diff --git a/sdks/kotlin/core/src/main/kotlin/io/getstream/agents/android/core/AgentTool.kt b/sdks/kotlin/core/src/main/kotlin/io/getstream/agents/android/core/AgentTool.kt index 953f89fec5..66fd15811b 100644 --- a/sdks/kotlin/core/src/main/kotlin/io/getstream/agents/android/core/AgentTool.kt +++ b/sdks/kotlin/core/src/main/kotlin/io/getstream/agents/android/core/AgentTool.kt @@ -2,12 +2,21 @@ package io.getstream.agents.android.core import io.getstream.agents.android.core.generated.SessionTool import io.getstream.agents.android.core.generated.SessionToolApproval +import kotlinx.serialization.ExperimentalSerializationApi +import kotlinx.serialization.KSerializer +import kotlinx.serialization.SerialInfo +import kotlinx.serialization.descriptors.PrimitiveKind +import kotlinx.serialization.descriptors.SerialDescriptor +import kotlinx.serialization.descriptors.SerialKind +import kotlinx.serialization.descriptors.StructureKind +import kotlinx.serialization.descriptors.elementNames import kotlinx.serialization.json.JsonObject import kotlinx.serialization.json.add import kotlinx.serialization.json.buildJsonObject import kotlinx.serialization.json.put import kotlinx.serialization.json.putJsonArray import kotlinx.serialization.json.putJsonObject +import kotlinx.serialization.serializer /** * A function of yours the agent can call. @@ -90,6 +99,15 @@ public class AgentTool private constructor( val declineTitle: String? = null, ) + /** + * What an argument of a [typed] tool is, in words the model reads to fill it in. On a class, + * it describes the object wherever the class is an argument. + */ + @OptIn(ExperimentalSerializationApi::class) + @SerialInfo + @Target(AnnotationTarget.PROPERTY, AnnotationTarget.CLASS) + public annotation class Describe(public val description: String) + internal val schema: SessionTool get() = SessionTool( name = name, @@ -131,6 +149,90 @@ public class AgentTool private constructor( ): AgentTool = AgentTool(run, name, description, parameters, executor, displayTitle, approval) + /** + * A tool whose arguments are a `@Serializable` class, which is also its schema. + * + * Strings, numbers, booleans, enums, lists and nested `@Serializable` classes are + * described; a property with no default that is not nullable is required. Arguments + * that do not decode tell the model why, as a tool that throws does. + * + * @Serializable + * data class WeatherLookup(@AgentTool.Describe("a city or region") val location: String) + * + * AgentTool.typed("weather_lookup", "Look up the current weather.") { + * weather.today(it.location) + * } + */ + public inline fun typed( + name: String, + description: String, + executor: Executor? = null, + displayTitle: String? = null, + approval: Approval? = null, + noinline run: suspend (arguments: T) -> String, + ): AgentTool = decoding(serializer(), name, description, executor, displayTitle, approval, run) + + @PublishedApi + internal fun decoding( + serializer: KSerializer, + name: String, + description: String, + executor: Executor?, + displayTitle: String?, + approval: Approval?, + run: suspend (arguments: T) -> String, + ): AgentTool { + require(serializer.descriptor.kind == StructureKind.CLASS) { + "A typed tool's arguments are a class, not ${serializer.descriptor.serialName}" + } + return AgentTool( + { ToolOutput(run(wire.decodeFromJsonElement(serializer, it))) }, + name, description, schemaOf(serializer.descriptor), executor, displayTitle, approval, + ) + } + + @OptIn(ExperimentalSerializationApi::class) + private fun schemaOf(descriptor: SerialDescriptor, description: String? = descriptor.annotations.described): JsonObject = + buildJsonObject { + when (descriptor.kind) { + PrimitiveKind.STRING, PrimitiveKind.CHAR -> put("type", "string") + PrimitiveKind.BOOLEAN -> put("type", "boolean") + PrimitiveKind.BYTE, PrimitiveKind.SHORT, PrimitiveKind.INT, PrimitiveKind.LONG -> put("type", "integer") + PrimitiveKind.FLOAT, PrimitiveKind.DOUBLE -> put("type", "number") + SerialKind.ENUM -> { + put("type", "string") + putJsonArray("enum") { descriptor.elementNames.forEach { add(it) } } + } + StructureKind.LIST -> { + put("type", "array") + put("items", schemaOf(descriptor.getElementDescriptor(0))) + } + StructureKind.CLASS -> { + val elements = 0 until descriptor.elementsCount + put("type", "object") + putJsonObject("properties") { + for (i in elements) { + val element = descriptor.getElementDescriptor(i) + val described = descriptor.getElementAnnotations(i).described ?: element.annotations.described + put(descriptor.getElementName(i), schemaOf(element, described)) + } + } + putJsonArray("required") { + for (i in elements) { + if (!descriptor.isElementOptional(i) && !descriptor.getElementDescriptor(i).isNullable) { + add(descriptor.getElementName(i)) + } + } + } + } + else -> throw IllegalArgumentException("${descriptor.serialName} cannot be a tool argument") + } + description?.let { put("description", it) } + } + + private val List.described: String? + get() = filterIsInstance().firstOrNull()?.description + /** * A JSON Schema object for a tool whose arguments are all strings. * diff --git a/sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentSessionTest.kt b/sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentSessionTest.kt index c5f19c290a..190eee0f7f 100644 --- a/sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentSessionTest.kt +++ b/sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentSessionTest.kt @@ -24,6 +24,7 @@ import kotlinx.coroutines.cancelAndJoin import kotlinx.coroutines.coroutineScope import kotlinx.coroutines.flow.first import kotlinx.coroutines.runBlocking +import kotlinx.serialization.Serializable import kotlinx.serialization.json.Json import kotlinx.serialization.json.JsonPrimitive import kotlinx.serialization.json.jsonArray @@ -31,6 +32,9 @@ import kotlinx.serialization.json.jsonObject import kotlinx.serialization.json.jsonPrimitive private const val SESSION = """{"id":"s1","call_id":"","call_type":"default","user_id":"u1","agent_id":"a1","state":"live","modality":"text","created_at":"2026-09-24T04:03:19Z","text":true}""" +@Serializable +private data class WeatherLookup(val location: String) + private const val ASKED = """{"id":"r1","session_id":"s1","status":"running","created_at":"2026-09-24T04:03:19Z"}""" /** Socket tests run in real time, because what they wait for is a real connection. */ @@ -186,6 +190,26 @@ class AgentSessionTest { chat.close() } + @Test + fun `a typed tool called without a required argument tells the model which one`() = runBlocking { + val ran = AtomicBoolean(false) + val weather = AgentTool.typed("weather_lookup", "Look up the weather.") { + ran.set(true) + "sunny in ${it.location}" + } + val chat = agents.chat(SessionOptions(tools = listOf(weather))) + + router.script.send( + Script.Send("""{"type":"tool_call","id":"c1","name":"weather_lookup","arguments":"{\"city\":\"Boulder\"}","request_id":"","turn_id":""}"""), + ) + + val answer = Json.parseToJsonElement(router.nextSent()).jsonObject + assertTrue("'location'" in answer["error"]!!.jsonPrimitive.content, answer.toString()) + assertEquals(JsonPrimitive(""), answer["output"]) + assertFalse(ran.get()) + chat.close() + } + @Test fun `a tool's pictures go back beside its words`() = runBlocking { val photograph = AgentTool.withPictures("photograph_room", "Photograph the room.") { diff --git a/sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentToolTest.kt b/sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentToolTest.kt new file mode 100644 index 0000000000..41223840d2 --- /dev/null +++ b/sdks/kotlin/core/src/test/kotlin/io/getstream/agents/android/core/AgentToolTest.kt @@ -0,0 +1,66 @@ +package io.getstream.agents.android.core + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlinx.coroutines.runBlocking +import kotlinx.serialization.SerialName +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.jsonObject + +@Serializable +@AgentTool.Describe("a place on the map") +private data class Place(val city: String, val country: String? = null) + +@Serializable +private enum class Units { + @SerialName("metric") Metric, + @SerialName("imperial") Imperial, +} + +@Serializable +private data class Forecast( + @AgentTool.Describe("where to forecast") val place: Place, + @SerialName("days_ahead") val daysAhead: Int = 1, + val units: Units, + val note: String?, + val tags: List = emptyList(), + val home: Place?, + val score: Double, + val strict: Boolean, +) + +class AgentToolTest { + private val forecast = AgentTool.typed("forecast", "Forecast the weather.") { + "${it.place.city} ${it.daysAhead} ${it.units} ${it.note} ${it.home}" + } + + @Test + fun `a typed tool's schema is its class, required where nothing fills it in`() { + val place = """{"type":"object","properties":{"city":{"type":"string"},"country":{"type":"string"}},"required":["city"]""" + assertEquals( + Json.parseToJsonElement( + """{"type":"object","properties":{""" + + """"place":$place,"description":"where to forecast"},""" + + """"days_ahead":{"type":"integer"},""" + + """"units":{"type":"string","enum":["metric","imperial"]},""" + + """"note":{"type":"string"},""" + + """"tags":{"type":"array","items":{"type":"string"}},""" + + """"home":$place,"description":"a place on the map"},""" + + """"score":{"type":"number"},""" + + """"strict":{"type":"boolean"}},""" + + """"required":["place","units","score","strict"]}""", + ).jsonObject, + forecast.parameters, + ) + } + + @Test + fun `a typed tool runs on its decoded arguments, ignoring ones it does not know`() = runBlocking { + val arguments = Json.parseToJsonElement( + """{"place":{"city":"Boulder"},"days_ahead":3,"units":"imperial","score":0.5,"strict":true,"mood":"sunny"}""", + ).jsonObject + + assertEquals(ToolOutput("Boulder 3 Imperial null null"), forecast.run(arguments)) + } +} diff --git a/sdks/php/README.md b/sdks/php/README.md index 1c374fb96f..2950f3b573 100644 --- a/sdks/php/README.md +++ b/sdks/php/README.md @@ -221,6 +221,21 @@ foreach ($watch as $event) { } ``` +When every argument is a string, `Tools::strings` writes the schema, from each argument's +description by its name. Every argument is required unless you pass the list that is (the mobile +SDKs' `AgentTool.strings` requires none by default): + +```php +use GetStream\VisionAgents\Tools; + +$client->tools->register( + name: 'weather_lookup', + description: 'Look up the current weather for a location', + parameters: Tools::strings(['location' => 'a city or region']), + run: fn (array $args): string => weatherIn($args['location']), +); +``` + A tool runs while the session is watched, in its own fiber. What it throws is sent to the model as the tool's error. `register` also takes `displayTitle:`, what the reply's tool call shows, and `executor: 'client'` for a tool a person's device runs instead. diff --git a/sdks/php/src/Generated/ToolCallState.php b/sdks/php/src/Generated/ToolCallState.php index 3bf5192b30..9f01ec528c 100644 --- a/sdks/php/src/Generated/ToolCallState.php +++ b/sdks/php/src/Generated/ToolCallState.php @@ -34,6 +34,8 @@ public function __construct( public ?string $progress = null, // When a person declined the call, or its wait for them passed. public ?\DateTimeImmutable $rejectedAt = null, + // What the call is doing, in words for the people in the conversation, such as "Checking the weather": the di... + public ?string $displayTitle = null, ) { } @@ -54,6 +56,7 @@ public static function fromArray(array $data): self error: array_key_exists('error', $data) && $data['error'] !== null ? Json::string($data, 'error') : null, progress: array_key_exists('progress', $data) && $data['progress'] !== null ? Json::string($data, 'progress') : null, rejectedAt: array_key_exists('rejected_at', $data) && $data['rejected_at'] !== null ? Json::date($data, 'rejected_at') : null, + displayTitle: array_key_exists('display_title', $data) && $data['display_title'] !== null ? Json::string($data, 'display_title') : null, ); } @@ -84,6 +87,9 @@ public function toArray(): array if ($this->rejectedAt !== null) { $out['rejected_at'] = Json::dateValue($this->rejectedAt); } + if ($this->displayTitle !== null) { + $out['display_title'] = $this->displayTitle; + } return $out; } } diff --git a/sdks/php/src/Tools.php b/sdks/php/src/Tools.php index 0de2ac56f3..86f902036a 100644 --- a/sdks/php/src/Tools.php +++ b/sdks/php/src/Tools.php @@ -85,6 +85,26 @@ public function register( return $this; } + /** + * A JSON Schema for arguments that are all strings, to pass as `register`'s `$parameters`. + * + * $tools->register('weather_lookup', 'The weather somewhere', Tools::strings(['location' => 'a city or region']), $run); + * + * @param array $properties each argument's description, by its name + * @param ?list $required the arguments the model must send; every one when null, + * unlike the mobile SDKs' `AgentTool.strings`, which require none + * @return array + */ + public static function strings(array $properties, ?array $required = null): array + { + $schema = []; + foreach ($properties as $name => $description) { + $schema[$name] = ['type' => 'string', 'description' => $description]; + } + // Some models refuse a tool whose parameters have no `properties` object. + return ['type' => 'object', 'properties' => Json::objectValue($schema), 'required' => $required ?? array_keys($properties)]; + } + /** * What the session is opened with. * diff --git a/sdks/php/tests/Unit/ToolCallTest.php b/sdks/php/tests/Unit/ToolCallTest.php index f4f7d43740..fecad755e5 100644 --- a/sdks/php/tests/Unit/ToolCallTest.php +++ b/sdks/php/tests/Unit/ToolCallTest.php @@ -12,6 +12,7 @@ use GetStream\VisionAgents\Generated\SessionTool; use GetStream\VisionAgents\Json; use GetStream\VisionAgents\Session; +use GetStream\VisionAgents\Tests\Support\LocalRouter; use GetStream\VisionAgents\Tests\Support\LocalSocketServer; use GetStream\VisionAgents\Tests\Support\Rows; use GetStream\VisionAgents\ToolCallContext; @@ -152,6 +153,51 @@ public function testAToolIsHandedTheCallFirstOnlyWhenItTakesOne(): void self::assertSame('c1', $tools->call('only_call', '{}', $call)); } + public function testStringArgumentsAreAllRequiredUnlessTheCallerSaysWhich(): void + { + $properties = ['location' => 'a city or region', 'unit' => 'celsius or fahrenheit']; + $described = [ + 'location' => ['type' => 'string', 'description' => 'a city or region'], + 'unit' => ['type' => 'string', 'description' => 'celsius or fahrenheit'], + ]; + + self::assertSame(['type' => 'object', 'properties' => $described, 'required' => ['location', 'unit']], Tools::strings($properties)); + self::assertSame(['type' => 'object', 'properties' => $described, 'required' => ['location']], Tools::strings($properties, ['location'])); + self::assertSame([], Tools::strings($properties, [])['required']); + } + + public function testAToolWithNoStringArgumentsIsSentAnEmptyPropertiesObject(): void + { + $router = new LocalRouter(); + try { + $router->answer('POST', '/v1/agents/sessions', 201, Rows::session()); + $agent = new Agent(name: 'jean', client: $router->client()); + $agent->tools->register('ping', 'Checks the line', Tools::strings([]), static fn (array $args): string => 'pong'); + + $agent->chat(); + + self::assertStringContainsString( + '"tools":[{"description":"Checks the line","name":"ping","parameters":{"type":"object","properties":{},"required":[]}}]', + $router->to('POST', '/v1/agents/sessions')[0]->body, + ); + } finally { + $router->stop(); + } + } + + public function testAToolRegisteredWithNamedArgumentsRuns(): void + { + $tools = (new Tools())->register( + name: 'weather_lookup', + description: 'Look up the current weather for a location', + parameters: Tools::strings(['location' => 'a city or region']), + run: static fn (array $args): string => 'sunny in ' . Json::string($args, 'location'), + ); + + self::assertSame('sunny in Lyon', $tools->call('weather_lookup', '{"location":"Lyon"}')); + self::assertSame(['location'], Json::object($tools->declared()[0]->toArray(), 'parameters')['required']); + } + public function testAToolIsDeclaredWithOnlyThePoliciesItSets(): void { $tools = new Tools(); diff --git a/sdks/python/vision_agents/core/llm/function_registry.py b/sdks/python/vision_agents/core/llm/function_registry.py index a4573154be..5fc8e3461e 100644 --- a/sdks/python/vision_agents/core/llm/function_registry.py +++ b/sdks/python/vision_agents/core/llm/function_registry.py @@ -29,6 +29,15 @@ class FunctionParameter: # A {name} in an approval question, filled in from the call's arguments. _PLACEHOLDER = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*)\}") +# A Google-style docstring section the model is not told as part of the description. +_SECTION = re.compile( + r"^(Args|Arguments|Parameters|Keyword Args|Keyword Arguments|Returns|Return|" + r"Yields|Yield|Raises):\s*$" +) +_ARGUMENT_SECTIONS = {"Args", "Arguments", "Parameters"} +# An argument in an Args section: `name: description` or `name (type): description`. +_ARGUMENT = re.compile(r"^\*{0,2}(\w+)\s*(?:\([^)]*\))?\s*:\s*(.*)$") + @dataclass(frozen=True) class ToolPolicy: @@ -136,7 +145,8 @@ def decorator(func: Callable) -> Callable: ) func_name = name or func.__name__ - func_description = description or func.__doc__ or "" + summary, argument_descriptions = _parse_docstring(func.__doc__ or "") + func_description = description or summary context = context_parameter(func) # If explicit schema provided, use it directly @@ -157,7 +167,9 @@ def decorator(func: Callable) -> Callable: # Otherwise, extract from type hints type_hints = get_type_hints(func, include_extras=True) sig = inspect.signature(func) - arguments = Arguments(func_name, func, type_hints, context) + arguments = Arguments( + func_name, func, type_hints, context, argument_descriptions + ) parameters = [] for param_name, param in sig.parameters.items(): @@ -165,13 +177,12 @@ def decorator(func: Callable) -> Callable: continue param_type = type_hints.get(param_name, type(None)) - param_description = None parameters.append( FunctionParameter( name=param_name, type=param_type, - description=param_description, + description=argument_descriptions.get(param_name), required=param.default == inspect.Parameter.empty, default=param.default if param.default != inspect.Parameter.empty @@ -302,6 +313,51 @@ def _function_to_tool_schema(self, func_def: FunctionDefinition) -> ToolSchema: ) +def _parse_docstring(docstring: str) -> tuple[str, dict[str, str]]: + """Split a Google-style docstring into what the function does and what each argument means. + + Args: + docstring: The function's docstring. + + Returns: + The docstring up to its first Args, Returns or Raises section (the whole docstring + when it has none), and each argument's description from its Args section, with + continuation lines joined. + """ + lines = inspect.cleandoc(docstring).splitlines() + starts = [index for index, line in enumerate(lines) if _SECTION.match(line)] + if not starts: + return docstring, {} + + descriptions: dict[str, list[str]] = {} + for start, end in zip(starts, starts[1:] + [len(lines)]): + if lines[start].rstrip(":") not in _ARGUMENT_SECTIONS: + continue + current: Optional[list[str]] = None + indent: Optional[int] = None + for line in lines[start + 1 : end]: + if not line.strip(): + continue + depth = len(line) - len(line.lstrip()) + if depth == 0: + break + if indent is None: + indent = depth + if depth > indent: + if current is not None: + current.append(line.strip()) + continue + entry = _ARGUMENT.match(line.strip()) + current = None + if entry is not None: + current = descriptions[entry.group(1)] = [entry.group(2)] + summary = "\n".join(lines[: starts[0]]).strip() + return summary, { + name: " ".join(part for part in parts if part) + for name, parts in descriptions.items() + } + + def _check_approval(name: str, approval: str, arguments: Any) -> None: """Refuse an approval question that names an argument `name` does not take.""" for placeholder in _PLACEHOLDER.findall(approval): diff --git a/sdks/python/vision_agents/core/llm/tool_arguments.py b/sdks/python/vision_agents/core/llm/tool_arguments.py index 9672b0792c..78cab6e183 100644 --- a/sdks/python/vision_agents/core/llm/tool_arguments.py +++ b/sdks/python/vision_agents/core/llm/tool_arguments.py @@ -4,11 +4,12 @@ from __future__ import annotations import inspect -from typing import Annotated, Any, Callable, Optional +from typing import Annotated, Any, Callable, Optional, get_origin from pydantic import ( BaseModel, ConfigDict, + Field, PydanticSchemaGenerationError, TypeAdapter, ValidationError, @@ -45,8 +46,12 @@ def __init__( func: Callable, type_hints: dict[str, Any], context: Optional[str] = None, + descriptions: Optional[dict[str, str]] = None, ): - """`context` names the parameter handed the `ToolCallContext`, which the model does not fill in.""" + """`context` names the parameter handed the `ToolCallContext`, which the model does not fill in. + + `descriptions` says what each argument means, for one whose type does not already say. + """ self.name = name parameters = [ parameter @@ -71,6 +76,9 @@ def __init__( if parameter.default is inspect.Parameter.empty else parameter.default ) + description = (descriptions or {}).get(parameter.name) + if description and not _described(annotation): + default = Field(default, description=description) fields[parameter.name] = (annotation, default) self.model = create_model(f"{name}_arguments", __config__=_CONFIG, **fields) @@ -125,6 +133,14 @@ def _describable(annotation: Any) -> Any: return annotation +def _described(annotation: Any) -> bool: + """Whether the annotation carries a `Field` description of its own.""" + return get_origin(annotation) is Annotated and any( + isinstance(metadata, FieldInfo) and metadata.description + for metadata in annotation.__metadata__ + ) + + def _plain(schema: Any, definitions: dict[str, Any]) -> Any: """`schema` with each `$ref` replaced by its definition, titles dropped, and an optional value's `anyOf: [X, null]` read as X, which is how a function's optional argument was diff --git a/sdks/ruby/README.md b/sdks/ruby/README.md index 64980cc09b..645890cb04 100644 --- a/sdks/ruby/README.md +++ b/sdks/ruby/README.md @@ -57,6 +57,17 @@ the conversation on in writing again; `session.voice.started?` says which. agent.chat { |session| session.responses.create("What changed in v3?") } ``` +`params:` is the short form for string arguments: each name is required, and its value says +what it is for. A block that takes keywords is handed the arguments it names, and a call +missing a required one fails with an error the model is told: + +```ruby +agent.tools.register("weather_lookup", description: "Look up the current weather for a location", + params: { location: "a city or region" }) do |location:| + weather_in(location) +end +``` + A tool can say how the backend treats its calls, and its block can take the call it runs for as its first argument, ahead of the arguments: diff --git a/sdks/ruby/lib/getstream/vision_agents/generated/spec.rb b/sdks/ruby/lib/getstream/vision_agents/generated/spec.rb index cb30e82b7f..fd93ce500b 100644 --- a/sdks/ruby/lib/getstream/vision_agents/generated/spec.rb +++ b/sdks/ruby/lib/getstream/vision_agents/generated/spec.rb @@ -3222,7 +3222,7 @@ module Generated required: %w[allowed request_id tool_call_id type].freeze, open: false }.freeze, "ToolCallState" => { - properties: %w[approved_at duration_ms error progress rejected_at started_at status tool tool_call_id turn_id type].freeze, + properties: %w[approved_at display_title duration_ms error progress rejected_at started_at status tool tool_call_id turn_id type].freeze, required: %w[type tool_call_id tool turn_id status started_at duration_ms].freeze, open: false }.freeze, "ToolGroup" => { diff --git a/sdks/ruby/lib/getstream/vision_agents/tools.rb b/sdks/ruby/lib/getstream/vision_agents/tools.rb index 55717f8049..29dbe66fcb 100644 --- a/sdks/ruby/lib/getstream/vision_agents/tools.rb +++ b/sdks/ruby/lib/getstream/vision_agents/tools.rb @@ -18,6 +18,14 @@ module VisionAgents # agent.tools.register("book_flight", description: "Book a flight", ...) do |tool_call_context, args| # bookings.book(args["flight_id"], key: tool_call_context.id) # end + # + # params: names each argument a required string, with what it is for, and a block that + # takes keywords is handed only the arguments it names: + # + # agent.tools.register("get_weather", description: "Weather for a city", + # params: { city: "a city or region" }) do |city:| + # weather_in(city) + # end class Tools Tool = Data.define(:name, :description, :parameters, :executor, :display_title, :interruptible, :timeout, :approval, :run) @@ -29,6 +37,8 @@ def initialize @lock = Mutex.new end + # @param params [Hash] argument name => what it is for, each a required string, in place + # of a JSON Schema in parameters:. # @param executor [String] who runs it: "server", the default, or "client" for a tool a # person's device runs. This process still answers it, once the device has reported. # @param display_title [String] what a call is doing, in words for the people in the @@ -41,8 +51,11 @@ def initialize # "Book flight {flight_id}?", where each {name} is an argument filled in from the call. # On a voice call the agent asks it out loud and waits 15 seconds for a yes; in # writing it goes to the app. - def register(name, description:, parameters: { type: "object", properties: {} }, executor: nil, + def register(name, description:, parameters: nil, params: nil, executor: nil, display_title: nil, interruptible: false, timeout: nil, approval: nil, &run) + raise ArgumentError, "#{name} takes params: or parameters:, not both" if params && parameters + + parameters ||= params ? strings(params) : { type: "object", properties: {} } raise ConfigurationError, "a tool needs a name" if name.to_s.empty? raise ConfigurationError, "#{name} needs a description, since it is all the model sees" if description.to_s.empty? raise ConfigurationError, "#{name} needs a block to run" unless run @@ -120,9 +133,29 @@ def adopt(tools) private # A block that takes two arguments, or any number, is handed the call first; one that - # takes one is handed only the arguments. + # takes one is handed only the arguments. A block that takes keywords is handed the + # arguments as keywords, and the call first when it takes any positional argument. def invoke(tool, parsed, tool_call_context) - takes_call?(tool.run) ? tool.run.call(tool_call_context, parsed) : tool.run.call(parsed) + run = tool.run + kinds = run.parameters.map(&:first) + return keywords(run, kinds, parsed, tool_call_context) if kinds.intersect?(%i[keyreq key keyrest]) + + takes_call?(run) ? run.call(tool_call_context, parsed) : run.call(parsed) + end + + # Only the keywords the block names, unless it takes **: an argument the model made up + # is not the block's mistake. + def keywords(run, kinds, parsed, tool_call_context) + arguments = parsed.transform_keys(&:to_sym) + unless kinds.include?(:keyrest) + arguments = arguments.slice(*run.parameters.filter_map { |kind, name| name if %i[keyreq key].include?(kind) }) + end + kinds.intersect?(%i[req opt rest]) ? run.call(tool_call_context, **arguments) : run.call(**arguments) + end + + def strings(params) + { type: "object", properties: params.transform_values { |about| { type: "string", description: about } }, + required: params.keys.map(&:to_s) } end def takes_call?(run) diff --git a/sdks/ruby/test/tool_call_test.rb b/sdks/ruby/test/tool_call_test.rb index e552840860..7a9e089e9b 100644 --- a/sdks/ruby/test/tool_call_test.rb +++ b/sdks/ruby/test/tool_call_test.rb @@ -56,6 +56,51 @@ def test_a_block_of_two_is_handed_its_call_first_and_a_block_of_one_only_the_arg assert_equal "done", tools.call("none", "{}") end + def test_params_declares_each_argument_a_required_string + tools = VA::Tools.new.register("weather_lookup", description: "Weather", params: { location: "a city or region" }, + approval: "Weather in {location}?") { |location:| location } + + assert_equal({ type: "object", properties: { location: { type: "string", description: "a city or region" } }, + required: ["location"] }, tools.declarations.first[:parameters]) + end + + def test_params_and_parameters_together_are_refused + assert_raises(ArgumentError) do + VA::Tools.new.register("weather_lookup", description: "Weather", params: { location: "a city" }, + parameters: BOOKING) { "" } + end + end + + def test_a_block_of_keywords_is_handed_the_arguments_it_names + tools = VA::Tools.new + tools.register("weather_lookup", description: "Weather", params: { location: "a city" }) do |location:, units: "C"| + "#{location} #{units}" + end + tools.register("everything", description: "All of it") { |location:, **rest| [location, rest] } + + assert_equal "Paris C", tools.call("weather_lookup", '{"location":"Paris","mood":"sunny"}') + assert_equal "Oslo F", tools.call("weather_lookup", '{"location":"Oslo","units":"F"}') + assert_equal '["Paris",{"mood":"sunny"}]', tools.call("everything", '{"location":"Paris","mood":"sunny"}') + end + + def test_a_block_of_keywords_missing_one_it_requires_fails_the_call + tools = VA::Tools.new + tools.register("weather_lookup", description: "Weather", params: { location: "a city" }) { |location:| location } + + missing = assert_raises(ArgumentError) { tools.call("weather_lookup", '{"city":"Paris"}') } + assert_match(/location/, missing.message) + end + + def test_a_block_of_keywords_with_a_positional_argument_is_handed_its_call_first + tools = VA::Tools.new + tools.register("weather_lookup", description: "Weather", + params: { location: "a city" }) do |tool_call_context, location:| + "#{tool_call_context.id} #{location}" + end + + assert_equal "t1 Paris", tools.call("weather_lookup", '{"location":"Paris"}', VA::ToolCallContext.new(id: "t1")) + end + def test_a_call_without_a_socket_does_nothing tool_call_context = VA::ToolCallContext.new diff --git a/sdks/rust/Cargo.lock b/sdks/rust/Cargo.lock index 42ad26c454..7dadfcc229 100644 --- a/sdks/rust/Cargo.lock +++ b/sdks/rust/Cargo.lock @@ -492,7 +492,7 @@ dependencies = [ "heck", "prettyplease", "quote", - "schemars", + "schemars 0.8.22", "serde_json", "syn 3.0.6", "typify", @@ -1244,6 +1244,26 @@ dependencies = [ "yasna", ] +[[package]] +name = "ref-cast" +version = "1.0.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e440fb4e4b4147295338efb76001ab9e4efc0e5839df2c47fc5ac2381d365c3" +dependencies = [ + "ref-cast-impl", +] + +[[package]] +name = "ref-cast-impl" +version = "1.0.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + [[package]] name = "regress" version = "0.12.0" @@ -1455,7 +1475,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3fbf2ae1b8bc8e02df939598064d22402220cd5bbcca1c76f7d6a310974d5615" dependencies = [ "dyn-clone", - "schemars_derive", + "schemars_derive 0.8.22", + "serde", + "serde_json", +] + +[[package]] +name = "schemars" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab508826f74a77ca9d5aba6ff19b522583ee3eaf28a19384ff3d0e5835fadf6e" +dependencies = [ + "dyn-clone", + "ref-cast", + "schemars_derive 1.2.3", "serde", "serde_json", ] @@ -1468,10 +1501,22 @@ checksum = "32e265784ad618884abaea0600a9adf15393368d840e0222d101a072f3f7534d" dependencies = [ "proc-macro2", "quote", - "serde_derive_internals", + "serde_derive_internals 0.29.1", "syn 2.0.119", ] +[[package]] +name = "schemars_derive" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1c3a92094fa7d61aa124645844facb6b554dfc797136d0f5fd1f890e2bffc69" +dependencies = [ + "proc-macro2", + "quote", + "serde_derive_internals 0.30.0", + "syn 3.0.6", +] + [[package]] name = "security-framework" version = "3.7.0" @@ -1542,6 +1587,17 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "serde_derive_internals" +version = "0.30.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f852137cce035d6a4df67ccce505ff6b3e9fd3a10e3e52b24dc71e650bb1a9bd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + [[package]] name = "serde_json" version = "1.0.151" @@ -2008,7 +2064,7 @@ dependencies = [ "proc-macro2", "quote", "regress", - "schemars", + "schemars 0.8.22", "semver", "serde", "serde_json", @@ -2066,6 +2122,7 @@ dependencies = [ "rand 0.10.3", "rcgen", "reqwest", + "schemars 1.2.3", "serde", "serde_json", "sha2", diff --git a/sdks/rust/Cargo.toml b/sdks/rust/Cargo.toml index 4d7ab17773..719b99b6ff 100644 --- a/sdks/rust/Cargo.toml +++ b/sdks/rust/Cargo.toml @@ -23,6 +23,7 @@ tokio-util = "0.7" futures-util = { version = "0.3", default-features = false, features = ["sink", "std"] } serde = { version = "1.0.181", features = ["derive"] } serde_json = "1" +schemars = "1" thiserror = "2" yaml-rust2 = "0.13" md-5 = "0.11" diff --git a/sdks/rust/src/lib.rs b/sdks/rust/src/lib.rs index 4e7f60d774..f25c1fe574 100644 --- a/sdks/rust/src/lib.rs +++ b/sdks/rust/src/lib.rs @@ -56,3 +56,5 @@ pub use teams::{TeamLimit, Teams}; pub use tool_call::{ToolCallContext, ToolConversation, ToolUser}; pub use tools::Tools; pub use usd::Usd; + +pub use schemars; diff --git a/sdks/rust/src/tools.rs b/sdks/rust/src/tools.rs index e14313a20f..41fa5859d8 100644 --- a/sdks/rust/src/tools.rs +++ b/sdks/rust/src/tools.rs @@ -6,7 +6,10 @@ use std::time::Duration; use futures_util::FutureExt; use futures_util::future::BoxFuture; +use schemars::JsonSchema; +use schemars::generate::SchemaSettings; use serde::Serialize; +use serde::de::DeserializeOwned; use serde_json::{Map, Value}; use crate::error::{Error, Result as CrateResult}; @@ -68,6 +71,30 @@ impl Tools { self.register_tool(declaration, run) } + /// [`Tools::register`] for a function whose arguments are a type: the model is offered + /// its JSON Schema, doc comments as descriptions, and the function is handed the arguments + /// decoded. Arguments that do not decode are sent back as the tool's error, so the model + /// can try again. + pub fn register_typed(&self, name: &str, description: &str, run: F) -> &Self + where + T: DeserializeOwned + JsonSchema + 'static, + F: Fn(T) -> Fut + Send + Sync + 'static, + Fut: Future> + Send + 'static, + O: Serialize, + E: Display, + { + let parameters = Value::Object(parameters_of::()); + self.register(name, description, parameters, move |arguments| { + let running = serde_json::from_value::(arguments).map(&run); + async move { + match running { + Ok(running) => running.await.map_err(|error| error.to_string()), + Err(error) => Err(format!("the arguments do not fit the parameters: {error}")), + } + } + }) + } + /// [`Tools::register`], declared whole: for a `display_title` shown on the reply, or an /// `executor` of `client` for a tool a person's device runs, whose call is still answered /// here once the device has reported. @@ -199,6 +226,19 @@ impl Tools { } } +/// `T`'s JSON Schema as a plain object, every subschema inline, as the other SDKs send it. +fn parameters_of() -> Map { + let mut settings = SchemaSettings::draft2020_12(); + settings.inline_subschemas = true; + let schema = settings.into_generator().into_root_schema_for::(); + let Value::Object(mut parameters) = Value::from(schema) else { + return Map::new(); + }; + parameters.remove("$schema"); + parameters.remove("title"); + parameters +} + /// Refuses a declaration whose policies cannot work, before any session is offered it. fn check(declaration: &types::SessionTool) -> CrateResult<()> { let name = &declaration.name; diff --git a/sdks/rust/tests/session.rs b/sdks/rust/tests/session.rs index 97133545a1..406e033a7b 100644 --- a/sdks/rust/tests/session.rs +++ b/sdks/rust/tests/session.rs @@ -1307,6 +1307,105 @@ fn a_policy_that_cannot_work_is_refused_when_it_is_registered() { assert!(tools.has("refund")); } +#[derive(serde::Deserialize, vision_agents::schemars::JsonSchema)] +#[schemars(crate = "vision_agents::schemars")] +struct WeatherLookup { + /// a city or region + location: String, + /// how many days ahead + days: Option, + near: Option, +} + +#[derive(serde::Deserialize, vision_agents::schemars::JsonSchema)] +#[schemars(crate = "vision_agents::schemars")] +struct Place { + /// how far from the location, in km + km: f64, +} + +fn typed_weather() -> Tools { + let tools = Tools::new(); + tools.register_typed( + "weather_lookup", + "Look up the current weather for a location", + async |args: WeatherLookup| { + let near = args.near.map_or(0.0, |place| place.km); + Ok::<_, String>(format!( + "sunny in {} for {} days, {near} km out", + args.location, + args.days.unwrap_or(1) + )) + }, + ); + tools +} + +#[test] +fn a_typed_tool_is_offered_its_types_schema_as_a_plain_object() { + let declared = typed_weather().declared(); + let parameters = Value::Object(declared[0].parameters.clone()); + + assert_eq!(parameters["type"], "object"); + assert_eq!(parameters["required"], json!(["location"])); + assert_eq!( + parameters["properties"]["location"], + json!({"type": "string", "description": "a city or region"}) + ); + assert_eq!( + parameters["properties"]["days"]["description"], + "how many days ahead" + ); + assert_eq!( + parameters["properties"]["near"]["properties"]["km"]["description"], + "how far from the location, in km", + "the nested type is not inline: {parameters}" + ); + for absent in ["$schema", "title", "$defs"] { + assert!(parameters.get(absent).is_none(), "{absent} in {parameters}"); + } +} + +#[tokio::test] +async fn a_typed_tool_is_handed_its_arguments_decoded() { + let tools = typed_weather(); + + assert_eq!( + tools + .call( + "weather_lookup", + r#"{"location": "Oslo", "days": 3, "near": {"km": 2.5}}"# + ) + .await, + Ok(json!("sunny in Oslo for 3 days, 2.5 km out")) + ); + assert_eq!( + tools + .call("weather_lookup", r#"{"location": "Lima"}"#) + .await, + Ok(json!("sunny in Lima for 1 days, 0 km out")) + ); +} + +#[tokio::test] +async fn arguments_that_do_not_fit_a_typed_tool_are_the_models_error() { + let tools = typed_weather(); + + assert_eq!( + tools.call("weather_lookup", r#"{"days": 3}"#).await, + Err("the arguments do not fit the parameters: missing field `location`".to_string()) + ); + assert_eq!( + tools + .call("weather_lookup", r#"{"location": "Oslo", "days": "soon"}"#) + .await, + Err( + "the arguments do not fit the parameters: invalid type: string \"soon\", expected u32" + .to_string() + ) + ); +} + const SESSION_TOOLS: &str = "/v1/agents/sessions/s1/tools"; const RESERVE_TABLE: &str = "/v1/agents/sessions/s1/tools/reserve_table"; diff --git a/sdks/swift/core/Sources/StreamAgents/AgentTool.swift b/sdks/swift/core/Sources/StreamAgents/AgentTool.swift index 998d42c355..fa713e5049 100644 --- a/sdks/swift/core/Sources/StreamAgents/AgentTool.swift +++ b/sdks/swift/core/Sources/StreamAgents/AgentTool.swift @@ -105,6 +105,79 @@ public struct AgentTool: Sendable { displayTitle: displayTitle, approval: approval ) { ToolOutput(text: try await run($0)) } } + + /// A tool whose arguments are decoded into a type of yours, answering with pictures as well + /// as words. Arguments that do not decode fail the call, and the model is told why. + public init( + name: String, + description: String, + parameters: RawJSON? = nil, + executor: Executor? = nil, + displayTitle: String? = nil, + approval: Approval? = nil, + run: @escaping @Sendable (Arguments) async throws -> ToolOutput + ) { + self.init( + name: name, description: description, parameters: parameters, executor: executor, + displayTitle: displayTitle, approval: approval, + run: { (arguments: [String: RawJSON]) in try await run(Self.decode(arguments)) }) + } + + /// A tool whose arguments are decoded into a type of yours, answering in words. Arguments + /// that do not decode fail the call, and the model is told why. + /// + /// struct WeatherLookup: Decodable { let location: String } + /// + /// AgentTool(name: "weather_lookup", description: "Look up the weather", + /// parameters: .strings(["location": "a city"], required: ["location"])) { + /// (args: WeatherLookup) in try await weather.today(args.location) + /// } + public init( + name: String, + description: String, + parameters: RawJSON? = nil, + executor: Executor? = nil, + displayTitle: String? = nil, + approval: Approval? = nil, + run: @escaping @Sendable (Arguments) async throws -> String + ) { + self.init( + name: name, description: description, parameters: parameters, executor: executor, + displayTitle: displayTitle, approval: approval, + run: { (arguments: Arguments) in ToolOutput(text: try await run(arguments)) }) + } + + private static func decode(_ arguments: [String: RawJSON]) throws -> Arguments { + do { + return try JSONDecoder().decode(Arguments.self, from: JSONEncoder().encode(arguments)) + } catch let error as DecodingError { + throw ToolArgumentsError(failure: error) + } + } +} + +/// Arguments the model sent that do not decode into a tool's type, said in words the model can +/// correct them from: `DecodingError`'s own description names no field. +struct ToolArgumentsError: LocalizedError { + let failure: DecodingError + + var errorDescription: String? { + switch failure { + case .keyNotFound(let key, let context): + return "the arguments have no \"\(Self.path(context.codingPath + [key]))\"" + case .typeMismatch(_, let context), .valueNotFound(_, let context), .dataCorrupted(let context): + let path = Self.path(context.codingPath) + return path.isEmpty + ? "the arguments do not fit the tool: \(context.debugDescription)" + : "the arguments do not fit the tool at \"\(path)\": \(context.debugDescription)" + @unknown default: + return failure.localizedDescription + } + } + + private static func path(_ keys: [any CodingKey]) -> String { + keys.map { $0.intValue.map(String.init) ?? $0.stringValue }.joined(separator: ".") + } } extension AgentTool { diff --git a/sdks/swift/core/Tests/StreamAgentsTests/AgentSessionTests.swift b/sdks/swift/core/Tests/StreamAgentsTests/AgentSessionTests.swift index 9110fe50f5..325f098e30 100644 --- a/sdks/swift/core/Tests/StreamAgentsTests/AgentSessionTests.swift +++ b/sdks/swift/core/Tests/StreamAgentsTests/AgentSessionTests.swift @@ -264,6 +264,46 @@ import os } } + private struct Booking: Decodable { + let location: String + let guests: Int + } + + /// A typed tool runs with the model's arguments decoded, and arguments missing a field reach + /// the model as the tool failing, naming the field, rather than running it with a blank. + @Test(arguments: [ + (#"{"location":"Boulder","guests":4}"#, "Booked Boulder for 4", ""), + (#"{"guests":4}"#, nil, #""location""#), + (#"{"location":"Boulder","guests":"four"}"#, nil, #""guests""#), + ]) + func aTypedToolGetsItsArgumentsDecoded(arguments: String, output: String?, error: String) async throws { + let server = try ToolEventServer(remoteTool: "server_lookup", arguments: arguments) + defer { server.listener.cancel() } + let url = try #require(try await server.addresses.first(where: { @Sendable _ in true })) + let tool = AgentTool(name: "local_lookup", description: "Book a table") { (booking: Booking) in + "Booked \(booking.location) for \(booking.guests)" + } + let session = AgentSession( + backend: Backend(url: url, customerID: "test"), session: Self.session, tools: [tool]) + await session.start() + do { + let data = try #require(try await server.commands.first(where: { @Sendable _ in true })) + let command = try JSONDecoder().decode([String: RawJSON].self, from: data) + #expect(command["type"]?.stringValue == "tool_result") + if let output { + #expect(command["output"]?.stringValue == output) + #expect(command["error"]?.stringValue == "") + } else { + #expect(command["output"]?.stringValue == "") + #expect(command["error"]?.stringValue?.contains(error) == true, "\(command["error"] ?? .nil)") + } + await session.close() + } catch { + await session.close() + throw error + } + } + @Test(arguments: [true, false]) func aCameraOnACallAnswersTheRequestToLook(recent: Bool) async throws { let server = try ToolEventServer(remoteTool: "get_video_frames") @@ -474,7 +514,8 @@ private struct ToolEventServer { let addresses: AsyncThrowingStream let commands: AsyncThrowingStream - init(remoteTool: String) throws { + init(remoteTool: String, arguments: String = "{}") throws { + let quoted = String(decoding: try JSONEncoder().encode(arguments), as: UTF8.self) let queue = DispatchQueue(label: "tool-event-server") let webSocket = NWProtocolWebSocket.Options() webSocket.autoReplyPing = true @@ -507,7 +548,7 @@ private struct ToolEventServer { connection.stateUpdateHandler = { state in guard case .ready = state else { return } for (id, name) in [("remote", remoteTool), ("local", "local_lookup")] { - let frame = #"{"type":"tool_call","id":"\#(id)","name":"\#(name)","arguments":"{}"}"# + let frame = #"{"type":"tool_call","id":"\#(id)","name":"\#(name)","arguments":\#(quoted)}"# let context = NWConnection.ContentContext( identifier: id, metadata: [NWProtocolWebSocket.Metadata(opcode: .text)]) diff --git a/tests/test_function_calling.py b/tests/test_function_calling.py index 4dc6b216b7..e41a708915 100644 --- a/tests/test_function_calling.py +++ b/tests/test_function_calling.py @@ -2,12 +2,13 @@ Tests for function calling functionality. """ -from typing import Any, Optional +from typing import Annotated, Any, Optional from unittest.mock import Mock, patch import pytest +from pydantic import BaseModel, Field from vision_agents.core.edge.types import Participant -from vision_agents.core.llm import FunctionRegistry, function_registry +from vision_agents.core.llm import FunctionRegistry, ToolCallContext, function_registry from vision_agents.core.llm.llm import LLM, LLMResponseEvent from vision_agents.plugins.anthropic import LLM as ClaudeLLM from vision_agents.plugins.gemini import LLM as GeminiLLM @@ -104,6 +105,134 @@ async def test_func(x: int, y: int = 5) -> int: assert schemas[0]["description"] == "Test function" assert "parameters_schema" in schemas[0] + async def test_docstring_args_section_is_left_out_of_the_description(self): + registry = FunctionRegistry() + + @registry.register() + async def weather_lookup(location: str) -> str: + """Look up the current weather for a location. + + Uses the nearest station. + + Args: + location: a city or region. + + Returns: + The weather, in words. + """ + return location + + assert registry.get_tool_schemas()[0]["description"] == ( + "Look up the current weather for a location.\n\nUses the nearest station." + ) + + async def test_docstring_args_describe_each_argument(self): + registry = FunctionRegistry() + + @registry.register() + async def weather_lookup( + context: ToolCallContext, location: str, units: str = "metric" + ) -> str: + """Look up the current weather for a location. + + Args: + context: the call this runs for. + location: a city or region. + units (str): metric or imperial. + """ + return location + + assert registry.get_tool_schemas()[0]["parameters_schema"] == { + "type": "object", + "properties": { + "location": {"type": "string", "description": "a city or region."}, + "units": { + "type": "string", + "default": "metric", + "description": "metric or imperial.", + }, + }, + "required": ["location"], + } + + async def test_multi_line_docstring_arg_description_is_joined(self): + registry = FunctionRegistry() + + @registry.register() + async def weather_lookup(location: str, days: int) -> str: + """Look up the weather. + + Args: + location: a city or region, + such as Amsterdam or Tuscany. + days: how many days ahead. + """ + return location + + properties = registry.get_tool_schemas()[0]["parameters_schema"]["properties"] + assert ( + properties["location"]["description"] + == "a city or region, such as Amsterdam or Tuscany." + ) + assert properties["days"]["description"] == "how many days ahead." + + async def test_field_description_wins_over_docstring(self): + registry = FunctionRegistry() + + @registry.register(description="Weather where you say") + async def weather_lookup( + location: Annotated[str, Field(description="City name, e.g. Amsterdam")], + ) -> str: + """Look up the weather. + + Args: + location: a city or region. + """ + return location + + schema = registry.get_tool_schemas()[0] + assert schema["description"] == "Weather where you say" + assert schema["parameters_schema"]["properties"]["location"] == { + "type": "string", + "description": "City name, e.g. Amsterdam", + } + + async def test_model_field_description_wins_over_docstring(self): + class Lookup(BaseModel): + location: str = Field(description="City name, e.g. Amsterdam") + + registry = FunctionRegistry() + + @registry.register() + async def weather_lookup(lookup: Lookup) -> str: + """Look up the weather. + + Args: + lookup: where to look. + """ + return lookup.location + + assert registry.get_tool_schemas()[0]["parameters_schema"]["properties"] == { + "location": {"type": "string", "description": "City name, e.g. Amsterdam"} + } + + async def test_docstring_without_args_is_the_description_as_written(self): + registry = FunctionRegistry() + + @registry.register() + async def weather_lookup(location: str) -> str: + """Look up the weather. + + Say which city: a region is too broad. + """ + return location + + schema = registry.get_tool_schemas()[0] + assert schema["description"] == weather_lookup.__doc__ + assert schema["parameters_schema"]["properties"] == { + "location": {"type": "string"} + } + async def test_get_callable(self): """Test getting callable function.""" registry = FunctionRegistry()