Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/skills/sdk/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -928,3 +928,4 @@ Connections (AI-3376): Go has `client.Connections()` for the accounts the app an
The session socket sends `reasoning_delta` (`turn_id`, `text`): the next piece of the model's thinking on a written reply, which a kept conversation also streams into the reply's Chat message and an incognito one has nowhere else to show. JavaScript folds it, with `tool_started`, `tool_updated` (not `verbatim`) and `tool_ran`, into each agent turn's `steps` in `sdks/js/src/conversation.ts`: a round of thinking `r1`, `r2`... opens on a delta after anything else and completes when the turn writes text or UI, calls a tool, answers or is cut off; a call runs until `tool_ran`, failed when it carries an `error`, its summary the last non-verbatim update. A turn is `done` on `responded` with no `pending_work`, or `interrupted`. `AgentChat` shows a session with no `conversationId` from these turns (`sdks/js/src/turn-messages.ts`): the agent's turns between two of the person's messages are one reply, shaped as the channel's messages (top level in stream-chat 9, under `custom` in 10) with `ai_reasoning` and `ai_tool_call` steps, rounds numbered across the reply, its whole thinking given to `reasoning`, and a call waiting in `approvals` shown `awaiting_approval` with its question and the session's `user_id` as `target_user_id`. The clients that carry the socket's description (JavaScript types, Python `plugins/stream`) are regenerated. Swift does the same: `Conversation` folds the frames into `Turn.steps` and `Turn.isDone`, and `TurnMessage.messages(of:approvals:userID:isConnected:)` (`session.turnMessages`) builds the replies, each step an attachment type and JSON payload for `AIMessagePart.parts(from:)`, the session's user now `Session.userID`; `swift_assistant`'s `AgentChat` shows an incognito session from them. Other SDKs need to update: Kotlin and Dart, whose AI components read the same steps, should fold the same frames into their `Conversation` turns and show a session with no channel through their transports; the server SDKs need nothing.
The session socket sends `reasoning_delta` (`turn_id`, `text`): the next piece of the model's thinking on a written reply, which a kept conversation also streams into the reply's Chat message and an incognito one has nowhere else to show. JavaScript folds it, with `tool_started`, `tool_updated` (not `verbatim`) and `tool_ran`, into each agent turn's `steps` in `sdks/js/src/conversation.ts`: a round of thinking `r1`, `r2`... opens on a delta after anything else and completes when the turn writes text or UI, calls a tool, answers or is cut off; a call runs until `tool_ran`, failed when it carries an `error`, its summary the last non-verbatim update. A turn is `done` on `responded` with no `pending_work`, or `interrupted`. `AgentChat` shows a session with no `conversationId` from these turns (`sdks/js/src/turn-messages.ts`): the agent's turns between two of the person's messages are one reply, shaped as the channel's messages (top level in stream-chat 9, under `custom` in 10) with `ai_reasoning` and `ai_tool_call` steps, rounds numbered across the reply, its whole thinking given to `reasoning`, and a call waiting in `approvals` shown `awaiting_approval` with its question and the session's `user_id` as `target_user_id`. The clients that carry the socket's description (JavaScript types, Python `plugins/stream`) are regenerated. Other SDKs need to update: Swift, Kotlin and Dart, whose AI components read the same steps, should fold the same frames into their `Conversation` turns and show a session with no channel through their transports; the server SDKs need nothing.
An agent's UI instructions are kept apart from its instructions, so a session can turn UI off: `AgentConfig`, `AgentConfigRequest`, `AgentConfigPatch` and `SyncAgentRequest` have `ui_instructions` (an additive migration), and `CreateSessionRequest` has `ui`, a boolean a device may send. The router tells the model `ui_instructions` after the instructions (a backend's own session instructions included) unless the session was opened with `ui: false`, which reopening or forking it keeps (`agent_sessions.no_ui`). A sync sends the directory whole, so a sync without `ui_instructions` removes them; a PUT that leaves them out keeps them, so a client that does not know the field does not take the UI away. Go's `Load` compiles agent.yaml's `ui:` into `Folder.UIInstructions` and `syncFolder` sends it as `ui_instructions`; the folder's hash takes it in only when there is UI, so a directory without `ui:` is not synced again, and one with it is synced once more, which moves the UI out of the stored instructions. A router older than the field takes the sync and drops it, so `syncFolder` does not trust a stamp covering UI that the stored config lacks, and syncs again, and the router writes a sync of the same hash that declares UI the config lacks rather than finding it unchanged. Deploy the router before releasing an SDK that sends the field. JavaScript does the same: `loadFolder` puts the compiled `ui:` in `Folder.uiInstructions`, `Agent.sync` sends it as `ui_instructions` (left out without `ui:`, so such a directory keeps its fingerprint), and a stamp covering UI the stored config lacks is synced again. When a sync's answer has no `ui_instructions` though it sent them (a router older than the field), JavaScript syncs once more with the UI joined onto `instructions`, as before, under that body's own fingerprint and without writing the stamp, so the agent keeps its UI and the next sync moves it apart once the router knows the field. That fallback cannot honour `ui: false`: the UI is in `instructions` then, which every session is told, so the Interactive UI switch needs a router that knows the field and a sync it stored apart. Go only syncs again there, so an older router leaves a Go-synced agent with no UI, and Go should fall back the same way. `Agent`'s `instructions` option replaces instructions.md only: the folder's `ui:` is still sent as `ui_instructions`, as in Go. `react_assistant`'s settings have the same Interactive UI switch as `swift_assistant`'s, which opens new conversations with `ui: false`. The shared fixtures are unchanged: they are what the model is told, the instructions and the UI joined by a blank line, as the router joins them. Every generated client is regenerated (Go, JavaScript types, Python `plugins/stream`, Swift, Rust's `types.rs`); Rust compiling and sending `ui:` is still to do, below. Other SDKs need to update: Python, .NET, Ruby, Rust and PHP, when they read `ui:`, should send it as `ui_instructions` the same way; Swift has `SessionOptions.ui`, which `swift_assistant`'s Settings switch sets, and JavaScript's `sessions.create` passes `ui` through with the regenerated types; Python, Kotlin, Dart and the other server SDKs should offer `ui` on their create-session options, so an app that shows no UI can say so.
A tool call's state (`ToolCallState`, the `tool_call.started`, `tool_call.progress` and `tool_call.finished` frames) carries `display_title`: what the call is doing, as its tool was declared with (a session's own tool's `display_title`, or a hosted tool's), or "Searching the web" for the router's own search, the same default a Chat reply's tool step uses (`conversation.ToolTitle`). A worker's `host_tools` frame takes `display_title` on each tool, beside `interruptible` and `timeout_ms`. Go has `tools.DisplayTitle(title)`, sent on `host_tools` and on a session's own tools. The generated clients are regenerated (Go, JavaScript types, Python `plugins/stream`, Ruby, PHP, .NET); Swift's generates no `ToolCallState`, so it is unchanged. JavaScript needs nothing: its `host_tools` sends `tools.declared()`, `displayTitle` included, which the router now keeps. Other SDKs need to update: Python, .NET, Ruby, Rust and PHP should send their tool's display title on `host_tools`, their registries taking one where they do not yet; Swift, Kotlin and Dart read `display_title` off the tool call state once regenerated.
5 changes: 4 additions & 1 deletion acceleration/api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11013,6 +11013,9 @@ components:
description: When a person allowed the call.
format: date-time
type: string
display_title:
description: 'What the call is doing, in words for the people in the conversation, such as "Checking the weather": the display_title its tool was declared with, or "Searching the web" for the router''s own search. Absent when the tool declared none.'
type: string
duration_ms:
description: From started_at to this change; on tool_call.finished, how long the call took.
format: int64
Expand Down Expand Up @@ -18950,7 +18953,7 @@ paths:
`connector_unavailable` names an optional connector binding the session opened without: name (its alias), connector_id and reason, one of no_selection, shared_session, caller_unverified, connection_unavailable, provider_mismatch, needs_reauthorization, credential_rejected (the provider rejected the token or key a bearer or api_key connection holds, or the connection reads a connector revision marked broken; only saving its credentials again fixes it, so no login is offered), not_connected, open_failed, tool_unavailable and selection_dropped (a fork's or a reopened chat's selection for an alias its config no longer declares). Every watcher is sent each one when it attaches.
`connector_scope_required` says a connector tool call was refused because the caller's own connection lacks access the provider asked for (insufficient_scope or a claims challenge), and a step-up consent was begun for it: name (the binding's alias), connector_id, connection_id, scopes (what the provider asked for, empty for a claims challenge), authorization_id, launch_url, handoff_token and expires_at. A page of the session's return_url's origin, or the dashboard for a session created without one, opens launch_url in a popup and posts it handoff_token, as for a createAuthorization with that return_url: the launch page takes the handoff from that origin only, and the consent goes back to the session's return_url, or else to the dashboard. The old grant keeps working until the step-up succeeds, and the same call works afterwards in the same session. While that step-up is open, calls refused for the same access send no second event.
Persistent text sessions also emit `conversation_updated` with conversation_id and a complete message snapshot: id, request_id, question_id, role, text, state, response_started_at, state_started_at, finished_at, duration_ms, saved, persistence_error, refused (true on a reply the prompt injection screen refused) and attachments. Each tool_calling attachment has tool_call_id, name, title, status, phase, summary, immutable started_at, execution_started_at, finished_at and duration_ms. A connector_authorization attachment asks the end user to connect a connector binding the reply needed with their own account, with name (the binding's alias), connector_id, connection_id, authorization_id, title, launch_url, handoff_token and expires_at: a page of the session's return_url's origin, or the dashboard for a session created without one, opens launch_url in a popup and posts it handoff_token, as for a createAuthorization with that return_url, and the consent goes back to the session's return_url, or else to the dashboard. Once the user finishes that login the message is sent again with status connected and no handoff_token, and the agent carries on by itself. The call that needed the login waits for it on its own ai_tool_call step, in parts: status awaiting_authorization, target_user_id the caller, and an authorization of type oauth with provider (the connector id), provider_name, title, message, url (the launch page), handoff_token and expires_at. A page of the return_url's origin opens url and posts it handoff_token; a native app opens url#handoff_token=<handoff_token> in its web authentication sheet. Connected, the step is running again with authorization.status connected and no handoff_token, and the call runs in the same reply. A tool_approval naming its tool_call_id with allowed false, or the person refusing at the provider, declines the login (declined); a consent that ended otherwise without connecting, such as with another account, is failed, and one not finished by expires_at is expired. Each way the step is cancelled, authorization.status says how, and the call runs nothing. Activity states are thinking, queued, tools, writing, completed, failed and cancelled. tool_started includes tool_call_id, tool, turn_id and started_at, and pre_speech when the tool's connector binding sets one in its policy; tool_ran also includes tool_call_id, and files when the tool was run_code, as on task_settled.
`tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, waiting for a person's approval, success, failed or cancelled), progress (the last `tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id.
`tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status (in_progress, waiting for a person's approval, success, failed or cancelled), display_title (what the call is doing, as its tool was declared with), progress (the last `tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id.
A respond command carrying request_id emits command_accepted with a nested command receipt (request_id, user_message_id, assistant_message_id, state, duplicate). Personal persistent text sessions require this ID. A retry with the same text and images returns the existing IDs without invoking the model again; reuse with different content emits an error. Its images are data URIs, at most 8, uploaded to the conversation's channel and shown on the person's message. On a session a dispatch worker opened for a message handed out from a channel, that message is already there: nothing is written or uploaded for it, user_message_id is empty, and a reused id is answered as a duplicate even with other text or images. After restart an interrupted command is reported, not rerun.
An `interrupt` command carrying `request_id` stops that command and emits `command_stopped` with its terminal receipt. A stop arriving after its command finished replays that command's receipt and leaves the command running now alone; an unknown command is reported as an error. Without `request_id` the frame stops whichever reply is current, which is what a caller with no command to name means by it.
A `decision` frame is one judgement the conversation made, carrying the same fields as a CallEvent. Together they are why the call went the way it did, and they are also written down, so a finished call replays them from `/v1/agents/calls/{id}/events`.
Expand Down
15 changes: 15 additions & 0 deletions acceleration/internal/api/dispatch_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,21 @@ func (s *DispatchSuite) TestHostedToolsAreKeptForTheAgentTheWorkerNames() {
s.Equal(30*time.Second, timeout, "a worker that names no timeout takes the session's default")
}

func (s *DispatchSuite) TestAHostedToolKeepsWhatItsCallsAreShownDoing() {
s.serverClient.agentNamed("stream-support")
worker := s.worker()

s.Require().NoError(worker.WriteJSON(frame{
"type": "host_tools", "agent_id": "stream-support",
"tools": []frame{{"name": "weather_lookup", "description": "Look up the weather", "display_title": " Checking the weather "}},
}))

s.Require().Equal("hosting", s.next(worker)["type"])
tools, _ := s.dispatch.HostedTools(s.customerID(), "stream-support")
s.Require().Len(tools, 1)
s.Equal("Checking the weather", tools[0].DisplayTitle)
}

func (s *DispatchSuite) TestAHostedToolsOwnTimeoutReplacesTheWorkers() {
s.serverClient.agentNamed("stream-support")
worker := s.worker()
Expand Down
8 changes: 5 additions & 3 deletions acceleration/internal/api/dispatchws.go
Original file line number Diff line number Diff line change
Expand Up @@ -380,9 +380,10 @@ type hostedTool struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters map[string]any `json:"parameters"`
// Interruptible and TimeoutMs mean what they do on a session's own tool.
Interruptible bool `json:"interruptible"`
TimeoutMs int `json:"timeout_ms"`
// DisplayTitle, Interruptible and TimeoutMs mean what they do on a session's own tool.
DisplayTitle string `json:"display_title"`
Interruptible bool `json:"interruptible"`
TimeoutMs int `json:"timeout_ms"`
// Approval is the question a voice caller is asked before each call runs, as on a
// session's own tool. Only its title is asked.
Approval *hostedApproval `json:"approval"`
Expand Down Expand Up @@ -423,6 +424,7 @@ func (s *Server) hostTools(worker *dispatch.Worker, agentID string, everyAgent b
}
hosted := dispatch.Tool{
Name: tool.Name, Description: tool.Description, Parameters: tool.Parameters,
DisplayTitle: strings.TrimSpace(tool.DisplayTitle),
Interruptible: tool.Interruptible, Timeout: time.Duration(tool.TimeoutMs) * time.Millisecond,
}
if tool.Approval != nil {
Expand Down
3 changes: 2 additions & 1 deletion acceleration/internal/api/handwritten.go
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,8 @@ func documentHandWritten(api huma.API) {
"tool_ran also includes tool_call_id, and files when the tool was run_code, as on task_settled.\n" +
"`tool_call.started`, `tool_call.progress` and `tool_call.finished` (`ToolCallState`) are a " +
"call's state as a client shows it, whoever runs the call: tool_call_id, tool, turn_id, status " +
"(in_progress, waiting for a person's approval, success, failed or cancelled), progress (the last " +
"(in_progress, waiting for a person's approval, success, failed or cancelled), display_title (what " +
"the call is doing, as its tool was declared with), progress (the last " +
"`tool_update`, the binding's pre_speech or the approval question), started_at, duration_ms, " +
"approved_at, rejected_at and error. Each carries the whole state: keep the last per tool_call_id.\n" +
"A respond command carrying request_id emits command_accepted with a nested command " +
Expand Down
Loading
Loading