diff --git a/assets/agw-docs/pages/agentgateway/llm/providers/bedrock.md b/assets/agw-docs/pages/agentgateway/llm/providers/bedrock.md index 16c3f4ebd..d18ee08de 100644 --- a/assets/agw-docs/pages/agentgateway/llm/providers/bedrock.md +++ b/assets/agw-docs/pages/agentgateway/llm/providers/bedrock.md @@ -277,6 +277,21 @@ Whether the preference changes the request format that a model accepts depends o > Requests to the Mantle endpoint are signed for the `bedrock-mantle` service rather than `bedrock`. If you scope an IAM policy by service name, grant both before you switch a route to Mantle. {{% /version %}} +{{% version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" %}} +## Tool choice + +When a chat request goes to Bedrock Runtime, the `tool_choice` value from the client request is converted to the Bedrock Converse `toolChoice` field. + +| Client `tool_choice` | Bedrock `toolChoice` | +|---|---| +| `auto` | `auto` | +| `required` (Chat Completions, Responses) or `any` (Messages) | `any` | +| A named function or tool | `tool`, with that name | +| `none`, or omitted | Omitted. Bedrock Converse has no `none` option. The tools are still sent, so the model can still call them. | + +The Bedrock model decides which values it accepts. If the model rejects forced tool use, such as `any` or a named tool, the client receives the Bedrock error. For example, Anthropic models reject forced tool use when [extended thinking](https://platform.claude.com/docs/en/build-with-claude/thinking#thinking-with-tool-use) is enabled. The tool choice is not changed for requests that use extended thinking. +{{% /version %}} + ## Prompt caching Prompt Caching is a performance, cost-optimization, and cost-reduction feature that allows the model to "remember" frequently used parts of your prompt, including long system instructions, reference documents, or tool definitions. This way, the model does not need to reprocess these parts every time you send a new prompt. diff --git a/content/docs/kubernetes/latest/release-notes/release-notes.md b/content/docs/kubernetes/latest/release-notes/release-notes.md index 904ad9dfa..63b458b31 100644 --- a/content/docs/kubernetes/latest/release-notes/release-notes.md +++ b/content/docs/kubernetes/latest/release-notes/release-notes.md @@ -185,6 +185,7 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `pre + **Traffic management** @@ -197,6 +198,7 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `pre **MCP** - Access-log CEL expressions can now read dynamic metadata that ExtMCP request-phase guardrails return through `mcpGuardrails`, including on resumed stateful MCP sessions. For more information, see [Log MCP guardrail metadata]({{< link-hextra path="/documentation/observability/access-logs/view/#mcp-guardrails" >}}). +- OpenAPI MCP tools return `image/*` responses as MCP image content instead of UTF-8-decoded text. **Security** diff --git a/content/docs/kubernetes/main/release-notes/release-notes.md b/content/docs/kubernetes/main/release-notes/release-notes.md index eca638657..9e5547e0c 100644 --- a/content/docs/kubernetes/main/release-notes/release-notes.md +++ b/content/docs/kubernetes/main/release-notes/release-notes.md @@ -35,6 +35,8 @@ Review the release notes for agentgateway on Kubernetes. + + **Traffic management** @@ -45,6 +47,7 @@ Review the release notes for agentgateway on Kubernetes. - Gemini OpenAI-compatible responses count reasoning tokens as output usage when agentgateway converts the response to Messages or Responses and when agentgateway records normalized usage. - Messages requests that ask for reasoning no longer send reasoning parameters to older OpenAI chat models whose IDs start with `gpt-3.5`, `gpt-4`, or `chatgpt-4o`. - Bedrock conversion keeps prompt-cache markers when unsupported Messages blocks or blank Chat Completions text are dropped, and Anthropic cache TTLs reach Bedrock cache points. +- Bedrock Runtime requests now send client `tool_choice` settings as Converse `toolChoice`, so forced tool choices can take effect and models that reject them return Bedrock errors. **MCP** @@ -60,6 +63,7 @@ Review the release notes for agentgateway on Kubernetes. **Operations** - HTTP/1 clients receive `Connection: close` on the next response when a proxy drains or reaches its maximum connection duration. +- Metrics requests use the OpenMetrics content type by default, while browser requests use `text/plain` so browsers display metrics instead of downloading them. ## ✨ Highlights {#v16-highlights} @@ -241,7 +245,6 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `pre - **Traffic management** @@ -261,7 +264,3 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `pre **Security** - The controller no longer fails on a JWT authentication policy that sets `jwks.remote.url` without a `backendRef` when the `AGW_BACKEND_REF_GRANT_MODE` environment variable is set to `route-and-policy`. - -**Operations** - -- Metrics requests use the OpenMetrics content type by default, while browser requests use `text/plain` so browsers display metrics instead of downloading them. diff --git a/content/docs/standalone/latest/release-notes/release-notes.md b/content/docs/standalone/latest/release-notes/release-notes.md index 15c66f8b9..8551e3c01 100644 --- a/content/docs/standalone/latest/release-notes/release-notes.md +++ b/content/docs/standalone/latest/release-notes/release-notes.md @@ -200,11 +200,15 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `fro + + + **Traffic management** - Connections with `maxConnectionDuration` close with up to 10% jitter, which reduces synchronized reconnects at the configured connection age. - Bare backend references in standalone HTTP routes, TCP routes, and MCP target backend hosts now resolve to top-level backends. A reference such as `backend: upstream` no longer fails at request time with `service not found` when the top-level backend is named `upstream`. +- Route path regexes now use full-path matching consistently, including patterns with alternation or lazy quantifiers. **LLM** @@ -213,3 +217,8 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `fro **MCP** - Access-log CEL expressions can now read dynamic metadata that ExtMCP request-phase guardrails return through `mcpGuardrails`, including on resumed stateful MCP sessions. For more information, see [Log MCP guardrail metadata]({{< link-hextra path="/documentation/observability/access-logs/view/#mcp-guardrails" >}}). +- OpenAPI MCP tools return `image/*` responses as MCP image content instead of UTF-8-decoded text. + +**Operations** + +- Standalone Helm installs can set `podDisruptionBudget.maxUnavailable` without also rendering the default `podDisruptionBudget.minAvailable` value. diff --git a/content/docs/standalone/main/documentation/llm/api-types/messages.md b/content/docs/standalone/main/documentation/llm/api-types/messages.md index 59a8e5a46..c7c0e9601 100644 --- a/content/docs/standalone/main/documentation/llm/api-types/messages.md +++ b/content/docs/standalone/main/documentation/llm/api-types/messages.md @@ -154,6 +154,8 @@ In both conversions, these prefixes are matched against the model name that is s The Bedrock Converse conversion carries extended-thinking history in both directions, including encrypted reasoning. +The Bedrock Converse conversion preserves tool choice for Messages requests. The `auto` value becomes the Bedrock `toolChoice.auto` value, and `any` becomes `toolChoice.any`. A `tool` choice becomes `toolChoice.tool`. The `none` value or an omitted tool choice is omitted, and the tools are still sent, so the model can still call them. Extended thinking does not rewrite the choice. A model that rejects forced tool use with thinking returns a provider error. + On the way out, a `thinking` block in the message history becomes a Bedrock `reasoningContent.reasoningText` block, with its signature. A `redacted_thinking` block becomes a Bedrock `reasoningContent.redactedContent` block, so that a client can replay encrypted reasoning that Bedrock returned on an earlier turn. `document`, `search_result`, `server_tool_use`, and `web_search_tool_result` blocks are dropped because Bedrock Converse has no equivalent block types. Each `cache_control` marker becomes a Bedrock cache point after the content that the marker is on, and the `cache_control.ttl` value becomes the TTL of the cache point. The conversion drops some content before sending the request to Bedrock. When dropped content has a cache marker, the marker moves to nearby content that the conversion keeps: diff --git a/content/docs/standalone/main/integrations/llm/providers/bedrock.md b/content/docs/standalone/main/integrations/llm/providers/bedrock.md index dce453605..7f69b5407 100644 --- a/content/docs/standalone/main/integrations/llm/providers/bedrock.md +++ b/content/docs/standalone/main/integrations/llm/providers/bedrock.md @@ -270,6 +270,19 @@ Whether the preference changes the request format that a model accepts depends o > [!NOTE] > Requests to the Mantle endpoint are signed for the `bedrock-mantle` service rather than `bedrock`. If you scope an IAM policy by service name, grant both before you switch a route to Mantle. +## Tool choice + +When a chat request goes to Bedrock Runtime, the `tool_choice` value from the client request is converted to the Bedrock Converse `toolChoice` field. + +| Client `tool_choice` | Bedrock `toolChoice` | +|---|---| +| `auto` | `auto` | +| `required` (Chat Completions, Responses) or `any` (Messages) | `any` | +| A named function or tool | `tool`, with that name | +| `none`, or omitted | Omitted. Bedrock Converse has no `none` option. The tools are still sent, so the model can still call them. | + +The Bedrock model decides which values it accepts. If the model rejects forced tool use, such as `any` or a named tool, the client receives the Bedrock error. For example, Anthropic models reject forced tool use when [extended thinking](https://platform.claude.com/docs/en/build-with-claude/thinking#thinking-with-tool-use) is enabled. The tool choice is not changed for requests that use extended thinking. + ## Token counting Bedrock supports token counting for Anthropic models via the `count_tokens` endpoint. diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 1e14a2f74..4645e06f4 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -30,6 +30,8 @@ Review the release notes for agentgateway standalone. + + **Traffic management** @@ -40,6 +42,7 @@ Review the release notes for agentgateway standalone. - Gemini OpenAI-compatible responses count reasoning tokens as output usage when agentgateway converts the response to Messages or Responses and when agentgateway records normalized usage. - Messages requests that ask for reasoning no longer send reasoning parameters to older OpenAI chat models whose IDs start with `gpt-3.5`, `gpt-4`, or `chatgpt-4o`. - Bedrock conversion keeps prompt-cache markers when unsupported Messages blocks or blank Chat Completions text are dropped, and Anthropic cache TTLs reach Bedrock cache points. +- Bedrock Runtime requests now send client `tool_choice` settings as Converse `toolChoice`, so forced tool choices can take effect and models that reject them return Bedrock errors. **MCP** @@ -51,6 +54,7 @@ Review the release notes for agentgateway standalone. **Operations** - Standalone configuration reloads now track files that agentgateway reads during parsing, such as `backendTLS.root` certificate files, so later file changes are picked up without a process restart. +- Metrics requests use the OpenMetrics content type by default, while browser requests use `text/plain` so browsers display metrics instead of downloading them. ## ✨ Highlights {#v16-highlights} @@ -249,7 +253,6 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `fro - **Traffic management** @@ -271,4 +274,3 @@ You can now opt in to OpenTelemetry field names for stdout access logs. Set `fro **Operations** - Standalone Helm installs can set `podDisruptionBudget.maxUnavailable` without also rendering the default `podDisruptionBudget.minAvailable` value. -- Metrics requests use the OpenMetrics content type by default, while browser requests use `text/plain` so browsers display metrics instead of downloading them.