diff --git a/content/docs/kubernetes/main/documentation/llm/models/virtual.md b/content/docs/kubernetes/main/documentation/llm/models/virtual.md index 0295005bc..c042e505b 100644 --- a/content/docs/kubernetes/main/documentation/llm/models/virtual.md +++ b/content/docs/kubernetes/main/documentation/llm/models/virtual.md @@ -361,7 +361,7 @@ Use `virtualModel.conditional` to select a target with a CEL expression. Targets Use `virtualModel.failover` to group targets by priority. Lower values are preferred. Targets in the same priority group are selected by a score that considers health and latency. The next group is used only when every target in the current group is degraded. -Failover depends on eviction. A virtual model with more than one priority group uses default eviction: a single 5xx response or connection failure evicts a target for 3 seconds, and each repeated eviction lasts longer. To change when a target is evicted, configure `policies.health` on the concrete target models. A health policy replaces the default eviction instead of adding to it, so include an `eviction` block, as in the following example. +Failover depends on eviction. A virtual model with more than one priority group uses default eviction: a single 5xx response, server-side gRPC failure, or connection failure evicts a target for 3 seconds, and each repeated eviction lasts longer. To change when a target is evicted, configure `policies.health` on the concrete target models. A health policy replaces the default eviction instead of adding to it, so include an `eviction` block, as in the following example. 1. Create a model that points at an address with no backing workload, so that requests to it always fail. In a real deployment, the target would be a healthy primary provider. diff --git a/content/docs/kubernetes/main/release-notes/release-notes.md b/content/docs/kubernetes/main/release-notes/release-notes.md index 9e5547e0c..774552da2 100644 --- a/content/docs/kubernetes/main/release-notes/release-notes.md +++ b/content/docs/kubernetes/main/release-notes/release-notes.md @@ -37,6 +37,7 @@ Review the release notes for agentgateway on Kubernetes. + **Traffic management** @@ -48,6 +49,7 @@ Review the release notes for agentgateway on Kubernetes. - Messages requests that ask for reasoning no longer send reasoning parameters to older OpenAI chat models whose IDs start with `gpt-3.5`, `gpt-4`, or `chatgpt-4o`. - Bedrock conversion keeps prompt-cache markers when unsupported Messages blocks or blank Chat Completions text are dropped, and Anthropic cache TTLs reach Bedrock cache points. - Bedrock Runtime requests now send client `tool_choice` settings as Converse `toolChoice`, so forced tool choices can take effect and models that reject them return Bedrock errors. +- A client-side gRPC status from an LLM provider no longer evicts the target from a failover group. **MCP** diff --git a/content/docs/standalone/main/documentation/llm/virtual-models.md b/content/docs/standalone/main/documentation/llm/virtual-models.md index 2a86b3a71..9d3c334ea 100644 --- a/content/docs/standalone/main/documentation/llm/virtual-models.md +++ b/content/docs/standalone/main/documentation/llm/virtual-models.md @@ -173,7 +173,7 @@ Configure health on the concrete `llm.models[]` entries that the virtual model t | Setting | What it does | | -- | -- | -| No `health` policy | The default unhealthy classifier covers `5xx` responses, non-zero gRPC statuses, and connection failures. When the virtual model has more than one priority group, these failures use default eviction: a single unhealthy response evicts the endpoint for `3s`, and each repeated eviction lasts longer. Traffic fails over to the next priority. | +| No `health` policy | The default unhealthy classifier covers `5xx` responses, server-side gRPC failures, and connection failures. When the virtual model has more than one priority group, these failures use default eviction: a single unhealthy response evicts the endpoint for `3s`, and each repeated eviction lasts longer. Traffic fails over to the next priority. | | `health` without `eviction` | Setting `health` replaces the default eviction instead of adding to it. You can use `health.unhealthyExpression` to classify additional responses, such as `429`, as unhealthy. Without an `eviction` block, the endpoint is evicted only when a retry policy's `backoff`, or a `Retry-After` header on a response that is classified as unhealthy, supplies an eviction duration. To keep the default eviction settings, add `eviction: {}`. | | `health.eviction` | Override how long an unhealthy endpoint leaves the active set, and which thresholds trigger eviction. When every endpoint in a priority group is evicted, later requests use the next priority. | @@ -181,7 +181,7 @@ You do not need a `health` policy for basic failover on server errors or connect Useful `health` fields: -- `unhealthyExpression`: Optional CEL expression; `true` marks the response unhealthy. When unset, any `5xx`, non-zero gRPC status, or connection failure is unhealthy. +- `unhealthyExpression`: Optional CEL expression; `true` marks the response unhealthy. When unset, any `5xx`, server-side gRPC failure, or connection failure is unhealthy. - `eviction.duration`: Base time to keep an endpoint evicted. When you omit `duration`, agentgateway uses the `Retry-After` value from a 429 response, then the `backoff` from a retry policy, and then a default of `3s`. Repeated evictions use multiplicative backoff, with no upper bound. - `eviction.consecutiveFailures`: Unhealthy responses required before eviction. When this and `healthThreshold` are both unset, a single unhealthy response can evict. - `eviction.healthThreshold`: Evict when the endpoint health score (0.0–1.0) is below this value. Either this or `consecutiveFailures` can trigger eviction when both are set. diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 4645e06f4..0259eaeae 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -32,6 +32,7 @@ Review the release notes for agentgateway standalone. + **Traffic management** @@ -43,6 +44,7 @@ Review the release notes for agentgateway standalone. - Messages requests that ask for reasoning no longer send reasoning parameters to older OpenAI chat models whose IDs start with `gpt-3.5`, `gpt-4`, or `chatgpt-4o`. - Bedrock conversion keeps prompt-cache markers when unsupported Messages blocks or blank Chat Completions text are dropped, and Anthropic cache TTLs reach Bedrock cache points. - Bedrock Runtime requests now send client `tool_choice` settings as Converse `toolChoice`, so forced tool choices can take effect and models that reject them return Bedrock errors. +- A client-side gRPC status from an LLM provider no longer evicts the target from a failover group. **MCP**