From d53f8d9474e1562a49d259ff4a790c74d48f7a57 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Tue, 13 Oct 2026 14:14:09 -0400 Subject: [PATCH 1/6] Added callout proxy Signed-off-by: Kristin Brown --- .../configuration/callout-proxy.md | 138 ++++++++++++++++++ .../configuration/egress-proxy.md | 2 + .../main/release-notes/release-notes.md | 4 + 3 files changed, 144 insertions(+) create mode 100644 content/docs/standalone/main/documentation/configuration/callout-proxy.md diff --git a/content/docs/standalone/main/documentation/configuration/callout-proxy.md b/content/docs/standalone/main/documentation/configuration/callout-proxy.md new file mode 100644 index 000000000..cf5b5abfb --- /dev/null +++ b/content/docs/standalone/main/documentation/configuration/callout-proxy.md @@ -0,0 +1,138 @@ +--- +title: Callout proxy +weight: 38 +description: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches and external authorization calls, through an HTTP proxy. +--- + +In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that agentgateway makes on its own behalf through that proxy. These requests are called callouts, and they go through the proxy in an HTTP CONNECT tunnel. + +## About callouts + +A callout is any request that agentgateway sends for itself rather than for a client. Callouts include the following requests. + +* Remote JSON Web Key Set (JWKS) fetches for JSON Web Token (JWT) authentication. +* OpenID Connect (OIDC) discovery. +* [External authorization]({{< link-hextra path="/documentation/configuration/security/external-authz" >}}) requests. +* Cloud provider credential fetches, such as AWS and Azure credentials. + +Client traffic to your primary backends is not a callout, so the callout proxy does not apply to it. To send backend traffic through a proxy, set the `backendTunnel` policy on the backend instead. To proxy CONNECT requests from your clients instead, see [Egress proxy]({{< link-hextra path="/documentation/configuration/egress-proxy/" >}}). + +## Configure the proxy + +You can configure the callout proxy in either of the following ways. + +* **In the configuration file**: Set `config.callouts.tunnel`. Settings in the `config` section take effect at startup, so restart agentgateway after you change this setting. For more information, see [Static configuration]({{< link-hextra path="/documentation/configuration/static-configuration/" >}}). +* **With environment variables**: Omit `config.callouts.tunnel` and set the standard proxy environment variables in the environment that runs agentgateway. When you set `config.callouts.tunnel`, the proxy environment variables have no effect on callouts. + +### Configuration file + +```yaml +config: + callouts: + tunnel: + url: http://proxy.example.com:3128 + noProxy: + - internal.example.com + - 10.0.0.0/8 +``` + +| Setting | Description | +| -- | -- | +| `config.callouts.tunnel.url` | The URL of the proxy. This proxy handles every callout, whether the callout uses Transport Layer Security (TLS) or plaintext. The scheme must be `http` or `https`. With `https`, the connection to the proxy uses TLS, and the proxy certificate is verified against the system trust bundle. If you omit the port, the default is `80` for `http` and `443` for `https`. To authenticate to the proxy with basic authentication, include the credentials in the URL, such as `http://user:password@proxy.example.com:3128`. The `--validate-only` flag does not check the scheme. A URL with an unsupported scheme, such as `socks5://`, passes validation, and then agentgateway fails to start with an `invalid callouts config` error. | +| `config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the proxy. Entries use the `NO_PROXY` syntax: a hostname, which also matches its subdomains, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` to bypass the proxy for every destination. | + +### Environment variables + +If you omit `config.callouts.tunnel`, agentgateway reads the following environment variables at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. + +| Variable | Description | +| -- | -- | +| `HTTPS_PROXY` | The proxy for callouts that use TLS. | +| `HTTP_PROXY` | The proxy for plaintext callouts. | +| `ALL_PROXY` | The proxy for callouts whose scheme-specific variable is unset. | +| `NO_PROXY` | A comma-separated list of destinations that bypass the proxy, in the same syntax as `noProxy`. | + +> [!IMPORTANT] +> The proxy environment variables apply to callouts even when you set the variables for another purpose. If the environment that runs agentgateway exports `HTTPS_PROXY` for other tools, callouts also go through that proxy. To keep callouts direct, unset the variables for the agentgateway process, or set `NO_PROXY` to `*`. + +## Destinations that bypass the proxy + +The following callouts go directly to their destination, even when you configure a proxy. + +* Destinations that match `noProxy` or `NO_PROXY`. +* Loopback and link-local addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote proxy cannot reach these addresses. +* Callouts to a backend that has its own `backendTunnel` policy. These callouts use that tunnel instead. +* Callouts to a service from the top-level `services` list, and to Unix domain sockets. +* Google Cloud Platform (GCP) credential fetches, when you configure the proxy with `config.callouts.tunnel`. To send these fetches through a proxy, use the environment variables instead. + +## Before you begin + +{{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} + +You also need an HTTP proxy that supports the CONNECT method and logs the requests that it receives, such as Squid. The following steps assume that the proxy listens on `http://127.0.0.1:3128`. + +## Send a JWKS fetch through the proxy + +Configure JWT authentication with a remote JWKS, and confirm in the proxy log that the JWKS fetch goes through the proxy. A JWKS fetch is a good test because agentgateway makes it at startup, before any client traffic arrives. + +1. Create a configuration file that fetches a remote JWKS and sends callouts through the proxy. + ```yaml + cat < callout-proxy.yaml + # yaml-language-server: \$schema=https://agentgateway.dev/schema/config + config: + callouts: + tunnel: + url: http://127.0.0.1:3128 + binds: + - port: 3000 + listeners: + - routes: + - policies: + jwtAuth: + mode: strict + issuer: https://accounts.google.com + jwks: + url: https://www.googleapis.com/oauth2/v3/certs + backends: + - host: httpbin.org:80 + EOF + ``` + + {{% reuse "agw-docs/snippets/review-table.md" %}} + + | Setting | Description | + | -- | -- | + | `config.callouts.tunnel.url` | The address of the proxy. The loopback bypass applies to callout destinations, not to the proxy. A proxy on `127.0.0.1` still receives the callouts. | + | `jwtAuth.jwks.url` | The remote JWKS. The JWKS fetch is a callout and goes through the proxy. | + | `backends[].host` | The primary backend. Client requests to this backend are not callouts and do not use the proxy. | + +2. Validate the configuration. + ```sh + agentgateway --validate-only -f callout-proxy.yaml + ``` + + Example output: + ```console + Configuration is valid! + ``` + +3. Start agentgateway. The JWKS fetch happens at startup. If the proxy is unreachable, the fetch fails and agentgateway exits with a `failed to load JWKS` error. + ```sh + agentgateway -f callout-proxy.yaml + ``` + + The startup logs include a line that names the callout proxy. + ```console + info client::callout callouts will use proxy Matcher { http: Intercept { uri: http://127.0.0.1:3128/ }, https: Intercept { uri: http://127.0.0.1:3128/ } } + ``` + +4. Check the access log of your proxy for a CONNECT request to `www.googleapis.com:443`, the JWKS host. The log format depends on your proxy. + +5. Optional: To use environment variables instead, remove the `config.callouts` section from the configuration file. Then, start agentgateway with the `HTTPS_PROXY` variable set. The proxy log shows the same CONNECT request. + ```sh + HTTPS_PROXY=http://127.0.0.1:3128 agentgateway -f callout-proxy.yaml + ``` + +## Configuration reference + +For every field in the `config.callouts` section, see the [configuration schema]({{< link-hextra path="/reference/configuration/schema/" >}}). diff --git a/content/docs/standalone/main/documentation/configuration/egress-proxy.md b/content/docs/standalone/main/documentation/configuration/egress-proxy.md index c00476541..abfc49338 100644 --- a/content/docs/standalone/main/documentation/configuration/egress-proxy.md +++ b/content/docs/standalone/main/documentation/configuration/egress-proxy.md @@ -27,6 +27,8 @@ The client then sends `CONNECT :443` to agentgateway instead of connecting Neither mode is a replacement for the other. Tunnel mode gives you SNI-based control and the option to terminate TLS, and route mode gives you the HTTP policy engine, including client authentication. +To send the requests that agentgateway makes on its own behalf, such as JWKS fetches, through an upstream proxy instead of proxying traffic from your clients, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). + > [!NOTE] > Policies that you attach to a listener on a `tunnelProtocol: connect` bind are not applied to the CONNECT request, because tunnel mode handles the request before route matching. To authenticate clients, use route mode. diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 8a5dca869..a6ffdd1b2 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -78,6 +78,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo ## 🔄 Other behavior changes {#v16-behavior-changes} + @@ -99,6 +100,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo - **Token usage in responses**: The `usage` that agentgateway returns to clients now follows the conventions of the API that the client calls, whichever provider serves the request. Anthropic Messages responses leave cache tokens out of `input_tokens` and report them in `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI Chat Completions and Responses include cache tokens in the input and total token counts. For example, a Chat Completions request to an Anthropic or Amazon Bedrock model that writes to the prompt cache now reports those cache tokens in `prompt_tokens`. If a client calculates usage or cost from these fields, review its counts after you upgrade. - **Policy service timeouts**: Calls to external services default to a timeout of 2 seconds for gRPC external authorization and 10 seconds for rate limit and external processing services. - **Streaming guardrails**: Agentgateway now rejects content if a provider guard fails while checking a streamed response or realtime connection. To keep the 1.5 behavior, set `failureMode: failOpen` on the guard. For example, set it in `ai.promptGuard.response[].bedrockGuardrails`. For more information, see [Provider failures]({{< link-hextra path="/documentation/llm/prompt-guards/overview/#provider-failures" >}}). +- **Proxy environment variables**: Callouts, such as JWKS fetches and external authorization requests, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables when `config.callouts.tunnel` is omitted. If the environment that runs agentgateway exports these variables for other tools, callouts go through that proxy after you upgrade. To keep callouts direct, set `NO_PROXY` to `*`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). ## 🌟 New features {#v16-new-features} @@ -173,6 +175,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo ### Traffic management and operations {#v16-operations} + @@ -187,6 +190,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo - **Graceful drain**: During shutdown, agentgateway stops accepting new connections after `config.connectionMinTerminationDeadline`. It continues to serve open connections until `config.connectionTerminationDeadline`. For more information, see [Shutdown and drain]({{< link-hextra path="/documentation/configuration/static-configuration/#shutdown-drain" >}}). - **Helm chart scaling**: The standalone chart can create a PodDisruptionBudget and a HorizontalPodAutoscaler. For more information, see [Create a PodDisruptionBudget]({{< link-hextra path="/documentation/setup/install/helm/#helm-pdb" >}}). - **Telemetry**: OTLP access logs use the `agentgateway.access` instrumentation scope. The request duration metric records failed requests with an `error_type` label. For the metrics, see the [metrics reference]({{< link-hextra path="/documentation/observability/metrics/reference/#llm" >}}). +- **Callout proxy**: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches, OIDC discovery, and cloud provider credential fetches, through an HTTP proxy. Set `config.callouts.tunnel`, or use the standard proxy environment variables. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). ### OpenTelemetry field names {#v16-otel-attributes} From baca44f06caf85930e7ed1280a35a7b6cc3b6d12 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Tue, 13 Oct 2026 14:25:00 -0400 Subject: [PATCH 2/6] Answered questions Signed-off-by: Kristin Brown --- .../main/documentation/configuration/callout-proxy.md | 10 ++++++---- .../standalone/main/release-notes/release-notes.md | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/content/docs/standalone/main/documentation/configuration/callout-proxy.md b/content/docs/standalone/main/documentation/configuration/callout-proxy.md index cf5b5abfb..e447ea1af 100644 --- a/content/docs/standalone/main/documentation/configuration/callout-proxy.md +++ b/content/docs/standalone/main/documentation/configuration/callout-proxy.md @@ -4,15 +4,17 @@ weight: 38 description: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches and external authorization calls, through an HTTP proxy. --- -In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that agentgateway makes on its own behalf through that proxy. These requests are called callouts, and they go through the proxy in an HTTP CONNECT tunnel. +In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that agentgateway makes on its own behalf through that proxy. These requests are called callouts. Callouts that use TLS go through the proxy in an HTTP CONNECT tunnel, and plaintext callouts go to the proxy as standard forward-proxy requests. ## About callouts A callout is any request that agentgateway sends for itself rather than for a client. Callouts include the following requests. * Remote JSON Web Key Set (JWKS) fetches for JSON Web Token (JWT) authentication. -* OpenID Connect (OIDC) discovery. -* [External authorization]({{< link-hextra path="/documentation/configuration/security/external-authz" >}}) requests. +* OpenID Connect (OIDC) discovery, including for MCP authentication. +* [External authorization]({{< link-hextra path="/documentation/configuration/security/external-authz" >}}), external processing, and remote rate limit requests. +* Guardrail requests, such as webhooks, OpenAI moderation, AWS Bedrock Guardrails, Google Model Armor, and Azure AI Content Safety. +* Virtual model callout routing requests. * Cloud provider credential fetches, such as AWS and Azure credentials. Client traffic to your primary backends is not a callout, so the callout proxy does not apply to it. To send backend traffic through a proxy, set the `backendTunnel` policy on the backend instead. To proxy CONNECT requests from your clients instead, see [Egress proxy]({{< link-hextra path="/documentation/configuration/egress-proxy/" >}}). @@ -60,7 +62,7 @@ If you omit `config.callouts.tunnel`, agentgateway reads the following environme The following callouts go directly to their destination, even when you configure a proxy. * Destinations that match `noProxy` or `NO_PROXY`. -* Loopback and link-local addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote proxy cannot reach these addresses. +* Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote proxy cannot reach these addresses. This rule applies to IP addresses only. A callout to the hostname `localhost` goes through the proxy, so add `localhost` to `noProxy` or `NO_PROXY` if your callout targets use it. * Callouts to a backend that has its own `backendTunnel` policy. These callouts use that tunnel instead. * Callouts to a service from the top-level `services` list, and to Unix domain sockets. * Google Cloud Platform (GCP) credential fetches, when you configure the proxy with `config.callouts.tunnel`. To send these fetches through a proxy, use the environment variables instead. diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index a6ffdd1b2..4b11cfee3 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -100,7 +100,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo - **Token usage in responses**: The `usage` that agentgateway returns to clients now follows the conventions of the API that the client calls, whichever provider serves the request. Anthropic Messages responses leave cache tokens out of `input_tokens` and report them in `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI Chat Completions and Responses include cache tokens in the input and total token counts. For example, a Chat Completions request to an Anthropic or Amazon Bedrock model that writes to the prompt cache now reports those cache tokens in `prompt_tokens`. If a client calculates usage or cost from these fields, review its counts after you upgrade. - **Policy service timeouts**: Calls to external services default to a timeout of 2 seconds for gRPC external authorization and 10 seconds for rate limit and external processing services. - **Streaming guardrails**: Agentgateway now rejects content if a provider guard fails while checking a streamed response or realtime connection. To keep the 1.5 behavior, set `failureMode: failOpen` on the guard. For example, set it in `ai.promptGuard.response[].bedrockGuardrails`. For more information, see [Provider failures]({{< link-hextra path="/documentation/llm/prompt-guards/overview/#provider-failures" >}}). -- **Proxy environment variables**: Callouts, such as JWKS fetches and external authorization requests, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables when `config.callouts.tunnel` is omitted. If the environment that runs agentgateway exports these variables for other tools, callouts go through that proxy after you upgrade. To keep callouts direct, set `NO_PROXY` to `*`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). +- **Proxy environment variables**: Callouts, such as JWKS fetches, external authorization requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables when `config.callouts.tunnel` is omitted. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. If the environment that runs agentgateway exports these variables for other tools, callouts go through that proxy after you upgrade. To keep callouts direct, set `NO_PROXY` to `*`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). ## 🌟 New features {#v16-new-features} From bd084ccb91ccc8a12df201937a4c3a4b986a849d Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Tue, 13 Oct 2026 14:31:39 -0400 Subject: [PATCH 3/6] Release notes updates Signed-off-by: Kristin Brown --- content/docs/kubernetes/main/release-notes/release-notes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/docs/kubernetes/main/release-notes/release-notes.md b/content/docs/kubernetes/main/release-notes/release-notes.md index 2ec6cd80c..b29686c63 100644 --- a/content/docs/kubernetes/main/release-notes/release-notes.md +++ b/content/docs/kubernetes/main/release-notes/release-notes.md @@ -64,6 +64,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ ## 🔄 Other behavior changes {#v16-behavior-changes} + @@ -89,6 +90,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ - **Streaming guardrails**: Agentgateway now rejects content if a provider guard fails while checking a streamed response or realtime connection. To keep the 1.5 behavior, set `failureMode: FailOpen` on the guard. For example, set it in `spec.backend.ai.promptGuard.response[].bedrockGuardrails` of an {{< reuse "agw-docs/snippets/policy.md" >}}. For more information, see [Provider failures]({{< link-hextra path="/documentation/llm/guardrails/overview/#provider-failures" >}}). - **Token usage in responses**: The `usage` that agentgateway returns to clients now follows the conventions of the API that the client calls, whichever provider serves the request. Anthropic Messages responses leave cache tokens out of `input_tokens` and report them in `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI Chat Completions and Responses include cache tokens in the input and total token counts. For example, a Chat Completions request to an Anthropic or Amazon Bedrock model that writes to the prompt cache now reports those cache tokens in `prompt_tokens`. If a client calculates usage or cost from these fields, review its counts after you upgrade. - **Deployer ownership**: The controller no longer overwrites an existing resource of the same name that it does not own. It reports an error instead. +- **Proxy environment variables**: Callouts, such as JWKS fetches, external authorization requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables. If you set these variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource for another purpose, callouts go through that proxy after you upgrade. Callouts to a Kubernetes Service go directly to the Service. To keep all callouts direct, set `NO_PROXY` to `*`. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. ## 🌟 New features {#v16-new-features} From ee2c95b1f0808e6cbca4be47ad0f233db5f73c2e Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Wed, 14 Oct 2026 09:53:01 -0400 Subject: [PATCH 4/6] Added callout-proxy for kube Signed-off-by: Kristin Brown --- .../llm/providers/backend-tunnel-proxy.md | 3 + .../setup/customize/callout-proxy.md | 270 ++++++++++++++++++ .../main/release-notes/release-notes.md | 4 +- .../configuration/callout-proxy.md | 24 +- 4 files changed, 289 insertions(+), 12 deletions(-) create mode 100644 content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md diff --git a/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md b/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md index ad431757d..30257342b 100644 --- a/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md +++ b/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md @@ -5,6 +5,9 @@ Use the {{< reuse "agw-docs/snippets/backend.md" >}} `tunnel` policy to route ba The `tunnel` field on an {{< reuse "agw-docs/snippets/backend.md" >}} configures an HTTP CONNECT tunnel to the target backend. Agentgateway establishes an HTTP tunnel through the proxy and forwards all requests to the upstream endpoint. The tunnel supports `Service` and {{< reuse "agw-docs/snippets/backend.md" >}} types as the proxy target. +{{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}} +The `tunnel` policy applies to one backend. To send every request that the agentgateway proxy makes on its own behalf through a forward proxy, such as external authorization and guardrail requests, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). For callouts to a backend that has its own `tunnel` policy, that tunnel takes precedence over the callout proxy. +{{< /version >}} ## Before you begin diff --git a/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md new file mode 100644 index 000000000..fd16acad5 --- /dev/null +++ b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md @@ -0,0 +1,270 @@ +--- +title: Callout proxy +weight: 40 +description: Send the requests that the agentgateway proxy makes on its own behalf, such as external authorization calls and cloud credential fetches, through a forward proxy. +--- + +In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that the agentgateway proxy makes on its own behalf through that forward proxy. These requests are called callouts. Callouts that use TLS go through the forward proxy in an HTTP CONNECT tunnel, and plaintext callouts go to the forward proxy as standard forward-proxy requests. + +## About callouts + +A callout is any request that the agentgateway proxy sends for itself rather than for a client. Callouts include the following requests. + +* External authorization, external processing, and remote rate limit requests. +* Guardrail requests, such as webhooks, OpenAI moderation, AWS Bedrock Guardrails, Google Model Armor, and Azure AI Content Safety. +* OpenID Connect (OIDC) requests from the agentgateway proxy. +* Virtual model callout routing requests. +* Credential fetches from cloud providers, such as AWS, Azure, and Google Cloud Platform (GCP). + +The callout proxy does not apply to the following traffic. + +* **Client traffic to your backends**: This traffic is not a callout. To send the traffic to one backend through a forward proxy, set the `tunnel` policy on that backend instead. For more information, see [Backend tunnel proxy]({{< link-hextra path="/integrations/llm/providers/backend-tunnel-proxy/" >}}). +* **Remote JSON Web Key Set (JWKS) fetches**: In Kubernetes, the control plane fetches remote JWKS, not the agentgateway proxy, so the callout proxy settings have no effect on these fetches. To send a JWKS fetch through a forward proxy, set the `tunnel` policy on the {{< reuse "agw-docs/snippets/backend.md" >}} that serves the JWKS. + +## Configure the callout proxy + +You configure the callout proxy on the agentgateway proxy deployment through an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, in either of the following ways. If you use both, `config.callouts.tunnel` takes precedence, and the agentgateway proxy ignores the proxy environment variables for callouts. + +* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`, in `spec.env`. Use this configuration for most setups. Note that this configuration is also the only way to send GCP credential fetches through the forward proxy. +* **[In `rawConfig`](#rawconfig)**: Set `config.callouts.tunnel` in `spec.rawConfig`. Use this way to keep the forward proxy settings separate from environment variables that other tools in the pod read. The control plane reads `rawConfig` only from an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource that a Gateway references. + +### Option 1: Environment variables + +Set the following variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. The agentgateway proxy reads them at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. + +| Variable | Description | +| -- | -- | +| `HTTPS_PROXY` | The forward proxy for callouts that use TLS. | +| `HTTP_PROXY` | The forward proxy for plaintext callouts. | +| `ALL_PROXY` | The forward proxy for callouts when you omit `HTTPS_PROXY` or `HTTP_PROXY`. | +| `NO_PROXY` | A comma-separated list of destinations that bypass the forward proxy. Each entry is a hostname, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` for every destination. A hostname also matches its subdomains. | + +> [!IMPORTANT] +> The proxy environment variables apply to callouts even when you set the variables for another purpose. If `spec.env` already sets `HTTPS_PROXY` for other tools, callouts also go through that forward proxy. To keep callouts direct, remove the variables from `spec.env`, or set `NO_PROXY` to `*`. + +### Option 2: rawConfig + +Set `config.callouts.tunnel` under `spec.rawConfig` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. + +```yaml +apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}} +kind: {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} +metadata: + name: callout-proxy + namespace: {{< reuse "agw-docs/snippets/namespace.md" >}} +spec: + rawConfig: + config: + callouts: + tunnel: + url: http://proxy.example.com:3128 + noProxy: + - internal.example.com + - 10.0.0.0/8 +``` + +| Setting | Description | +| -- | -- | +| `spec.rawConfig.config.callouts.tunnel.url` | The URL of the forward proxy. This forward proxy handles every callout, whether the callout uses TLS or plaintext. The scheme must be `http` or `https`. With `https`, the connection to the forward proxy uses TLS, and the forward proxy certificate is verified against the system trust bundle. If you omit the port, the default is `80` for `http` and `443` for `https`. To authenticate to the forward proxy with basic authentication, include the credentials in the URL, such as `http://user:password@proxy.example.com:3128`. | +| `spec.rawConfig.config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the forward proxy, in the same syntax as `NO_PROXY`. | + +## Destinations that bypass the forward proxy + +The following callouts go directly to their destination, even when you configure a callout proxy. + +* Callouts to a Kubernetes Service that you reference with a `Service` backendRef, such as an external authorization server in the cluster. An {{< reuse "agw-docs/snippets/backend.md" >}} with a `static` host is not a Service reference, even when the host is a cluster hostname such as `ext-authz.auth.svc.cluster.local`. Callouts to that backend go through the forward proxy, unless you add `svc.cluster.local` to `NO_PROXY` or `noProxy`. +* Destinations that match `noProxy` or `NO_PROXY`. +* Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote forward proxy cannot reach these addresses. +* Callouts to an {{< reuse "agw-docs/snippets/backend.md" >}} that has its own `tunnel` policy. These callouts use that tunnel instead. +* GCP credential fetches, when you configure the callout proxy with `config.callouts.tunnel`. To send these fetches through a forward proxy, use the environment variables instead. + +## Before you begin + +{{< reuse "agw-docs/snippets/prereq.md" >}} + +## Send an external authorization callout through the forward proxy + +Deploy a test forward proxy, point the agentgateway proxy at it, and confirm that an external authorization callout goes through the forward proxy. The example uses `httpbin.org` as a stand-in authorization server, because the server returns any status code that you ask for. + +1. Deploy Squid as a test forward proxy. If you already have a forward proxy that the cluster can reach, skip this step and use its URL in the next step. + ```yaml + kubectl apply -f- <}} resource that points callouts at the forward proxy. The external authorization callout in this guide uses TLS, so set `HTTPS_PROXY`. + ```yaml + kubectl apply -f- <}} + kind: {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} + metadata: + name: callout-proxy + namespace: {{< reuse "agw-docs/snippets/namespace.md" >}} + spec: + env: + - name: HTTPS_PROXY + value: http://squid.squid.svc.cluster.local:3128 + EOF + ``` + + To use `rawConfig` instead, replace `spec.env` with `spec.rawConfig.config.callouts.tunnel.url`, as shown in [rawConfig](#rawconfig). + +3. Attach the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource to your Gateway with `infrastructure.parametersRef`. The patch keeps the rest of your Gateway as it is. If your Gateway already references an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, add the `spec.env` setting to that resource instead, because a Gateway references one resource only. + ```sh + kubectl patch gateway agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} --type merge \ + -p '{"spec":{"infrastructure":{"parametersRef":{"group":"{{< reuse "agw-docs/snippets/group.md" >}}","kind":"{{< reuse "agw-docs/snippets/gatewayparameters.md" >}}","name":"callout-proxy"}}}}' + ``` + +4. Wait for the agentgateway proxy to roll out with the new settings. + ```sh + kubectl rollout status deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + ``` + +5. Check the agentgateway proxy logs for the line that names the forward proxy. The label selector reads every agentgateway proxy pod, so the command also works while an old pod is still shutting down. + ```sh + kubectl logs -n {{< reuse "agw-docs/snippets/namespace.md" >}} -l gateway.networking.k8s.io/gateway-name=agentgateway-proxy --tail=-1 | grep "callouts will use proxy" + ``` + + Example output: + ```console + info client::callout callouts will use proxy Matcher { https: Intercept { uri: http://squid.squid.svc.cluster.local:3128/ } } + ``` + +6. Create an {{< reuse "agw-docs/snippets/backend.md" >}} for the stand-in authorization server. Because the backend has a `static` host, callouts to it go through the forward proxy. + ```yaml + kubectl apply -f- <}} + kind: {{< reuse "agw-docs/snippets/backend.md" >}} + metadata: + name: httpbin-org + namespace: httpbin + spec: + static: + host: httpbin.org + port: 443 + policies: + tls: + sni: httpbin.org + EOF + ``` + +7. Create an {{< reuse "agw-docs/snippets/policy.md" >}} that sends an external authorization request for every request to the httpbin route. The `/status/200` path makes `httpbin.org` allow every request. The `http.path` value is a CEL expression, so the path is a string literal in double quotes. + ```yaml + kubectl apply -f- <}} + kind: {{< reuse "agw-docs/snippets/policy.md" >}} + metadata: + name: callout-ext-auth + namespace: httpbin + spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: HTTPRoute + name: httpbin + traffic: + extAuth: + backendRef: + group: {{< reuse "agw-docs/snippets/group.md" >}} + kind: {{< reuse "agw-docs/snippets/backend.md" >}} + name: httpbin-org + http: + path: '"/status/200"' + EOF + ``` + +8. Send a request to the httpbin app. The request succeeds after the external authorization callout returns `200`. + {{< tabs >}} + {{% tab name="Cloud Provider LoadBalancer" %}} + ```sh + curl -i http://$INGRESS_GW_ADDRESS:80/headers -H "host: www.example.com" + ``` + {{% /tab %}} + {{% tab name="Port-forward for local testing" %}} + The rollout in step 4 replaced the agentgateway proxy pod, so start the port-forward again. + ```sh + kubectl port-forward deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} 8080:80 & + curl -i localhost:8080/headers -H "host: www.example.com" + ``` + {{% /tab %}} + {{< /tabs >}} + + Example output: + ```console + HTTP/1.1 200 OK + ``` + +9. Check the Squid access log for a CONNECT request to `httpbin.org:443`. + ```sh + kubectl exec -n squid deployment/squid -- cat /var/log/squid/access.log + ``` + + Example output: + ```console + 1791983992.466 46059 10.244.0.19 TCP_TUNNEL/200 4933 CONNECT httpbin.org:443 - HIER_DIRECT/54.159.186.149 - + ``` + +10. If the log has no CONNECT entry, restart the agentgateway proxy, and then check the log again. Squid writes the entry only when the tunnel closes, and the agentgateway proxy keeps the tunnel open to reuse it. The restart closes the tunnel. + ```sh + kubectl rollout restart deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + kubectl rollout status deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + kubectl exec -n squid deployment/squid -- cat /var/log/squid/access.log + ``` + +## Cleanup + +{{< reuse "agw-docs/snippets/cleanup.md" >}} + +1. Delete the external authorization policy and the backend. + ```sh + kubectl delete {{< reuse "agw-docs/snippets/policy.md" >}} callout-ext-auth -n httpbin + kubectl delete {{< reuse "agw-docs/snippets/backend.md" >}} httpbin-org -n httpbin + ``` + +2. Remove `infrastructure.parametersRef` from your Gateway, and delete the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. + ```sh + kubectl patch gateway agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} --type json -p '[{"op":"remove","path":"/spec/infrastructure/parametersRef"}]' + kubectl delete {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} callout-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + ``` + +3. Delete Squid. + ```sh + kubectl delete namespace squid + ``` diff --git a/content/docs/kubernetes/main/release-notes/release-notes.md b/content/docs/kubernetes/main/release-notes/release-notes.md index b29686c63..506215665 100644 --- a/content/docs/kubernetes/main/release-notes/release-notes.md +++ b/content/docs/kubernetes/main/release-notes/release-notes.md @@ -90,7 +90,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ - **Streaming guardrails**: Agentgateway now rejects content if a provider guard fails while checking a streamed response or realtime connection. To keep the 1.5 behavior, set `failureMode: FailOpen` on the guard. For example, set it in `spec.backend.ai.promptGuard.response[].bedrockGuardrails` of an {{< reuse "agw-docs/snippets/policy.md" >}}. For more information, see [Provider failures]({{< link-hextra path="/documentation/llm/guardrails/overview/#provider-failures" >}}). - **Token usage in responses**: The `usage` that agentgateway returns to clients now follows the conventions of the API that the client calls, whichever provider serves the request. Anthropic Messages responses leave cache tokens out of `input_tokens` and report them in `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI Chat Completions and Responses include cache tokens in the input and total token counts. For example, a Chat Completions request to an Anthropic or Amazon Bedrock model that writes to the prompt cache now reports those cache tokens in `prompt_tokens`. If a client calculates usage or cost from these fields, review its counts after you upgrade. - **Deployer ownership**: The controller no longer overwrites an existing resource of the same name that it does not own. It reports an error instead. -- **Proxy environment variables**: Callouts, such as JWKS fetches, external authorization requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables. If you set these variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource for another purpose, callouts go through that proxy after you upgrade. Callouts to a Kubernetes Service go directly to the Service. To keep all callouts direct, set `NO_PROXY` to `*`. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. +- **Proxy environment variables**: Callouts from the proxy, such as external authorization requests, guardrail requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables. If you set these variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource for another purpose, callouts go through that proxy after you upgrade. Callouts to a Kubernetes Service go directly to the Service. Callouts to an {{< reuse "agw-docs/snippets/backend.md" >}} with a `static` host go through the proxy, even for a `svc.cluster.local` hostname, unless `NO_PROXY` lists `svc.cluster.local`. To keep all callouts direct, set `NO_PROXY` to `*`. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). ## 🌟 New features {#v16-new-features} @@ -167,6 +167,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ + - **Session affinity**: Session affinity sends related requests to the same endpoint. Set `spec.backend.sessionAffinity` in an {{< reuse "agw-docs/snippets/policy.md" >}}. Its CEL `source` expression provides the value to hash, such as a session header. For more information, see [Session affinity]({{< link-hextra path="/documentation/traffic-management/load-balancing/#session-affinity" >}}). - **Per-key local rate limits**: Local rate limits can now give each user or API key a separate bucket. Set a CEL `key` in `spec.traffic.rateLimit.local` of an {{< reuse "agw-docs/snippets/policy.md" >}}. For more information, see [Claim-level rate limits]({{< link-hextra path="/documentation/security/rate-limit-http/#claim-level" >}}). @@ -174,6 +175,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ - **Telemetry**: OTLP access logs use the `agentgateway.access` instrumentation scope. The request duration metric records failed requests with an `error_type` label. - **Istio permissions**: Set the `istio.enabled=false` Helm value to drop the Istio permissions from the controller ClusterRole. For more information, see [Istio resource discovery]({{< link-hextra path="/documentation/install/advanced/#istio-discovery" >}}). - **Gateway name in proxy metrics**: When `monitoring.enabled` is `true`, the Helm chart creates a PodMonitor. The PodMonitor now copies the `gateway.networking.k8s.io/gateway-name` pod label onto proxy metrics. This label lets the Grafana dashboard filter by Gateway. To change the copied labels, set the `monitoring.proxy.podMonitor.podTargetLabels` Helm value. +- **Callout proxy**: You can now send the requests that the agentgateway proxy makes on its own behalf, such as external authorization requests, guardrail requests, and cloud provider credential fetches, through a forward proxy. Set the standard proxy environment variables in `spec.env` of an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, or set `config.callouts.tunnel` in `spec.rawConfig`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). ### OpenTelemetry field names {#v16-otel-attributes} diff --git a/content/docs/standalone/main/documentation/configuration/callout-proxy.md b/content/docs/standalone/main/documentation/configuration/callout-proxy.md index e447ea1af..6a945419f 100644 --- a/content/docs/standalone/main/documentation/configuration/callout-proxy.md +++ b/content/docs/standalone/main/documentation/configuration/callout-proxy.md @@ -15,18 +15,20 @@ A callout is any request that agentgateway sends for itself rather than for a cl * [External authorization]({{< link-hextra path="/documentation/configuration/security/external-authz" >}}), external processing, and remote rate limit requests. * Guardrail requests, such as webhooks, OpenAI moderation, AWS Bedrock Guardrails, Google Model Armor, and Azure AI Content Safety. * Virtual model callout routing requests. -* Cloud provider credential fetches, such as AWS and Azure credentials. +* Credential fetches from cloud providers, such as AWS, Azure, and Google Cloud Platform (GCP). Client traffic to your primary backends is not a callout, so the callout proxy does not apply to it. To send backend traffic through a proxy, set the `backendTunnel` policy on the backend instead. To proxy CONNECT requests from your clients instead, see [Egress proxy]({{< link-hextra path="/documentation/configuration/egress-proxy/" >}}). -## Configure the proxy +## Configure the callout proxy -You can configure the callout proxy in either of the following ways. +You can configure the callout proxy in either of the following ways. If you use both, `config.callouts.tunnel` takes precedence, and agentgateway ignores the proxy environment variables for callouts. -* **In the configuration file**: Set `config.callouts.tunnel`. Settings in the `config` section take effect at startup, so restart agentgateway after you change this setting. For more information, see [Static configuration]({{< link-hextra path="/documentation/configuration/static-configuration/" >}}). -* **With environment variables**: Omit `config.callouts.tunnel` and set the standard proxy environment variables in the environment that runs agentgateway. When you set `config.callouts.tunnel`, the proxy environment variables have no effect on callouts. +* **[In the configuration file](#configuration-file)**: Set `config.callouts.tunnel`. Use this way to keep the proxy settings with the rest of your configuration. +* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`. Use this way to send GCP credential fetches through the proxy, or to reuse variables that your environment already sets. -### Configuration file +### Option 1: Configuration file + +Set `config.callouts.tunnel` in your configuration file. Settings in the `config` section take effect at startup, so restart agentgateway after you change this setting. For more information, see [Static configuration]({{< link-hextra path="/documentation/configuration/static-configuration/" >}}). ```yaml config: @@ -41,17 +43,17 @@ config: | Setting | Description | | -- | -- | | `config.callouts.tunnel.url` | The URL of the proxy. This proxy handles every callout, whether the callout uses Transport Layer Security (TLS) or plaintext. The scheme must be `http` or `https`. With `https`, the connection to the proxy uses TLS, and the proxy certificate is verified against the system trust bundle. If you omit the port, the default is `80` for `http` and `443` for `https`. To authenticate to the proxy with basic authentication, include the credentials in the URL, such as `http://user:password@proxy.example.com:3128`. The `--validate-only` flag does not check the scheme. A URL with an unsupported scheme, such as `socks5://`, passes validation, and then agentgateway fails to start with an `invalid callouts config` error. | -| `config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the proxy. Entries use the `NO_PROXY` syntax: a hostname, which also matches its subdomains, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` to bypass the proxy for every destination. | +| `config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the proxy. Each entry is a hostname, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` for every destination. A hostname also matches its subdomains. | -### Environment variables +### Option 2: Environment variables -If you omit `config.callouts.tunnel`, agentgateway reads the following environment variables at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. +Set the following variables in the environment that runs agentgateway. Agentgateway reads them at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. | Variable | Description | | -- | -- | | `HTTPS_PROXY` | The proxy for callouts that use TLS. | | `HTTP_PROXY` | The proxy for plaintext callouts. | -| `ALL_PROXY` | The proxy for callouts whose scheme-specific variable is unset. | +| `ALL_PROXY` | The proxy for callouts when you omit `HTTPS_PROXY` or `HTTP_PROXY`. | | `NO_PROXY` | A comma-separated list of destinations that bypass the proxy, in the same syntax as `noProxy`. | > [!IMPORTANT] @@ -65,7 +67,7 @@ The following callouts go directly to their destination, even when you configure * Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote proxy cannot reach these addresses. This rule applies to IP addresses only. A callout to the hostname `localhost` goes through the proxy, so add `localhost` to `noProxy` or `NO_PROXY` if your callout targets use it. * Callouts to a backend that has its own `backendTunnel` policy. These callouts use that tunnel instead. * Callouts to a service from the top-level `services` list, and to Unix domain sockets. -* Google Cloud Platform (GCP) credential fetches, when you configure the proxy with `config.callouts.tunnel`. To send these fetches through a proxy, use the environment variables instead. +* GCP credential fetches, when you configure the proxy with `config.callouts.tunnel`. To send these fetches through a proxy, use the environment variables instead. ## Before you begin From a7cb79c8a7a32eadcd4c2993d2a43d09f6f42007 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Wed, 14 Oct 2026 09:58:21 -0400 Subject: [PATCH 5/6] Wording tweaks Signed-off-by: Kristin Brown --- .../main/documentation/setup/customize/callout-proxy.md | 2 +- .../main/documentation/configuration/callout-proxy.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md index fd16acad5..f83a1d264 100644 --- a/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md +++ b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md @@ -84,7 +84,7 @@ The following callouts go directly to their destination, even when you configure ## Send an external authorization callout through the forward proxy -Deploy a test forward proxy, point the agentgateway proxy at it, and confirm that an external authorization callout goes through the forward proxy. The example uses `httpbin.org` as a stand-in authorization server, because the server returns any status code that you ask for. +Deploy a test forward proxy, point the agentgateway proxy at it with the `HTTPS_PROXY` environment variable, and confirm that an external authorization callout goes through the forward proxy. To use `rawConfig` instead, change step 2 as that step describes. The example uses `httpbin.org` as a stand-in authorization server, because the server returns any status code that you ask for. 1. Deploy Squid as a test forward proxy. If you already have a forward proxy that the cluster can reach, skip this step and use its URL in the next step. ```yaml diff --git a/content/docs/standalone/main/documentation/configuration/callout-proxy.md b/content/docs/standalone/main/documentation/configuration/callout-proxy.md index 6a945419f..00e73d52f 100644 --- a/content/docs/standalone/main/documentation/configuration/callout-proxy.md +++ b/content/docs/standalone/main/documentation/configuration/callout-proxy.md @@ -23,8 +23,8 @@ Client traffic to your primary backends is not a callout, so the callout proxy d You can configure the callout proxy in either of the following ways. If you use both, `config.callouts.tunnel` takes precedence, and agentgateway ignores the proxy environment variables for callouts. -* **[In the configuration file](#configuration-file)**: Set `config.callouts.tunnel`. Use this way to keep the proxy settings with the rest of your configuration. -* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`. Use this way to send GCP credential fetches through the proxy, or to reuse variables that your environment already sets. +* **[In the configuration file](#configuration-file)**: Set `config.callouts.tunnel`. Use this method to keep the proxy settings with the rest of your configuration. +* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`. Use this method to send GCP credential fetches through the proxy, or to reuse variables that your environment already sets. ### Option 1: Configuration file @@ -77,7 +77,7 @@ You also need an HTTP proxy that supports the CONNECT method and logs the reques ## Send a JWKS fetch through the proxy -Configure JWT authentication with a remote JWKS, and confirm in the proxy log that the JWKS fetch goes through the proxy. A JWKS fetch is a good test because agentgateway makes it at startup, before any client traffic arrives. +Configure JWT authentication with a remote JWKS, and send callouts through the proxy with `config.callouts.tunnel`. Then confirm in the proxy log that the JWKS fetch goes through the proxy. The last step shows the same test with environment variables instead. A JWKS fetch is a good test because agentgateway makes it at startup, before any client traffic arrives. 1. Create a configuration file that fetches a remote JWKS and sends callouts through the proxy. ```yaml From 6f3a8b004809edd01ed0fabe78b4f04cb4a79801 Mon Sep 17 00:00:00 2001 From: Kristin Brown Date: Fri, 16 Oct 2026 12:24:14 -0400 Subject: [PATCH 6/6] Review fixes Signed-off-by: Kristin Brown --- .../setup/customize/callout-proxy.md | 46 ++++++-- .../main/release-notes/release-notes.md | 4 +- .../configuration/callout-proxy.md | 105 ++++++++++++------ .../main/release-notes/release-notes.md | 2 +- 4 files changed, 109 insertions(+), 48 deletions(-) diff --git a/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md index f83a1d264..729215b95 100644 --- a/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md +++ b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md @@ -23,12 +23,14 @@ The callout proxy does not apply to the following traffic. ## Configure the callout proxy -You configure the callout proxy on the agentgateway proxy deployment through an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, in either of the following ways. If you use both, `config.callouts.tunnel` takes precedence, and the agentgateway proxy ignores the proxy environment variables for callouts. +You configure the callout proxy on the agentgateway proxy deployment through an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, in either of the following ways. * **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`, in `spec.env`. Use this configuration for most setups. Note that this configuration is also the only way to send GCP credential fetches through the forward proxy. -* **[In `rawConfig`](#rawconfig)**: Set `config.callouts.tunnel` in `spec.rawConfig`. Use this way to keep the forward proxy settings separate from environment variables that other tools in the pod read. The control plane reads `rawConfig` only from an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource that a Gateway references. +* **[In `rawConfig`](#rawconfig)**: Set `config.callouts.tunnel` in `spec.rawConfig`. Use this way to keep the forward proxy settings separate from environment variables that other tools in the pod read. -### Option 1: Environment variables +If you use both, `config.callouts.tunnel` takes precedence, and the agentgateway proxy ignores the proxy environment variables for most callouts. GCP credential fetches are the exception. These fetches never use `config.callouts.tunnel`, and they still use the proxy environment variables if you set them. + +### Option 1: Environment variables {#environment-variables} Set the following variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. The agentgateway proxy reads them at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. @@ -39,12 +41,30 @@ Set the following variables in `spec.env` of the {{< reuse "agw-docs/snippets/ga | `ALL_PROXY` | The forward proxy for callouts when you omit `HTTPS_PROXY` or `HTTP_PROXY`. | | `NO_PROXY` | A comma-separated list of destinations that bypass the forward proxy. Each entry is a hostname, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` for every destination. A hostname also matches its subdomains. | +You can set `NO_PROXY` together with the other variables. The following example sends TLS and plaintext callouts through the same forward proxy, and keeps callouts to `internal.example.com`, its subdomains, and the `10.0.0.0/8` range direct. + +```yaml +apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}} +kind: {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} +metadata: + name: callout-proxy + namespace: {{< reuse "agw-docs/snippets/namespace.md" >}} +spec: + env: + - name: HTTPS_PROXY + value: http://proxy.example.com:3128 + - name: HTTP_PROXY + value: http://proxy.example.com:3128 + - name: NO_PROXY + value: internal.example.com,10.0.0.0/8 +``` + > [!IMPORTANT] > The proxy environment variables apply to callouts even when you set the variables for another purpose. If `spec.env` already sets `HTTPS_PROXY` for other tools, callouts also go through that forward proxy. To keep callouts direct, remove the variables from `spec.env`, or set `NO_PROXY` to `*`. -### Option 2: rawConfig +### Option 2: rawConfig {#rawconfig} -Set `config.callouts.tunnel` under `spec.rawConfig` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. +Set `config.callouts.tunnel` under `spec.rawConfig` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. You can set `rawConfig` on an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource that a Gateway or a GatewayClass references. If both resources set `rawConfig`, the Gateway's `rawConfig` replaces the GatewayClass's `rawConfig` as a whole. The two are not merged. ```yaml apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}} @@ -77,6 +97,7 @@ The following callouts go directly to their destination, even when you configure * Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote forward proxy cannot reach these addresses. * Callouts to an {{< reuse "agw-docs/snippets/backend.md" >}} that has its own `tunnel` policy. These callouts use that tunnel instead. * GCP credential fetches, when you configure the callout proxy with `config.callouts.tunnel`. To send these fetches through a forward proxy, use the environment variables instead. +* Remote JWKS fetches. The control plane makes these fetches, not the agentgateway proxy. For more information, see [About callouts](#about-callouts). ## Before you begin @@ -147,7 +168,7 @@ Deploy a test forward proxy, point the agentgateway proxy at it with the `HTTPS_ To use `rawConfig` instead, replace `spec.env` with `spec.rawConfig.config.callouts.tunnel.url`, as shown in [rawConfig](#rawconfig). -3. Attach the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource to your Gateway with `infrastructure.parametersRef`. The patch keeps the rest of your Gateway as it is. If your Gateway already references an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, add the `spec.env` setting to that resource instead, because a Gateway references one resource only. +3. Attach the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource to your Gateway. The patch keeps the rest of your Gateway as it is. If your Gateway already references an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, add the `spec.env` setting to that resource instead, because a Gateway can reference one {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource only. ```sh kubectl patch gateway agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} --type merge \ -p '{"spec":{"infrastructure":{"parametersRef":{"group":"{{< reuse "agw-docs/snippets/group.md" >}}","kind":"{{< reuse "agw-docs/snippets/gatewayparameters.md" >}}","name":"callout-proxy"}}}}' @@ -158,7 +179,7 @@ Deploy a test forward proxy, point the agentgateway proxy at it with the `HTTPS_ kubectl rollout status deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} ``` -5. Check the agentgateway proxy logs for the line that names the forward proxy. The label selector reads every agentgateway proxy pod, so the command also works while an old pod is still shutting down. +5. Check the agentgateway proxy logs for the line that names the forward proxy. ```sh kubectl logs -n {{< reuse "agw-docs/snippets/namespace.md" >}} -l gateway.networking.k8s.io/gateway-name=agentgateway-proxy --tail=-1 | grep "callouts will use proxy" ``` @@ -186,7 +207,7 @@ Deploy a test forward proxy, point the agentgateway proxy at it with the `HTTPS_ EOF ``` -7. Create an {{< reuse "agw-docs/snippets/policy.md" >}} that sends an external authorization request for every request to the httpbin route. The `/status/200` path makes `httpbin.org` allow every request. The `http.path` value is a CEL expression, so the path is a string literal in double quotes. +7. Create an {{< reuse "agw-docs/snippets/policy.md" >}} that sends an external authorization request for every request to the httpbin stand-in authorization server. The `/status/200` path makes `httpbin.org` allow every request. The `http.path` value is a CEL expression, so the path is a string literal in double quotes. ```yaml kubectl apply -f- <}} @@ -231,7 +252,7 @@ Deploy a test forward proxy, point the agentgateway proxy at it with the `HTTPS_ HTTP/1.1 200 OK ``` -9. Check the Squid access log for a CONNECT request to `httpbin.org:443`. +9. Check the Squid access log for a CONNECT request to `httpbin.org:443`. This entry shows the whole callout path, from the agentgateway proxy through Squid to `httpbin.org`. ```sh kubectl exec -n squid deployment/squid -- cat /var/log/squid/access.log ``` @@ -241,6 +262,13 @@ Deploy a test forward proxy, point the agentgateway proxy at it with the `HTTPS_ 1791983992.466 46059 10.244.0.19 TCP_TUNNEL/200 4933 CONNECT httpbin.org:443 - HIER_DIRECT/54.159.186.149 - ``` + | Field | Description | + | -- | -- | + | `10.244.0.19` | The client of Squid, which is the agentgateway proxy pod. | + | `TCP_TUNNEL/200` | Squid opened the CONNECT tunnel. | + | `CONNECT httpbin.org:443` | The destination of the external authorization callout. | + | `HIER_DIRECT/54.159.186.149` | Squid connected directly to `httpbin.org` at this IP address. | + 10. If the log has no CONNECT entry, restart the agentgateway proxy, and then check the log again. Squid writes the entry only when the tunnel closes, and the agentgateway proxy keeps the tunnel open to reuse it. The restart closes the tunnel. ```sh kubectl rollout restart deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} diff --git a/content/docs/kubernetes/main/release-notes/release-notes.md b/content/docs/kubernetes/main/release-notes/release-notes.md index 506215665..b8fb2e75a 100644 --- a/content/docs/kubernetes/main/release-notes/release-notes.md +++ b/content/docs/kubernetes/main/release-notes/release-notes.md @@ -90,7 +90,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ - **Streaming guardrails**: Agentgateway now rejects content if a provider guard fails while checking a streamed response or realtime connection. To keep the 1.5 behavior, set `failureMode: FailOpen` on the guard. For example, set it in `spec.backend.ai.promptGuard.response[].bedrockGuardrails` of an {{< reuse "agw-docs/snippets/policy.md" >}}. For more information, see [Provider failures]({{< link-hextra path="/documentation/llm/guardrails/overview/#provider-failures" >}}). - **Token usage in responses**: The `usage` that agentgateway returns to clients now follows the conventions of the API that the client calls, whichever provider serves the request. Anthropic Messages responses leave cache tokens out of `input_tokens` and report them in `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI Chat Completions and Responses include cache tokens in the input and total token counts. For example, a Chat Completions request to an Anthropic or Amazon Bedrock model that writes to the prompt cache now reports those cache tokens in `prompt_tokens`. If a client calculates usage or cost from these fields, review its counts after you upgrade. - **Deployer ownership**: The controller no longer overwrites an existing resource of the same name that it does not own. It reports an error instead. -- **Proxy environment variables**: Callouts from the proxy, such as external authorization requests, guardrail requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables. If you set these variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource for another purpose, callouts go through that proxy after you upgrade. Callouts to a Kubernetes Service go directly to the Service. Callouts to an {{< reuse "agw-docs/snippets/backend.md" >}} with a `static` host go through the proxy, even for a `svc.cluster.local` hostname, unless `NO_PROXY` lists `svc.cluster.local`. To keep all callouts direct, set `NO_PROXY` to `*`. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). +- **Proxy environment variables**: Callouts from the proxy, such as external authorization requests, guardrail requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables to configure a forward proxy that these requests must be routed through. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. If you set these variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource for another purpose, callouts go through that forward proxy after you upgrade. To keep all callouts direct, set `NO_PROXY` to `*`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). ## 🌟 New features {#v16-new-features} @@ -175,7 +175,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ - **Telemetry**: OTLP access logs use the `agentgateway.access` instrumentation scope. The request duration metric records failed requests with an `error_type` label. - **Istio permissions**: Set the `istio.enabled=false` Helm value to drop the Istio permissions from the controller ClusterRole. For more information, see [Istio resource discovery]({{< link-hextra path="/documentation/install/advanced/#istio-discovery" >}}). - **Gateway name in proxy metrics**: When `monitoring.enabled` is `true`, the Helm chart creates a PodMonitor. The PodMonitor now copies the `gateway.networking.k8s.io/gateway-name` pod label onto proxy metrics. This label lets the Grafana dashboard filter by Gateway. To change the copied labels, set the `monitoring.proxy.podMonitor.podTargetLabels` Helm value. -- **Callout proxy**: You can now send the requests that the agentgateway proxy makes on its own behalf, such as external authorization requests, guardrail requests, and cloud provider credential fetches, through a forward proxy. Set the standard proxy environment variables in `spec.env` of an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, or set `config.callouts.tunnel` in `spec.rawConfig`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). +- **Callout proxy**: You can now send the requests that the agentgateway proxy makes on its own behalf, such as external authorization requests, guardrail requests, and cloud provider credential fetches, through a forward proxy. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). ### OpenTelemetry field names {#v16-otel-attributes} diff --git a/content/docs/standalone/main/documentation/configuration/callout-proxy.md b/content/docs/standalone/main/documentation/configuration/callout-proxy.md index 00e73d52f..2419407ec 100644 --- a/content/docs/standalone/main/documentation/configuration/callout-proxy.md +++ b/content/docs/standalone/main/documentation/configuration/callout-proxy.md @@ -1,10 +1,10 @@ --- title: Callout proxy weight: 38 -description: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches and external authorization calls, through an HTTP proxy. +description: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches and external authorization calls, through a forward proxy. --- -In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that agentgateway makes on its own behalf through that proxy. These requests are called callouts. Callouts that use TLS go through the proxy in an HTTP CONNECT tunnel, and plaintext callouts go to the proxy as standard forward-proxy requests. +In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that agentgateway makes on its own behalf through that forward proxy. These requests are called callouts. Callouts that use TLS go through the forward proxy in an HTTP CONNECT tunnel, and plaintext callouts go to the forward proxy as standard forward-proxy requests. ## About callouts @@ -17,18 +17,20 @@ A callout is any request that agentgateway sends for itself rather than for a cl * Virtual model callout routing requests. * Credential fetches from cloud providers, such as AWS, Azure, and Google Cloud Platform (GCP). -Client traffic to your primary backends is not a callout, so the callout proxy does not apply to it. To send backend traffic through a proxy, set the `backendTunnel` policy on the backend instead. To proxy CONNECT requests from your clients instead, see [Egress proxy]({{< link-hextra path="/documentation/configuration/egress-proxy/" >}}). +Client traffic to your primary backends is not a callout, so the callout proxy does not apply to it. To send backend traffic through a forward proxy, set the `backendTunnel` policy on the backend instead. To proxy CONNECT requests from your clients instead, see [Egress proxy]({{< link-hextra path="/documentation/configuration/egress-proxy/" >}}). ## Configure the callout proxy -You can configure the callout proxy in either of the following ways. If you use both, `config.callouts.tunnel` takes precedence, and agentgateway ignores the proxy environment variables for callouts. +You can configure the callout proxy in either of the following ways. -* **[In the configuration file](#configuration-file)**: Set `config.callouts.tunnel`. Use this method to keep the proxy settings with the rest of your configuration. -* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`. Use this method to send GCP credential fetches through the proxy, or to reuse variables that your environment already sets. +* **[In the configuration file](#configuration-file)**: Set `config.callouts.tunnel`. Use this method to keep the forward proxy settings with the rest of your configuration. +* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`. Use this method to send GCP credential fetches through the forward proxy, or to reuse variables that your environment already sets. -### Option 1: Configuration file +If you use both, `config.callouts.tunnel` takes precedence, and agentgateway ignores the proxy environment variables for most callouts. GCP credential fetches are the exception. These fetches never use `config.callouts.tunnel`, and they still use the proxy environment variables if you set them. -Set `config.callouts.tunnel` in your configuration file. Settings in the `config` section take effect at startup, so restart agentgateway after you change this setting. For more information, see [Static configuration]({{< link-hextra path="/documentation/configuration/static-configuration/" >}}). +### Option 1: Configuration file {#configuration-file} + +Set `config.callouts.tunnel` in your configuration file to configure the forward proxy that you want to route requests through. Settings in the `config` section take effect at startup, so restart agentgateway after you change this setting. For more information, see [Static configuration]({{< link-hextra path="/documentation/configuration/static-configuration/" >}}). ```yaml config: @@ -42,44 +44,65 @@ config: | Setting | Description | | -- | -- | -| `config.callouts.tunnel.url` | The URL of the proxy. This proxy handles every callout, whether the callout uses Transport Layer Security (TLS) or plaintext. The scheme must be `http` or `https`. With `https`, the connection to the proxy uses TLS, and the proxy certificate is verified against the system trust bundle. If you omit the port, the default is `80` for `http` and `443` for `https`. To authenticate to the proxy with basic authentication, include the credentials in the URL, such as `http://user:password@proxy.example.com:3128`. The `--validate-only` flag does not check the scheme. A URL with an unsupported scheme, such as `socks5://`, passes validation, and then agentgateway fails to start with an `invalid callouts config` error. | -| `config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the proxy. Each entry is a hostname, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` for every destination. A hostname also matches its subdomains. | +| `config.callouts.tunnel.url` | The URL and port of the forward proxy. This forward proxy handles every callout, whether the callout uses Transport Layer Security (TLS) or plaintext. The scheme must be `http` or `https`. With `https`, the connection to the forward proxy uses TLS, and the forward proxy certificate is verified against the system trust bundle. If you omit the port, the default is `80` for `http` and `443` for `https`. To authenticate to the forward proxy with basic authentication, include the credentials in the URL, such as `http://user:password@proxy.example.com:3128`. The `agentgateway --validate-only` command, which checks a configuration file without starting agentgateway, does not check the scheme. A URL with an unsupported scheme, such as `socks5://`, passes validation, and then agentgateway fails to start with an `invalid callouts config` error. | +| `config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the forward proxy. Each entry is a hostname, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` for every destination. A hostname also matches its subdomains. | -### Option 2: Environment variables +### Option 2: Environment variables {#environment-variables} Set the following variables in the environment that runs agentgateway. Agentgateway reads them at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. | Variable | Description | | -- | -- | -| `HTTPS_PROXY` | The proxy for callouts that use TLS. | -| `HTTP_PROXY` | The proxy for plaintext callouts. | -| `ALL_PROXY` | The proxy for callouts when you omit `HTTPS_PROXY` or `HTTP_PROXY`. | -| `NO_PROXY` | A comma-separated list of destinations that bypass the proxy, in the same syntax as `noProxy`. | +| `HTTPS_PROXY` | The forward proxy for callouts that use TLS. | +| `HTTP_PROXY` | The forward proxy for plaintext callouts. | +| `ALL_PROXY` | The forward proxy for callouts when you omit `HTTPS_PROXY` or `HTTP_PROXY`. | +| `NO_PROXY` | A comma-separated list of destinations that bypass the forward proxy, in the same syntax as `noProxy`. | + +You can set `NO_PROXY` together with the other variables. The following example sends TLS and plaintext callouts through the same forward proxy, and keeps callouts to `internal.example.com`, its subdomains, and the `10.0.0.0/8` range direct. + +```sh +export HTTPS_PROXY=http://proxy.example.com:3128 +export HTTP_PROXY=http://proxy.example.com:3128 +export NO_PROXY=internal.example.com,10.0.0.0/8 +agentgateway -f config.yaml +``` > [!IMPORTANT] -> The proxy environment variables apply to callouts even when you set the variables for another purpose. If the environment that runs agentgateway exports `HTTPS_PROXY` for other tools, callouts also go through that proxy. To keep callouts direct, unset the variables for the agentgateway process, or set `NO_PROXY` to `*`. +> The proxy environment variables apply to callouts even when you set the variables for another purpose. If the environment that runs agentgateway exports `HTTPS_PROXY` for other tools, callouts also go through that forward proxy. To keep callouts direct, unset the variables for the agentgateway process, or set `NO_PROXY` to `*`. -## Destinations that bypass the proxy +## Destinations that bypass the forward proxy {#destinations-that-bypass-the-proxy} -The following callouts go directly to their destination, even when you configure a proxy. +The following callouts go directly to their destination, even when you configure a forward proxy. * Destinations that match `noProxy` or `NO_PROXY`. -* Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote proxy cannot reach these addresses. This rule applies to IP addresses only. A callout to the hostname `localhost` goes through the proxy, so add `localhost` to `noProxy` or `NO_PROXY` if your callout targets use it. +* Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote forward proxy cannot reach these addresses. This rule applies to IP addresses only. A callout to the hostname `localhost` goes through the forward proxy, so add `localhost` to `noProxy` or `NO_PROXY` if your callout targets use it. * Callouts to a backend that has its own `backendTunnel` policy. These callouts use that tunnel instead. * Callouts to a service from the top-level `services` list, and to Unix domain sockets. -* GCP credential fetches, when you configure the proxy with `config.callouts.tunnel`. To send these fetches through a proxy, use the environment variables instead. +* GCP credential fetches, when you configure the proxy with `config.callouts.tunnel`. To send these fetches through a forward proxy, use the environment variables instead. ## Before you begin -{{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} - -You also need an HTTP proxy that supports the CONNECT method and logs the requests that it receives, such as Squid. The following steps assume that the proxy listens on `http://127.0.0.1:3128`. - -## Send a JWKS fetch through the proxy - -Configure JWT authentication with a remote JWKS, and send callouts through the proxy with `config.callouts.tunnel`. Then confirm in the proxy log that the JWKS fetch goes through the proxy. The last step shows the same test with environment variables instead. A JWKS fetch is a good test because agentgateway makes it at startup, before any client traffic arrives. - -1. Create a configuration file that fetches a remote JWKS and sends callouts through the proxy. +1. {{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} +2. Set up a forward proxy that supports the CONNECT method and logs the requests that it receives. The following steps assume that the forward proxy listens on `http://127.0.0.1:3128`. If you do not have a forward proxy, run Squid in Docker as a test forward proxy. + 1. Create a Squid configuration file that allows requests from any client. The default Squid configuration rejects requests from the Docker bridge network with a `403` response. + ```sh + cat < squid.conf + http_port 3128 + acl all src all + http_access allow all + EOF + ``` + 2. Start Squid with this configuration file, and publish port `3128` on your machine. + ```sh + docker run -d --name squid -p 3128:3128 \ + -v "$(pwd)/squid.conf:/etc/squid/squid.conf" ubuntu/squid:latest + ``` + +## Send a JWKS fetch through the forward proxy {#send-a-jwks-fetch-through-the-proxy} + +Configure JWT authentication with a remote JWKS, and send callouts through the forward proxy with `config.callouts.tunnel`. Then confirm in the forward proxy log that the JWKS fetch goes through the forward proxy. The last step shows the same test with environment variables instead. A JWKS fetch is a good test because agentgateway makes it at startup, before any client traffic arrives. + +1. Create a configuration file that fetches a remote JWKS and sends callouts through the forward proxy. ```yaml cat < callout-proxy.yaml # yaml-language-server: \$schema=https://agentgateway.dev/schema/config @@ -106,9 +129,9 @@ Configure JWT authentication with a remote JWKS, and send callouts through the p | Setting | Description | | -- | -- | - | `config.callouts.tunnel.url` | The address of the proxy. The loopback bypass applies to callout destinations, not to the proxy. A proxy on `127.0.0.1` still receives the callouts. | - | `jwtAuth.jwks.url` | The remote JWKS. The JWKS fetch is a callout and goes through the proxy. | - | `backends[].host` | The primary backend. Client requests to this backend are not callouts and do not use the proxy. | + | `config.callouts.tunnel.url` | The address of the forward proxy. The loopback bypass applies to callout destinations, not to the forward proxy. A forward proxy on `127.0.0.1` still receives the callouts. | + | `jwtAuth.jwks.url` | The remote JWKS. The JWKS fetch is a callout and goes through the forward proxy. | + | `backends[].host` | The primary backend. Client requests to this backend are not callouts and do not use the forward proxy. | 2. Validate the configuration. ```sh @@ -120,19 +143,29 @@ Configure JWT authentication with a remote JWKS, and send callouts through the p Configuration is valid! ``` -3. Start agentgateway. The JWKS fetch happens at startup. If the proxy is unreachable, the fetch fails and agentgateway exits with a `failed to load JWKS` error. +3. Start agentgateway. The JWKS fetch happens at startup. If the forward proxy is unreachable, the fetch fails and agentgateway exits with a `failed to load JWKS` error. ```sh agentgateway -f callout-proxy.yaml ``` - The startup logs include a line that names the callout proxy. + The startup logs include a line that names the forward proxy. ```console info client::callout callouts will use proxy Matcher { http: Intercept { uri: http://127.0.0.1:3128/ }, https: Intercept { uri: http://127.0.0.1:3128/ } } ``` -4. Check the access log of your proxy for a CONNECT request to `www.googleapis.com:443`, the JWKS host. The log format depends on your proxy. +4. In a second terminal, check the access log of your forward proxy for a CONNECT request to `www.googleapis.com:443`, the JWKS host. If you use the Squid container from [Before you begin](#before-you-begin), run the following command. For another forward proxy, the log location and format differ. + ```sh + docker exec squid cat /var/log/squid/access.log + ``` + + Example output: + ```console + 1791983992.466 1203 172.17.0.1 TCP_TUNNEL/200 6287 CONNECT www.googleapis.com:443 - HIER_DIRECT/142.250.80.74 - + ``` + + If the log has no CONNECT entry, stop agentgateway with Ctrl+C, and then check the log again. Squid writes the entry only when the tunnel closes, and agentgateway keeps the tunnel open to reuse it. -5. Optional: To use environment variables instead, remove the `config.callouts` section from the configuration file. Then, start agentgateway with the `HTTPS_PROXY` variable set. The proxy log shows the same CONNECT request. +5. Optional: To use environment variables instead, remove the `config.callouts` section from the configuration file. Then, start agentgateway with the `HTTPS_PROXY` variable set. The forward proxy log shows the same CONNECT request. ```sh HTTPS_PROXY=http://127.0.0.1:3128 agentgateway -f callout-proxy.yaml ``` diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 4b11cfee3..f3cb029b7 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -190,7 +190,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo - **Graceful drain**: During shutdown, agentgateway stops accepting new connections after `config.connectionMinTerminationDeadline`. It continues to serve open connections until `config.connectionTerminationDeadline`. For more information, see [Shutdown and drain]({{< link-hextra path="/documentation/configuration/static-configuration/#shutdown-drain" >}}). - **Helm chart scaling**: The standalone chart can create a PodDisruptionBudget and a HorizontalPodAutoscaler. For more information, see [Create a PodDisruptionBudget]({{< link-hextra path="/documentation/setup/install/helm/#helm-pdb" >}}). - **Telemetry**: OTLP access logs use the `agentgateway.access` instrumentation scope. The request duration metric records failed requests with an `error_type` label. For the metrics, see the [metrics reference]({{< link-hextra path="/documentation/observability/metrics/reference/#llm" >}}). -- **Callout proxy**: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches, OIDC discovery, and cloud provider credential fetches, through an HTTP proxy. Set `config.callouts.tunnel`, or use the standard proxy environment variables. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). +- **Callout proxy**: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches, OIDC discovery, and cloud provider credential fetches, through a forward proxy. Set `config.callouts.tunnel`, or use the standard proxy environment variables. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). ### OpenTelemetry field names {#v16-otel-attributes}