diff --git a/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md b/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md index ad431757d..30257342b 100644 --- a/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md +++ b/assets/agw-docs/pages/agentgateway/llm/providers/backend-tunnel-proxy.md @@ -5,6 +5,9 @@ Use the {{< reuse "agw-docs/snippets/backend.md" >}} `tunnel` policy to route ba The `tunnel` field on an {{< reuse "agw-docs/snippets/backend.md" >}} configures an HTTP CONNECT tunnel to the target backend. Agentgateway establishes an HTTP tunnel through the proxy and forwards all requests to the upstream endpoint. The tunnel supports `Service` and {{< reuse "agw-docs/snippets/backend.md" >}} types as the proxy target. +{{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,1.6.x,2.2.x" >}} +The `tunnel` policy applies to one backend. To send every request that the agentgateway proxy makes on its own behalf through a forward proxy, such as external authorization and guardrail requests, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). For callouts to a backend that has its own `tunnel` policy, that tunnel takes precedence over the callout proxy. +{{< /version >}} ## Before you begin diff --git a/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md new file mode 100644 index 000000000..729215b95 --- /dev/null +++ b/content/docs/kubernetes/main/documentation/setup/customize/callout-proxy.md @@ -0,0 +1,298 @@ +--- +title: Callout proxy +weight: 40 +description: Send the requests that the agentgateway proxy makes on its own behalf, such as external authorization calls and cloud credential fetches, through a forward proxy. +--- + +In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that the agentgateway proxy makes on its own behalf through that forward proxy. These requests are called callouts. Callouts that use TLS go through the forward proxy in an HTTP CONNECT tunnel, and plaintext callouts go to the forward proxy as standard forward-proxy requests. + +## About callouts + +A callout is any request that the agentgateway proxy sends for itself rather than for a client. Callouts include the following requests. + +* External authorization, external processing, and remote rate limit requests. +* Guardrail requests, such as webhooks, OpenAI moderation, AWS Bedrock Guardrails, Google Model Armor, and Azure AI Content Safety. +* OpenID Connect (OIDC) requests from the agentgateway proxy. +* Virtual model callout routing requests. +* Credential fetches from cloud providers, such as AWS, Azure, and Google Cloud Platform (GCP). + +The callout proxy does not apply to the following traffic. + +* **Client traffic to your backends**: This traffic is not a callout. To send the traffic to one backend through a forward proxy, set the `tunnel` policy on that backend instead. For more information, see [Backend tunnel proxy]({{< link-hextra path="/integrations/llm/providers/backend-tunnel-proxy/" >}}). +* **Remote JSON Web Key Set (JWKS) fetches**: In Kubernetes, the control plane fetches remote JWKS, not the agentgateway proxy, so the callout proxy settings have no effect on these fetches. To send a JWKS fetch through a forward proxy, set the `tunnel` policy on the {{< reuse "agw-docs/snippets/backend.md" >}} that serves the JWKS. + +## Configure the callout proxy + +You configure the callout proxy on the agentgateway proxy deployment through an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, in either of the following ways. + +* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`, in `spec.env`. Use this configuration for most setups. Note that this configuration is also the only way to send GCP credential fetches through the forward proxy. +* **[In `rawConfig`](#rawconfig)**: Set `config.callouts.tunnel` in `spec.rawConfig`. Use this way to keep the forward proxy settings separate from environment variables that other tools in the pod read. + +If you use both, `config.callouts.tunnel` takes precedence, and the agentgateway proxy ignores the proxy environment variables for most callouts. GCP credential fetches are the exception. These fetches never use `config.callouts.tunnel`, and they still use the proxy environment variables if you set them. + +### Option 1: Environment variables {#environment-variables} + +Set the following variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. The agentgateway proxy reads them at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. + +| Variable | Description | +| -- | -- | +| `HTTPS_PROXY` | The forward proxy for callouts that use TLS. | +| `HTTP_PROXY` | The forward proxy for plaintext callouts. | +| `ALL_PROXY` | The forward proxy for callouts when you omit `HTTPS_PROXY` or `HTTP_PROXY`. | +| `NO_PROXY` | A comma-separated list of destinations that bypass the forward proxy. Each entry is a hostname, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` for every destination. A hostname also matches its subdomains. | + +You can set `NO_PROXY` together with the other variables. The following example sends TLS and plaintext callouts through the same forward proxy, and keeps callouts to `internal.example.com`, its subdomains, and the `10.0.0.0/8` range direct. + +```yaml +apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}} +kind: {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} +metadata: + name: callout-proxy + namespace: {{< reuse "agw-docs/snippets/namespace.md" >}} +spec: + env: + - name: HTTPS_PROXY + value: http://proxy.example.com:3128 + - name: HTTP_PROXY + value: http://proxy.example.com:3128 + - name: NO_PROXY + value: internal.example.com,10.0.0.0/8 +``` + +> [!IMPORTANT] +> The proxy environment variables apply to callouts even when you set the variables for another purpose. If `spec.env` already sets `HTTPS_PROXY` for other tools, callouts also go through that forward proxy. To keep callouts direct, remove the variables from `spec.env`, or set `NO_PROXY` to `*`. + +### Option 2: rawConfig {#rawconfig} + +Set `config.callouts.tunnel` under `spec.rawConfig` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. You can set `rawConfig` on an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource that a Gateway or a GatewayClass references. If both resources set `rawConfig`, the Gateway's `rawConfig` replaces the GatewayClass's `rawConfig` as a whole. The two are not merged. + +```yaml +apiVersion: {{< reuse "agw-docs/snippets/api-version.md" >}} +kind: {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} +metadata: + name: callout-proxy + namespace: {{< reuse "agw-docs/snippets/namespace.md" >}} +spec: + rawConfig: + config: + callouts: + tunnel: + url: http://proxy.example.com:3128 + noProxy: + - internal.example.com + - 10.0.0.0/8 +``` + +| Setting | Description | +| -- | -- | +| `spec.rawConfig.config.callouts.tunnel.url` | The URL of the forward proxy. This forward proxy handles every callout, whether the callout uses TLS or plaintext. The scheme must be `http` or `https`. With `https`, the connection to the forward proxy uses TLS, and the forward proxy certificate is verified against the system trust bundle. If you omit the port, the default is `80` for `http` and `443` for `https`. To authenticate to the forward proxy with basic authentication, include the credentials in the URL, such as `http://user:password@proxy.example.com:3128`. | +| `spec.rawConfig.config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the forward proxy, in the same syntax as `NO_PROXY`. | + +## Destinations that bypass the forward proxy + +The following callouts go directly to their destination, even when you configure a callout proxy. + +* Callouts to a Kubernetes Service that you reference with a `Service` backendRef, such as an external authorization server in the cluster. An {{< reuse "agw-docs/snippets/backend.md" >}} with a `static` host is not a Service reference, even when the host is a cluster hostname such as `ext-authz.auth.svc.cluster.local`. Callouts to that backend go through the forward proxy, unless you add `svc.cluster.local` to `NO_PROXY` or `noProxy`. +* Destinations that match `noProxy` or `NO_PROXY`. +* Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote forward proxy cannot reach these addresses. +* Callouts to an {{< reuse "agw-docs/snippets/backend.md" >}} that has its own `tunnel` policy. These callouts use that tunnel instead. +* GCP credential fetches, when you configure the callout proxy with `config.callouts.tunnel`. To send these fetches through a forward proxy, use the environment variables instead. +* Remote JWKS fetches. The control plane makes these fetches, not the agentgateway proxy. For more information, see [About callouts](#about-callouts). + +## Before you begin + +{{< reuse "agw-docs/snippets/prereq.md" >}} + +## Send an external authorization callout through the forward proxy + +Deploy a test forward proxy, point the agentgateway proxy at it with the `HTTPS_PROXY` environment variable, and confirm that an external authorization callout goes through the forward proxy. To use `rawConfig` instead, change step 2 as that step describes. The example uses `httpbin.org` as a stand-in authorization server, because the server returns any status code that you ask for. + +1. Deploy Squid as a test forward proxy. If you already have a forward proxy that the cluster can reach, skip this step and use its URL in the next step. + ```yaml + kubectl apply -f- <}} resource that points callouts at the forward proxy. The external authorization callout in this guide uses TLS, so set `HTTPS_PROXY`. + ```yaml + kubectl apply -f- <}} + kind: {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} + metadata: + name: callout-proxy + namespace: {{< reuse "agw-docs/snippets/namespace.md" >}} + spec: + env: + - name: HTTPS_PROXY + value: http://squid.squid.svc.cluster.local:3128 + EOF + ``` + + To use `rawConfig` instead, replace `spec.env` with `spec.rawConfig.config.callouts.tunnel.url`, as shown in [rawConfig](#rawconfig). + +3. Attach the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource to your Gateway. The patch keeps the rest of your Gateway as it is. If your Gateway already references an {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource, add the `spec.env` setting to that resource instead, because a Gateway can reference one {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource only. + ```sh + kubectl patch gateway agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} --type merge \ + -p '{"spec":{"infrastructure":{"parametersRef":{"group":"{{< reuse "agw-docs/snippets/group.md" >}}","kind":"{{< reuse "agw-docs/snippets/gatewayparameters.md" >}}","name":"callout-proxy"}}}}' + ``` + +4. Wait for the agentgateway proxy to roll out with the new settings. + ```sh + kubectl rollout status deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + ``` + +5. Check the agentgateway proxy logs for the line that names the forward proxy. + ```sh + kubectl logs -n {{< reuse "agw-docs/snippets/namespace.md" >}} -l gateway.networking.k8s.io/gateway-name=agentgateway-proxy --tail=-1 | grep "callouts will use proxy" + ``` + + Example output: + ```console + info client::callout callouts will use proxy Matcher { https: Intercept { uri: http://squid.squid.svc.cluster.local:3128/ } } + ``` + +6. Create an {{< reuse "agw-docs/snippets/backend.md" >}} for the stand-in authorization server. Because the backend has a `static` host, callouts to it go through the forward proxy. + ```yaml + kubectl apply -f- <}} + kind: {{< reuse "agw-docs/snippets/backend.md" >}} + metadata: + name: httpbin-org + namespace: httpbin + spec: + static: + host: httpbin.org + port: 443 + policies: + tls: + sni: httpbin.org + EOF + ``` + +7. Create an {{< reuse "agw-docs/snippets/policy.md" >}} that sends an external authorization request for every request to the httpbin stand-in authorization server. The `/status/200` path makes `httpbin.org` allow every request. The `http.path` value is a CEL expression, so the path is a string literal in double quotes. + ```yaml + kubectl apply -f- <}} + kind: {{< reuse "agw-docs/snippets/policy.md" >}} + metadata: + name: callout-ext-auth + namespace: httpbin + spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: HTTPRoute + name: httpbin + traffic: + extAuth: + backendRef: + group: {{< reuse "agw-docs/snippets/group.md" >}} + kind: {{< reuse "agw-docs/snippets/backend.md" >}} + name: httpbin-org + http: + path: '"/status/200"' + EOF + ``` + +8. Send a request to the httpbin app. The request succeeds after the external authorization callout returns `200`. + {{< tabs >}} + {{% tab name="Cloud Provider LoadBalancer" %}} + ```sh + curl -i http://$INGRESS_GW_ADDRESS:80/headers -H "host: www.example.com" + ``` + {{% /tab %}} + {{% tab name="Port-forward for local testing" %}} + The rollout in step 4 replaced the agentgateway proxy pod, so start the port-forward again. + ```sh + kubectl port-forward deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} 8080:80 & + curl -i localhost:8080/headers -H "host: www.example.com" + ``` + {{% /tab %}} + {{< /tabs >}} + + Example output: + ```console + HTTP/1.1 200 OK + ``` + +9. Check the Squid access log for a CONNECT request to `httpbin.org:443`. This entry shows the whole callout path, from the agentgateway proxy through Squid to `httpbin.org`. + ```sh + kubectl exec -n squid deployment/squid -- cat /var/log/squid/access.log + ``` + + Example output: + ```console + 1791983992.466 46059 10.244.0.19 TCP_TUNNEL/200 4933 CONNECT httpbin.org:443 - HIER_DIRECT/54.159.186.149 - + ``` + + | Field | Description | + | -- | -- | + | `10.244.0.19` | The client of Squid, which is the agentgateway proxy pod. | + | `TCP_TUNNEL/200` | Squid opened the CONNECT tunnel. | + | `CONNECT httpbin.org:443` | The destination of the external authorization callout. | + | `HIER_DIRECT/54.159.186.149` | Squid connected directly to `httpbin.org` at this IP address. | + +10. If the log has no CONNECT entry, restart the agentgateway proxy, and then check the log again. Squid writes the entry only when the tunnel closes, and the agentgateway proxy keeps the tunnel open to reuse it. The restart closes the tunnel. + ```sh + kubectl rollout restart deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + kubectl rollout status deployment/agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + kubectl exec -n squid deployment/squid -- cat /var/log/squid/access.log + ``` + +## Cleanup + +{{< reuse "agw-docs/snippets/cleanup.md" >}} + +1. Delete the external authorization policy and the backend. + ```sh + kubectl delete {{< reuse "agw-docs/snippets/policy.md" >}} callout-ext-auth -n httpbin + kubectl delete {{< reuse "agw-docs/snippets/backend.md" >}} httpbin-org -n httpbin + ``` + +2. Remove `infrastructure.parametersRef` from your Gateway, and delete the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource. + ```sh + kubectl patch gateway agentgateway-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} --type json -p '[{"op":"remove","path":"/spec/infrastructure/parametersRef"}]' + kubectl delete {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} callout-proxy -n {{< reuse "agw-docs/snippets/namespace.md" >}} + ``` + +3. Delete Squid. + ```sh + kubectl delete namespace squid + ``` diff --git a/content/docs/kubernetes/main/release-notes/release-notes.md b/content/docs/kubernetes/main/release-notes/release-notes.md index 328fc8382..eca638657 100644 --- a/content/docs/kubernetes/main/release-notes/release-notes.md +++ b/content/docs/kubernetes/main/release-notes/release-notes.md @@ -115,6 +115,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ ## 🔄 Other behavior changes {#v16-behavior-changes} + @@ -140,6 +141,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ - **Streaming guardrails**: Agentgateway now rejects content if a provider guard fails while checking a streamed response or realtime connection. To keep the 1.5 behavior, set `failureMode: FailOpen` on the guard. For example, set it in `spec.backend.ai.promptGuard.response[].bedrockGuardrails` of an {{< reuse "agw-docs/snippets/policy.md" >}}. For more information, see [Provider failures]({{< link-hextra path="/documentation/llm/guardrails/overview/#provider-failures" >}}). - **Token usage in responses**: The `usage` that agentgateway returns to clients now follows the conventions of the API that the client calls, whichever provider serves the request. Anthropic Messages responses leave cache tokens out of `input_tokens` and report them in `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI Chat Completions and Responses include cache tokens in the input and total token counts. For example, a Chat Completions request to an Anthropic or Amazon Bedrock model that writes to the prompt cache now reports those cache tokens in `prompt_tokens`. If a client calculates usage or cost from these fields, review its counts after you upgrade. - **Deployer ownership**: The controller no longer overwrites an existing resource of the same name that it does not own. It reports an error instead. +- **Proxy environment variables**: Callouts from the proxy, such as external authorization requests, guardrail requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables to configure a forward proxy that these requests must be routed through. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. If you set these variables in `spec.env` of the {{< reuse "agw-docs/snippets/gatewayparameters.md" >}} resource for another purpose, callouts go through that forward proxy after you upgrade. To keep all callouts direct, set `NO_PROXY` to `*`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). ## 🌟 New features {#v16-new-features} @@ -216,6 +218,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ + - **Session affinity**: Session affinity sends related requests to the same endpoint. Set `spec.backend.sessionAffinity` in an {{< reuse "agw-docs/snippets/policy.md" >}}. Its CEL `source` expression provides the value to hash, such as a session header. For more information, see [Session affinity]({{< link-hextra path="/documentation/traffic-management/load-balancing/#session-affinity" >}}). - **Per-key local rate limits**: Local rate limits can now give each user or API key a separate bucket. Set a CEL `key` in `spec.traffic.rateLimit.local` of an {{< reuse "agw-docs/snippets/policy.md" >}}. For more information, see [Claim-level rate limits]({{< link-hextra path="/documentation/security/rate-limit-http/#claim-level" >}}). @@ -223,6 +226,7 @@ Agentgateway now includes a built-in model cost catalog. Requests to common publ - **Telemetry**: OTLP access logs use the `agentgateway.access` instrumentation scope. The request duration metric records failed requests with an `error_type` label. - **Istio permissions**: Set the `istio.enabled=false` Helm value to drop the Istio permissions from the controller ClusterRole. For more information, see [Istio resource discovery]({{< link-hextra path="/documentation/install/advanced/#istio-discovery" >}}). - **Gateway name in proxy metrics**: When `monitoring.enabled` is `true`, the Helm chart creates a PodMonitor. The PodMonitor now copies the `gateway.networking.k8s.io/gateway-name` pod label onto proxy metrics. This label lets the Grafana dashboard filter by Gateway. To change the copied labels, set the `monitoring.proxy.podMonitor.podTargetLabels` Helm value. +- **Callout proxy**: You can now send the requests that the agentgateway proxy makes on its own behalf, such as external authorization requests, guardrail requests, and cloud provider credential fetches, through a forward proxy. For more information, see [Callout proxy]({{< link-hextra path="/documentation/setup/customize/callout-proxy/" >}}). ### OpenTelemetry field names {#v16-otel-attributes} diff --git a/content/docs/standalone/main/documentation/configuration/callout-proxy.md b/content/docs/standalone/main/documentation/configuration/callout-proxy.md new file mode 100644 index 000000000..2419407ec --- /dev/null +++ b/content/docs/standalone/main/documentation/configuration/callout-proxy.md @@ -0,0 +1,175 @@ +--- +title: Callout proxy +weight: 38 +description: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches and external authorization calls, through a forward proxy. +--- + +In networks where outbound traffic must leave through a corporate HTTP proxy, you can send the requests that agentgateway makes on its own behalf through that forward proxy. These requests are called callouts. Callouts that use TLS go through the forward proxy in an HTTP CONNECT tunnel, and plaintext callouts go to the forward proxy as standard forward-proxy requests. + +## About callouts + +A callout is any request that agentgateway sends for itself rather than for a client. Callouts include the following requests. + +* Remote JSON Web Key Set (JWKS) fetches for JSON Web Token (JWT) authentication. +* OpenID Connect (OIDC) discovery, including for MCP authentication. +* [External authorization]({{< link-hextra path="/documentation/configuration/security/external-authz" >}}), external processing, and remote rate limit requests. +* Guardrail requests, such as webhooks, OpenAI moderation, AWS Bedrock Guardrails, Google Model Armor, and Azure AI Content Safety. +* Virtual model callout routing requests. +* Credential fetches from cloud providers, such as AWS, Azure, and Google Cloud Platform (GCP). + +Client traffic to your primary backends is not a callout, so the callout proxy does not apply to it. To send backend traffic through a forward proxy, set the `backendTunnel` policy on the backend instead. To proxy CONNECT requests from your clients instead, see [Egress proxy]({{< link-hextra path="/documentation/configuration/egress-proxy/" >}}). + +## Configure the callout proxy + +You can configure the callout proxy in either of the following ways. + +* **[In the configuration file](#configuration-file)**: Set `config.callouts.tunnel`. Use this method to keep the forward proxy settings with the rest of your configuration. +* **[With environment variables](#environment-variables)**: Set the standard proxy environment variables, such as `HTTPS_PROXY`. Use this method to send GCP credential fetches through the forward proxy, or to reuse variables that your environment already sets. + +If you use both, `config.callouts.tunnel` takes precedence, and agentgateway ignores the proxy environment variables for most callouts. GCP credential fetches are the exception. These fetches never use `config.callouts.tunnel`, and they still use the proxy environment variables if you set them. + +### Option 1: Configuration file {#configuration-file} + +Set `config.callouts.tunnel` in your configuration file to configure the forward proxy that you want to route requests through. Settings in the `config` section take effect at startup, so restart agentgateway after you change this setting. For more information, see [Static configuration]({{< link-hextra path="/documentation/configuration/static-configuration/" >}}). + +```yaml +config: + callouts: + tunnel: + url: http://proxy.example.com:3128 + noProxy: + - internal.example.com + - 10.0.0.0/8 +``` + +| Setting | Description | +| -- | -- | +| `config.callouts.tunnel.url` | The URL and port of the forward proxy. This forward proxy handles every callout, whether the callout uses Transport Layer Security (TLS) or plaintext. The scheme must be `http` or `https`. With `https`, the connection to the forward proxy uses TLS, and the forward proxy certificate is verified against the system trust bundle. If you omit the port, the default is `80` for `http` and `443` for `https`. To authenticate to the forward proxy with basic authentication, include the credentials in the URL, such as `http://user:password@proxy.example.com:3128`. The `agentgateway --validate-only` command, which checks a configuration file without starting agentgateway, does not check the scheme. A URL with an unsupported scheme, such as `socks5://`, passes validation, and then agentgateway fails to start with an `invalid callouts config` error. | +| `config.callouts.tunnel.noProxy` | Optional. Destinations that bypass the forward proxy. Each entry is a hostname, an IP address, a Classless Inter-Domain Routing (CIDR) range, or `*` for every destination. A hostname also matches its subdomains. | + +### Option 2: Environment variables {#environment-variables} + +Set the following variables in the environment that runs agentgateway. Agentgateway reads them at startup. Uppercase names take precedence over lowercase names, such as `HTTPS_PROXY` over `https_proxy`. + +| Variable | Description | +| -- | -- | +| `HTTPS_PROXY` | The forward proxy for callouts that use TLS. | +| `HTTP_PROXY` | The forward proxy for plaintext callouts. | +| `ALL_PROXY` | The forward proxy for callouts when you omit `HTTPS_PROXY` or `HTTP_PROXY`. | +| `NO_PROXY` | A comma-separated list of destinations that bypass the forward proxy, in the same syntax as `noProxy`. | + +You can set `NO_PROXY` together with the other variables. The following example sends TLS and plaintext callouts through the same forward proxy, and keeps callouts to `internal.example.com`, its subdomains, and the `10.0.0.0/8` range direct. + +```sh +export HTTPS_PROXY=http://proxy.example.com:3128 +export HTTP_PROXY=http://proxy.example.com:3128 +export NO_PROXY=internal.example.com,10.0.0.0/8 +agentgateway -f config.yaml +``` + +> [!IMPORTANT] +> The proxy environment variables apply to callouts even when you set the variables for another purpose. If the environment that runs agentgateway exports `HTTPS_PROXY` for other tools, callouts also go through that forward proxy. To keep callouts direct, unset the variables for the agentgateway process, or set `NO_PROXY` to `*`. + +## Destinations that bypass the forward proxy {#destinations-that-bypass-the-proxy} + +The following callouts go directly to their destination, even when you configure a forward proxy. + +* Destinations that match `noProxy` or `NO_PROXY`. +* Loopback and link-local IP addresses, such as `127.0.0.1` and the cloud metadata server at `169.254.169.254`. A remote forward proxy cannot reach these addresses. This rule applies to IP addresses only. A callout to the hostname `localhost` goes through the forward proxy, so add `localhost` to `noProxy` or `NO_PROXY` if your callout targets use it. +* Callouts to a backend that has its own `backendTunnel` policy. These callouts use that tunnel instead. +* Callouts to a service from the top-level `services` list, and to Unix domain sockets. +* GCP credential fetches, when you configure the proxy with `config.callouts.tunnel`. To send these fetches through a forward proxy, use the environment variables instead. + +## Before you begin + +1. {{< reuse "agw-docs/snippets/prereq-agentgateway.md" >}} +2. Set up a forward proxy that supports the CONNECT method and logs the requests that it receives. The following steps assume that the forward proxy listens on `http://127.0.0.1:3128`. If you do not have a forward proxy, run Squid in Docker as a test forward proxy. + 1. Create a Squid configuration file that allows requests from any client. The default Squid configuration rejects requests from the Docker bridge network with a `403` response. + ```sh + cat < squid.conf + http_port 3128 + acl all src all + http_access allow all + EOF + ``` + 2. Start Squid with this configuration file, and publish port `3128` on your machine. + ```sh + docker run -d --name squid -p 3128:3128 \ + -v "$(pwd)/squid.conf:/etc/squid/squid.conf" ubuntu/squid:latest + ``` + +## Send a JWKS fetch through the forward proxy {#send-a-jwks-fetch-through-the-proxy} + +Configure JWT authentication with a remote JWKS, and send callouts through the forward proxy with `config.callouts.tunnel`. Then confirm in the forward proxy log that the JWKS fetch goes through the forward proxy. The last step shows the same test with environment variables instead. A JWKS fetch is a good test because agentgateway makes it at startup, before any client traffic arrives. + +1. Create a configuration file that fetches a remote JWKS and sends callouts through the forward proxy. + ```yaml + cat < callout-proxy.yaml + # yaml-language-server: \$schema=https://agentgateway.dev/schema/config + config: + callouts: + tunnel: + url: http://127.0.0.1:3128 + binds: + - port: 3000 + listeners: + - routes: + - policies: + jwtAuth: + mode: strict + issuer: https://accounts.google.com + jwks: + url: https://www.googleapis.com/oauth2/v3/certs + backends: + - host: httpbin.org:80 + EOF + ``` + + {{% reuse "agw-docs/snippets/review-table.md" %}} + + | Setting | Description | + | -- | -- | + | `config.callouts.tunnel.url` | The address of the forward proxy. The loopback bypass applies to callout destinations, not to the forward proxy. A forward proxy on `127.0.0.1` still receives the callouts. | + | `jwtAuth.jwks.url` | The remote JWKS. The JWKS fetch is a callout and goes through the forward proxy. | + | `backends[].host` | The primary backend. Client requests to this backend are not callouts and do not use the forward proxy. | + +2. Validate the configuration. + ```sh + agentgateway --validate-only -f callout-proxy.yaml + ``` + + Example output: + ```console + Configuration is valid! + ``` + +3. Start agentgateway. The JWKS fetch happens at startup. If the forward proxy is unreachable, the fetch fails and agentgateway exits with a `failed to load JWKS` error. + ```sh + agentgateway -f callout-proxy.yaml + ``` + + The startup logs include a line that names the forward proxy. + ```console + info client::callout callouts will use proxy Matcher { http: Intercept { uri: http://127.0.0.1:3128/ }, https: Intercept { uri: http://127.0.0.1:3128/ } } + ``` + +4. In a second terminal, check the access log of your forward proxy for a CONNECT request to `www.googleapis.com:443`, the JWKS host. If you use the Squid container from [Before you begin](#before-you-begin), run the following command. For another forward proxy, the log location and format differ. + ```sh + docker exec squid cat /var/log/squid/access.log + ``` + + Example output: + ```console + 1791983992.466 1203 172.17.0.1 TCP_TUNNEL/200 6287 CONNECT www.googleapis.com:443 - HIER_DIRECT/142.250.80.74 - + ``` + + If the log has no CONNECT entry, stop agentgateway with Ctrl+C, and then check the log again. Squid writes the entry only when the tunnel closes, and agentgateway keeps the tunnel open to reuse it. + +5. Optional: To use environment variables instead, remove the `config.callouts` section from the configuration file. Then, start agentgateway with the `HTTPS_PROXY` variable set. The forward proxy log shows the same CONNECT request. + ```sh + HTTPS_PROXY=http://127.0.0.1:3128 agentgateway -f callout-proxy.yaml + ``` + +## Configuration reference + +For every field in the `config.callouts` section, see the [configuration schema]({{< link-hextra path="/reference/configuration/schema/" >}}). diff --git a/content/docs/standalone/main/documentation/configuration/egress-proxy.md b/content/docs/standalone/main/documentation/configuration/egress-proxy.md index c00476541..abfc49338 100644 --- a/content/docs/standalone/main/documentation/configuration/egress-proxy.md +++ b/content/docs/standalone/main/documentation/configuration/egress-proxy.md @@ -27,6 +27,8 @@ The client then sends `CONNECT :443` to agentgateway instead of connecting Neither mode is a replacement for the other. Tunnel mode gives you SNI-based control and the option to terminate TLS, and route mode gives you the HTTP policy engine, including client authentication. +To send the requests that agentgateway makes on its own behalf, such as JWKS fetches, through an upstream proxy instead of proxying traffic from your clients, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). + > [!NOTE] > Policies that you attach to a listener on a `tunnelProtocol: connect` bind are not applied to the CONNECT request, because tunnel mode handles the request before route matching. To authenticate clients, use route mode. diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 627231a67..1e14a2f74 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -120,6 +120,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo ## 🔄 Other behavior changes {#v16-behavior-changes} + @@ -141,6 +142,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo - **Token usage in responses**: The `usage` that agentgateway returns to clients now follows the conventions of the API that the client calls, whichever provider serves the request. Anthropic Messages responses leave cache tokens out of `input_tokens` and report them in `cache_read_input_tokens` and `cache_creation_input_tokens`. OpenAI Chat Completions and Responses include cache tokens in the input and total token counts. For example, a Chat Completions request to an Anthropic or Amazon Bedrock model that writes to the prompt cache now reports those cache tokens in `prompt_tokens`. If a client calculates usage or cost from these fields, review its counts after you upgrade. - **Policy service timeouts**: Calls to external services default to a timeout of 2 seconds for gRPC external authorization and 10 seconds for rate limit and external processing services. - **Streaming guardrails**: Agentgateway now rejects content if a provider guard fails while checking a streamed response or realtime connection. To keep the 1.5 behavior, set `failureMode: failOpen` on the guard. For example, set it in `ai.promptGuard.response[].bedrockGuardrails`. For more information, see [Provider failures]({{< link-hextra path="/documentation/llm/prompt-guards/overview/#provider-failures" >}}). +- **Proxy environment variables**: Callouts, such as JWKS fetches, external authorization requests, and AWS and Azure credential fetches, now use the `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, and `NO_PROXY` environment variables when `config.callouts.tunnel` is omitted. In 1.6, only Google Cloud Platform (GCP) credential fetches used these variables. If the environment that runs agentgateway exports these variables for other tools, callouts go through that proxy after you upgrade. To keep callouts direct, set `NO_PROXY` to `*`. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). ## 🌟 New features {#v16-new-features} @@ -215,6 +217,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo ### Traffic management and operations {#v16-operations} + @@ -229,6 +232,7 @@ The `AGENTGATEWAY_LEGACY_LLM_USAGE_TOKEN_SEMANTICS` environment variable is remo - **Graceful drain**: During shutdown, agentgateway stops accepting new connections after `config.connectionMinTerminationDeadline`. It continues to serve open connections until `config.connectionTerminationDeadline`. For more information, see [Shutdown and drain]({{< link-hextra path="/documentation/configuration/static-configuration/#shutdown-drain" >}}). - **Helm chart scaling**: The standalone chart can create a PodDisruptionBudget and a HorizontalPodAutoscaler. For more information, see [Create a PodDisruptionBudget]({{< link-hextra path="/documentation/setup/install/helm/#helm-pdb" >}}). - **Telemetry**: OTLP access logs use the `agentgateway.access` instrumentation scope. The request duration metric records failed requests with an `error_type` label. For the metrics, see the [metrics reference]({{< link-hextra path="/documentation/observability/metrics/reference/#llm" >}}). +- **Callout proxy**: Send the requests that agentgateway makes on its own behalf, such as JWKS fetches, OIDC discovery, and cloud provider credential fetches, through a forward proxy. Set `config.callouts.tunnel`, or use the standard proxy environment variables. For more information, see [Callout proxy]({{< link-hextra path="/documentation/configuration/callout-proxy/" >}}). ### OpenTelemetry field names {#v16-otel-attributes}