From 1f0a0235c8462a6f33b3662b952a5050b8c24646 Mon Sep 17 00:00:00 2001 From: GitHub Action Date: Sun, 18 Oct 2026 10:27:49 +0000 Subject: [PATCH] [Automated] Draft docs (agentgateway): llm: support service tier pricing Signed-off-by: GitHub Action --- .../snippets/model-catalog-json-format.md | 21 +++++++++++++++++++ assets/agw-docs/standalone/cost-catalog.md | 5 +++-- .../documentation/llm/cost-controls/costs.md | 3 ++- .../main/release-notes/release-notes.md | 6 ++++++ .../main/release-notes/release-notes.md | 6 ++++++ 5 files changed, 38 insertions(+), 3 deletions(-) diff --git a/assets/agw-docs/snippets/model-catalog-json-format.md b/assets/agw-docs/snippets/model-catalog-json-format.md index 0da8eaf09..fd3bcf32d 100644 --- a/assets/agw-docs/snippets/model-catalog-json-format.md +++ b/assets/agw-docs/snippets/model-catalog-json-format.md @@ -92,10 +92,31 @@ Key points: {{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" >}}- `perPage` is **USD per page** for document or OCR models that report page usage, not USD per 1,000,000 tokens.{{< /version >}} - If a rate is omitted, that token type is not priced for the model. - `tiers[]` is optional. Each tier selects alternate `rates` when the request context length is **over** the tier's `contextOver` value. Tiers must be ordered by strictly increasing `contextOver`. +{{< version exclude-if="1.3.x,1.4.x,1.5.x,1.6.x" >}}- To select rates by the service tier that the provider served, set `serviceTier` to `standard`, `flex`, `priority`, or `reserved`. Omit `serviceTier` to match any served tier. Omit `contextOver` to make the rule apply from zero context tokens. Within one `serviceTier`, order rules by increasing `contextOver`.{{< /version >}} {{% version exclude-if="1.3.x,1.2.x,1.1.x,1.0.x,2.2.x" %}} - `tags[]` is optional. Tags describe a model instead of pricing it, so a model entry can carry tags with no rates at all.{{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,2.2.x" >}} For more information, see [Model tags](#model-tags).{{< /version >}} {{% /version %}} +{{% version exclude-if="1.3.x,1.4.x,1.5.x,1.6.x" %}} +The following example adds a Flex tier that applies from zero context tokens, plus a second Flex tier for requests over 200,000 context tokens: + +```json +{ + "tiers": [ + { + "serviceTier": "flex", + "rates": { "input": "1.0", "output": "6", "cacheRead": "0.1" } + }, + { + "contextOver": 200000, + "serviceTier": "flex", + "rates": { "input": "2.0", "output": "9", "cacheRead": "0.2" } + } + ] +} +``` +{{% /version %}} + The following minimal example prices two OpenAI models and one tiered Gemini model: ```json diff --git a/assets/agw-docs/standalone/cost-catalog.md b/assets/agw-docs/standalone/cost-catalog.md index 61f0ad3ad..60aa3d3c1 100644 --- a/assets/agw-docs/standalone/cost-catalog.md +++ b/assets/agw-docs/standalone/cost-catalog.md @@ -64,7 +64,7 @@ For general LLM telemetry setup, see [Observe traffic]({{< link-hextra path="/do include/exclude as a complete, matching pair — shortening either to "1.5.x" would also match 1.4.x and older, and adding "main" would need an edit every release. --> -Use `agctl {{< reuse "agw-docs/versions/agctl-catalog-cmd.md" >}} import` to generate a catalog file. {{< version include-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" >}}The command reads from a supported pricing source, and the default source is `models.dev`.{{< /version >}}{{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" >}}The `--source` flag takes a comma-separated list, and the sources merge in the order that you list them, so a later source overlays an earlier one. The default is `models.dev,aws-bedrock-mantle`, which prices every provider that the proxy supports and then tags the Amazon Bedrock models.{{< /version >}} +Use `agctl {{< reuse "agw-docs/versions/agctl-catalog-cmd.md" >}} import` to generate a catalog file. {{< version include-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" >}}The command reads from a supported pricing source, and the default source is `models.dev`.{{< /version >}}{{< version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" >}}The `--source` flag takes a comma-separated list, and the sources merge in the order that you list them, so a later source overlays an earlier one. The default is `models.dev,aws-bedrock-mantle`, which prices every provider that the proxy supports and then tags the Amazon Bedrock models.{{< /version >}}{{< version exclude-if="1.5.x,1.6.x" >}}The default also includes `vertex-pricing`, which adds Vertex AI service-tier prices.{{< /version >}} {{% version exclude-if="1.0.x,1.1.x,1.2.x,1.3.x,1.4.x,1.5.x,2.2.x" %}} A source is a catalog to import from, not an LLM provider: each source covers one or more providers and contributes rates, tags, or both. Use `--providers` to import a subset of the providers that a source covers. @@ -73,7 +73,8 @@ A source is a catalog to import from, not an LLM provider: each source covers on |--------|---------------------| | `models.dev` | Rates for every provider that the proxy supports, from [models.dev](https://models.dev). | | `aws-bedrock-mantle` | Tags for Amazon Bedrock models only, read from the AWS model cards. This source contributes no rates. The tags record which endpoint serves a model, `runtime` or `mantle`, and which request formats the Mantle endpoint accepts. | -| `github` | The curated catalog that the agentgateway project publishes at [agentgateway.dev/model-catalog](https://agentgateway.dev/model-catalog), which covers the models that the agentgateway project tracks rather than everything that models.dev lists. Not imported by default. | +| `github` | The curated catalog that the agentgateway project publishes at [agentgateway.dev/model-catalog](https://agentgateway.dev/model-catalog), which covers the models that the agentgateway project tracks rather than everything that models.dev lists. Not imported by default. |{{< version exclude-if="1.5.x,1.6.x" >}} +| `vertex-pricing` | Service-tier rates for Vertex AI Gemini models, read from Google Cloud pricing. This source augments models that earlier sources already added. |{{< /version >}} {{% /version %}} ```sh diff --git a/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md b/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md index b9b5344db..6b4490c5c 100644 --- a/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md +++ b/content/docs/kubernetes/main/documentation/llm/cost-controls/costs.md @@ -42,12 +42,13 @@ Prepare a catalog by creating your own JSON file or using the `agctl catalog imp Use `agctl catalog import` to generate a catalog JSON file, then load it into a ConfigMap. -1. Generate a catalog from one or more supported sources. The `--source` flag takes a comma-separated list, and the sources merge in the order that you list them, so a later source overlays an earlier one. By default, `agctl catalog import` imports `models.dev,aws-bedrock-mantle`, which prices every provider that the proxy supports and then tags the Amazon Bedrock models. A source is a catalog to import from, not an LLM provider: each source covers one or more providers and contributes rates, tags, or both. To import only a subset of the providers that a source covers, pass a comma-separated list to `--providers`. +1. Generate a catalog from one or more supported sources. The `--source` flag takes a comma-separated list, and the sources merge in the order that you list them, so a later source overlays an earlier one. By default, `agctl catalog import` imports `models.dev,aws-bedrock-mantle,vertex-pricing`, which prices every provider that the proxy supports, tags the Amazon Bedrock models, and adds Vertex AI service-tier prices. A source is a catalog to import from, not an LLM provider: each source covers one or more providers and contributes rates, tags, or both. To import only a subset of the providers that a source covers, pass a comma-separated list to `--providers`. | Source | What it contributes | |--------|---------------------| | `models.dev` | Rates for every provider that the proxy supports, from [models.dev](https://models.dev). | | `aws-bedrock-mantle` | Tags for Amazon Bedrock models only, read from the AWS model cards. This source contributes no rates. The tags record which endpoint serves a model, `runtime` or `mantle`, and which request formats the Mantle endpoint accepts. | + | `vertex-pricing` | Service-tier rates for Vertex AI Gemini models, read from Google Cloud pricing. This source augments models that earlier sources already added. | | `github` | The curated catalog that the agentgateway project publishes at [agentgateway.dev/model-catalog](https://agentgateway.dev/model-catalog), which covers the models that the agentgateway project tracks rather than everything that models.dev lists. Not imported by default. | ```sh diff --git a/content/docs/kubernetes/main/release-notes/release-notes.md b/content/docs/kubernetes/main/release-notes/release-notes.md index 9e5547e0c..ceff7c8f0 100644 --- a/content/docs/kubernetes/main/release-notes/release-notes.md +++ b/content/docs/kubernetes/main/release-notes/release-notes.md @@ -23,6 +23,12 @@ Review the release notes for agentgateway on Kubernetes. - **Conditional virtual MCP targets**: Set `spec.mcp.targets[].condition` on an {{< reuse "agw-docs/snippets/backend.md" >}} to a CEL expression to select which targets take part in a request before agentgateway initializes or contacts them. A backend whose only target is a `static` target cannot set a condition. For more information, see [Select targets with conditions]({{< link-hextra path="/documentation/mcp/virtual/#target-conditions" >}}). +### LLM {#v17-llm} + + + +- **Service-tier pricing in the model catalog**: The model catalog can price requests by normalized service tier, such as `standard`, `flex`, `priority`, or `reserved`. The `agctl catalog import` default sources now include `vertex-pricing`, which adds Vertex AI Gemini service-tier rates. + ## 🐛 Fixes {#v17-fixes} diff --git a/content/docs/standalone/main/release-notes/release-notes.md b/content/docs/standalone/main/release-notes/release-notes.md index 4645e06f4..ae2dd0553 100644 --- a/content/docs/standalone/main/release-notes/release-notes.md +++ b/content/docs/standalone/main/release-notes/release-notes.md @@ -18,6 +18,12 @@ Review the release notes for agentgateway standalone. - **Standalone Helm session key Secret**: Set `session.secretName` to load the AES-256-GCM session key from an existing Kubernetes Secret. The chart passes the Secret's `key` entry to the proxy as the `SESSION_KEY` environment variable, which takes precedence over `config.session.key`. For more information, see [Use an existing session key Secret]({{< link-hextra path="/documentation/setup/install/helm/#helm-session-secret" >}}). +### LLM {#v17-llm} + + + +- **Service-tier pricing in the model catalog**: The model catalog can price requests by normalized service tier, such as `standard`, `flex`, `priority`, or `reserved`. The `agctl catalog import` default sources now include `vertex-pricing`, which adds Vertex AI Gemini service-tier rates. + ## 🐛 Fixes {#v17-fixes}