diff --git a/assets/agw-docs/pages/agentgateway/benchmarking/README.md b/assets/agw-docs/pages/agentgateway/benchmarking/README.md new file mode 100644 index 000000000..79074d670 --- /dev/null +++ b/assets/agw-docs/pages/agentgateway/benchmarking/README.md @@ -0,0 +1,27 @@ +# Maintaining the inference benchmark pages + +The source of truth is `agentgateway/benchmarks`. This snapshot uses commit +`522fc04a595faad40e6cb070d0f2275e1314f1f9` (PR #3) and campaign +`optimized-baseline-v0230-gateway-refresh-20260817` under +`inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817`. + +- `environment.md`, `results-notes.md`, and `methodology.md` share the test + environment, summary guidance, and metric definitions between both modes. +- `latency-caveat.md` places the shared failure-rate and single-run qualifications + immediately after the charts. +- `kubernetes.md` and `standalone.md` contain the respective published tables and + interpretation, reused by the `latest` and `main` page wrappers at + `documentation/llm/inference/benchmarking.md`. The inference overview at `/documentation/inference/` links to these + reports and the routing guide at `documentation/llm/inference/inference-routing/`. +- The six PNGs under `assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/` + are regenerated from the pinned metrics with the published plotting code, + grouped by mode. The baseline legend reads `Service load-balancing` instead + of `k8s service (RR)`; the plotted measurements are unchanged. + +To publish another campaign, review its manifest, provenance, reports, and CSVs +at one pinned commit. Update the methodology, mode-specific tables and text, +evidence links, and all charts together. Use a campaign-specific image directory +and verify table values against the CSVs. Keep the tested component versions +explicit even when they differ from the selected docs version. Build the site +and check both modes in `latest` and `main`, including the inference section +navigation, images, and links. diff --git a/assets/agw-docs/pages/agentgateway/benchmarking/environment.md b/assets/agw-docs/pages/agentgateway/benchmarking/environment.md new file mode 100644 index 000000000..1c22efd2d --- /dev/null +++ b/assets/agw-docs/pages/agentgateway/benchmarking/environment.md @@ -0,0 +1,47 @@ +## Test environment + +This {{< gloss "Benchmark campaign" >}}campaign{{< /gloss >}} compares EPP-based agentgateway routing with a plain Kubernetes Service across eight vLLM model servers. The Kubernetes Service baseline uses Service load-balancing, with no [EPP](https://llm-d.ai/docs/architecture/core/router#llm-d-endpoint-picker-epp) or [scoring](https://llm-d.ai/docs/architecture/core/router/epp/configuration#plugins) to route inference requests to a model server endpoint. + +The `optimized-baseline-qwen3-32b-h100-v0.9` [reference profile](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/suites/llm-d-benchmark/references/optimized-baseline-qwen3-32b-h100-v0.9.yaml) is used for the campaign, +based on llm-d's [Optimized Baseline](https://llm-d.ai/docs/well-lit-paths/foundations/optimized-baseline). +The agentgateway {{< gloss "Benchmark treatment" >}}treatments{{< /gloss >}} use approximate prefix-cache affinity and token-load +scoring to select model-server endpoints. + +If you are interested in benchmarks for other llm-d [well-lit paths](https://llm-d.ai/docs/well-lit-paths) +or inference topologies, [create an issue](https://github.com/agentgateway/benchmarks/issues) +that explains your use case. + +**Prefill/decode (P/D) disaggregation is not used.** Each vLLM replica handles both +prefill and decode. The [comparison configuration](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/suites/llm-d-benchmark/scenarios/defaults/agentgateway-comparison.yaml) +disables separate prefill replicas and the P/D routing sidecar, so all treatments +use the same model-server topology. These results compare routing within that +topology, rather than measuring the effects of P/D disaggregation. + +The table summarizes the published campaign. See the [campaign manifest](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-manifest.yaml) +for configuration details and the [campaign provenance](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-provenance.yaml) +for execution times and shared configuration hashes. + +| Setting | Value | +| --- | --- | +| Cluster provider | `gke` (Google Kubernetes Engine) | +| Accelerator type and model | NVIDIA H100 GPU (`gpu`, `h100`) | +| Total accelerators | 16 GPUs (8 replicas × 2 GPUs per replica) | +| Backend type | `vllm` | +| Model | `Qwen/Qwen3-32B` | + +> [!NOTE] +> These are published results for agentgateway **v1.4.1**, regardless of the docs +> version selected. + +### Workload + +The [workload configuration](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/suites/llm-d-benchmark/workloads/upstream-optimized-baseline.yaml.in) +uses streaming completion requests with the following settings: + +- **Arrival pattern:** Randomized (Poisson). +- **Request timeout:** 300 seconds. +- **Shared-prefix groups:** 150 groups with five prompts per group. +- **Prompt lengths:** 6,000 tokens for the shared system prompt and 1,200 tokens for the user prompt. +- **Target output length:** 1,000 tokens. +- **Multi-turn chat:** Disabled. +- **Requested rates:** 3–60 requests/s across the measured stages, after warm-up. diff --git a/assets/agw-docs/pages/agentgateway/benchmarking/kubernetes.md b/assets/agw-docs/pages/agentgateway/benchmarking/kubernetes.md new file mode 100644 index 000000000..b19f6b007 --- /dev/null +++ b/assets/agw-docs/pages/agentgateway/benchmarking/kubernetes.md @@ -0,0 +1,90 @@ +Compare the performance of agentgateway on Kubernetes versus a plain Kubernetes Service for inference routing. + +In `kubernetes` mode, an agentgateway proxy runs as the `Gateway` data plane. +An `HTTPRoute` targets an `InferencePool`, and the [Endpoint Picker Extension (EPP)](https://llm-d.ai/docs/architecture/core/router/epp) +selects the model-server endpoint. + +For setup instructions, see the [inference benchmark README](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/README.md). + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/environment.md" >}} + +## Results + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/results-notes.md" >}} + +| Metric | Kubernetes Service | agentgateway on Kubernetes | Change vs. baseline | +| :--- | :--- | :--- | :--- | +| Peak output tokens/s | 6,910 | 14,241 | +106.1% | +| Achieved requests/s | 6.70 | 13.96 | +108.3% | +| TTFT p50 (s) | 62.9 | 0.2 | −99.7% | +| TTFT p90 (s) | 135.6 | 0.2 | −99.8% | +| ITL p50 (ms) | 30.3 | 49.6 | +63.7% | + +### Token throughput + +The charts show input, output, and total token throughput in tokens per second +across the tested request rates. + +{{< benchmark-chart src="img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/throughput_vs_qps.png" alt="Input, output, and total token throughput versus requested QPS for agentgateway on Kubernetes and the Kubernetes Service baseline." >}} + +### Mean latency + +The charts show mean TTFT, ITL, and NTPOT in milliseconds. The summary table +reports latency percentiles, with TTFT in seconds and ITL in milliseconds. + +{{< benchmark-chart src="img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/latency_vs_qps.png" alt="Mean TTFT, ITL, and NTPOT in milliseconds versus requested QPS for agentgateway on Kubernetes and the Kubernetes Service baseline." >}} + +### Time to first token (p90) + +The chart shows p90 TTFT in milliseconds while the tables report it in seconds. + +{{< benchmark-chart src="img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/ttft_p90_vs_qps.png" alt="Time to first token (p90) in milliseconds versus requested QPS for agentgateway on Kubernetes and the Kubernetes Service baseline." >}} + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/latency-caveat.md" >}} + +{{< details title="Per-rate breakdown" >}} + +Output throughput is in tokens/s (higher is better); TTFT is in seconds (lower +is better). + +| Requests/s | Kubernetes Service Output | agentgateway on Kubernetes Output | Kubernetes Service TTFT p50 | agentgateway on Kubernetes TTFT p50 | Kubernetes Service TTFT p90 | agentgateway on Kubernetes TTFT p90 | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 3 | 1,570 | 1,520 | 0.5 | 0.1 | 0.5 | 0.2 | +| 10 | 5,113 | 4,739 | 0.5 | 0.1 | 1.0 | 0.2 | +| 15 | 4,634 | 6,547 | 0.6 | 0.1 | 1.8 | 0.2 | +| 20 | 6,182 | 10,853 | 2.5 | 0.1 | 36.0 | 0.3 | +| 22 | 6,255 | 11,530 | 3.9 | 0.1 | 36.8 | 0.2 | +| 25 | 6,044 | 11,959 | 6.7 | 0.1 | 41.2 | 0.2 | +| 30 | 6,296 | 12,532 | 7.0 | 0.1 | 42.6 | 0.2 | +| 35 | 6,145 | 12,279 | 7.6 | 0.1 | 45.2 | 0.2 | +| 40 | 6,910 | 14,241 | 78.0 | 0.2 | 124.5 | 0.2 | +| 43 | 6,858 | 14,216 | 80.2 | 0.2 | 129.2 | 0.2 | +| 46 | 6,800 | 14,163 | 65.3 | 0.2 | 132.0 | 0.2 | +| 49 | 6,780 | 14,205 | 55.1 | 0.2 | 131.3 | 0.2 | +| 52 | 6,893 | 14,116 | 72.0 | 0.2 | 133.6 | 0.2 | +| 55 | 6,865 | 14,024 | 56.4 | 0.2 | 133.6 | 0.2 | +| 57 | 6,840 | 13,988 | 55.2 | 0.2 | 134.3 | 0.2 | +| 60 | 6,879 | 13,999 | 62.9 | 0.2 | 135.6 | 0.2 | + +{{< /details >}} + +## Interpreting ITL + +At a rate of 60 requests/s, agentgateway has higher p50 ITL (49.6 ms versus +30.3 ms for the Kubernetes Service), but higher throughput and lower time to +first token. Read ITL together with TTFT and throughput to assess this tradeoff. + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/methodology.md" >}} + +## Evidence and reproduction + +The [published report](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/README.md), +charts, and tables on this page come from campaign `optimized-baseline-v0230-gateway-refresh-20260817`. + +- [Normalized metrics (CSV)](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-gateway/metrics.csv) +- [Campaign manifest](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-manifest.yaml) +- [Campaign provenance](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-provenance.yaml) +- [Benchmark runner and reproduction instructions](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/README.md) + +Use the campaign manifest's versions and configuration when reproducing these +results. diff --git a/assets/agw-docs/pages/agentgateway/benchmarking/latency-caveat.md b/assets/agw-docs/pages/agentgateway/benchmarking/latency-caveat.md new file mode 100644 index 000000000..d5eb90479 --- /dev/null +++ b/assets/agw-docs/pages/agentgateway/benchmarking/latency-caveat.md @@ -0,0 +1,4 @@ +> [!NOTE] +> The published metrics record zero failed requests across all measured stages +> for the Kubernetes Service baseline and both agentgateway modes. Each treatment +> has only one repetition, so these results do not quantify run-to-run variability. diff --git a/assets/agw-docs/pages/agentgateway/benchmarking/methodology.md b/assets/agw-docs/pages/agentgateway/benchmarking/methodology.md new file mode 100644 index 000000000..e9cd88e2b --- /dev/null +++ b/assets/agw-docs/pages/agentgateway/benchmarking/methodology.md @@ -0,0 +1,18 @@ +## Methodology and metric definitions + +Each treatment ran independently with the same workload and model-server +topology. The provenance records the execution times and shared configuration +hashes. These reports compare each agentgateway deployment mode with the Kubernetes Service baseline. + +- **Requested rate (QPS)** is the load offered by the generator. +- **Requests/s** is the achieved request throughput, which can be lower under load. +- **Output tokens/s** measures output throughput; higher is better. The summary + reports each treatment's peak across the tested request rates. +- **TTFT** (time to first token) includes waiting for the first response token; + lower is better. p50 is the median, and p90 is the 90th percentile. +- **ITL** (inter-token latency) measures token cadence after the first token; + lower is better, but it must be considered together with TTFT and throughput. +- **NTPOT** (normalized time per output token), used in the latency chart, is + end-to-end latency divided by output tokens. + +Warm-up stages whose requested rate repeats in the measured ladder are excluded. diff --git a/assets/agw-docs/pages/agentgateway/benchmarking/results-notes.md b/assets/agw-docs/pages/agentgateway/benchmarking/results-notes.md new file mode 100644 index 000000000..d2796bb0e --- /dev/null +++ b/assets/agw-docs/pages/agentgateway/benchmarking/results-notes.md @@ -0,0 +1,12 @@ +Peak output throughput is the maximum number of tokens-per-second across the tested request rates. +All other summary metrics are at a rate of 60 requests-per-second. +See [Metric definitions](#methodology-and-metric-definitions) for more detail. + +Higher throughput and lower latency are better. A positive percentage indicates +an increase, which is favorable for throughput but unfavorable for latency. + +At 10 requests/s, the Service baseline has slightly higher output throughput; +both agentgateway modes pull ahead at 15 requests/s, and the baseline's TTFT +rises sharply at 20 requests/s as its throughput begins to plateau. The large +TTFT reductions at 60 requests/s therefore describe performance against an +overloaded baseline, rather than a uniform improvement across all loads. diff --git a/assets/agw-docs/pages/agentgateway/benchmarking/standalone.md b/assets/agw-docs/pages/agentgateway/benchmarking/standalone.md new file mode 100644 index 000000000..ee95d700c --- /dev/null +++ b/assets/agw-docs/pages/agentgateway/benchmarking/standalone.md @@ -0,0 +1,91 @@ +Compare the performance of agentgateway in standalone mode versus a plain Kubernetes Service for inference routing. + +In standalone request scheduler mode, agentgateway runs as a sidecar proxy +with the [Endpoint Picker Extension (EPP)](https://llm-d.ai/docs/architecture/core/router/epp) and communicates with it over localhost. +This benchmark runs that standalone configuration on Kubernetes, without a full +Gateway API stack. + +For setup instructions, see the [inference benchmark README](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/README.md). + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/environment.md" >}} + +## Results + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/results-notes.md" >}} + +| Metric | Kubernetes Service | agentgateway standalone | Change vs. baseline | +| :--- | :--- | :--- | :--- | +| Peak output tokens/s | 6,910 | 16,178 | +134.1% | +| Achieved requests/s | 6.70 | 16.52 | +146.5% | +| TTFT p50 (s) | 62.9 | 0.1 | −99.8% | +| TTFT p90 (s) | 135.6 | 0.2 | −99.8% | +| ITL p50 (ms) | 30.3 | 52.7 | +73.7% | + +### Token throughput + +The charts show input, output, and total token throughput in tokens per second +across the tested request rates. + +{{< benchmark-chart src="img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/throughput_vs_qps.png" alt="Input, output, and total token throughput versus requested QPS for agentgateway standalone and the Kubernetes Service baseline." >}} + +### Mean latency + +The charts show mean TTFT, ITL, and NTPOT in milliseconds. The summary table +reports latency percentiles, with TTFT in seconds and ITL in milliseconds. + +{{< benchmark-chart src="img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/latency_vs_qps.png" alt="Mean TTFT, ITL, and NTPOT in milliseconds versus requested QPS for agentgateway standalone and the Kubernetes Service baseline." >}} + +### Time to first token (p90) + +The chart shows p90 TTFT in milliseconds while the tables report it in seconds. + +{{< benchmark-chart src="img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/ttft_p90_vs_qps.png" alt="Time to first token (p90) in milliseconds versus requested QPS for agentgateway standalone and the Kubernetes Service baseline." >}} + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/latency-caveat.md" >}} + +{{< details title="Per-rate breakdown" >}} + +Output throughput is in tokens/s (higher is better); TTFT is in seconds (lower +is better). + +| Requests/s | Kubernetes Service Output | agentgateway standalone Output | Kubernetes Service TTFT p50 | agentgateway standalone TTFT p50 | Kubernetes Service TTFT p90 | agentgateway standalone TTFT p90 | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 3 | 1,570 | 1,694 | 0.5 | 0.1 | 0.5 | 0.1 | +| 10 | 5,113 | 4,723 | 0.5 | 0.1 | 1.0 | 0.2 | +| 15 | 4,634 | 6,480 | 0.6 | 0.1 | 1.8 | 0.2 | +| 20 | 6,182 | 10,974 | 2.5 | 0.2 | 36.0 | 0.3 | +| 22 | 6,255 | 11,831 | 3.9 | 0.1 | 36.8 | 0.1 | +| 25 | 6,044 | 12,227 | 6.7 | 0.1 | 41.2 | 0.1 | +| 30 | 6,296 | 12,923 | 7.0 | 0.1 | 42.6 | 0.1 | +| 35 | 6,145 | 13,032 | 7.6 | 0.1 | 45.2 | 0.1 | +| 40 | 6,910 | 15,964 | 78.0 | 0.1 | 124.5 | 0.2 | +| 43 | 6,858 | 15,781 | 80.2 | 0.1 | 129.2 | 0.2 | +| 46 | 6,800 | 15,566 | 65.3 | 0.2 | 132.0 | 0.2 | +| 49 | 6,780 | 15,834 | 55.1 | 0.1 | 131.3 | 0.2 | +| 52 | 6,893 | 16,031 | 72.0 | 0.1 | 133.6 | 0.2 | +| 55 | 6,865 | 15,766 | 56.4 | 0.1 | 133.6 | 0.2 | +| 57 | 6,840 | 15,829 | 55.2 | 0.1 | 134.3 | 0.2 | +| 60 | 6,879 | 16,178 | 62.9 | 0.1 | 135.6 | 0.2 | + +{{< /details >}} + +## Interpreting ITL + +At a rate of 60 requests/s, agentgateway has higher p50 ITL (52.7 ms versus +30.3 ms for the Kubernetes Service), but higher throughput and lower time to +first token. Read ITL together with TTFT and throughput to assess this tradeoff. + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/methodology.md" >}} + +## Evidence and reproduction + +The [published report](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/README.md), +charts, and tables on this page come from campaign `optimized-baseline-v0230-gateway-refresh-20260817`. + +- [Normalized metrics (CSV)](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/service-vs-agentgateway-standalone/metrics.csv) +- [Campaign manifest](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-manifest.yaml) +- [Campaign provenance](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/reports/llm-d-benchmark/optimized-baseline-qwen3-32b-h100/optimized-baseline-v0230-gateway-refresh-20260817/campaign-provenance.yaml) +- [Benchmark runner and reproduction instructions](https://github.com/agentgateway/benchmarks/blob/522fc04a595faad40e6cb070d0f2275e1314f1f9/inference/README.md) + +Use the campaign manifest's versions and configuration when reproducing these +results. diff --git a/assets/agw-docs/pages/agentgateway/inference.md b/assets/agw-docs/pages/agentgateway/inference.md index 610ff07a2..ec1a92460 100644 --- a/assets/agw-docs/pages/agentgateway/inference.md +++ b/assets/agw-docs/pages/agentgateway/inference.md @@ -1,6 +1,4 @@ -Use {{< reuse "agw-docs/snippets/kgateway.md" >}} with the Kubernetes Gateway -API Inference Extension to route requests to Large Language Model (LLM) -workloads in your Kubernetes environment. +Use {{< reuse "agw-docs/snippets/kgateway.md" >}} with the [Kubernetes Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) and [llm-d Router](https://github.com/llm-d/llm-d-router) to route inference requests to Large Language Model (LLM) workloads in your Kubernetes environment. The Gateway API Inference Extension defines the `InferencePool` API and the protocol between gateways and endpoint pickers. The diff --git a/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/latency_vs_qps.png b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/latency_vs_qps.png new file mode 100644 index 000000000..b7578e437 Binary files /dev/null and b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/latency_vs_qps.png differ diff --git a/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/throughput_vs_qps.png b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/throughput_vs_qps.png new file mode 100644 index 000000000..e262798c8 Binary files /dev/null and b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/throughput_vs_qps.png differ diff --git a/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/ttft_p90_vs_qps.png b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/ttft_p90_vs_qps.png new file mode 100644 index 000000000..dd9b6b295 Binary files /dev/null and b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/kubernetes/ttft_p90_vs_qps.png differ diff --git a/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/latency_vs_qps.png b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/latency_vs_qps.png new file mode 100644 index 000000000..be4a887ab Binary files /dev/null and b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/latency_vs_qps.png differ diff --git a/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/throughput_vs_qps.png b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/throughput_vs_qps.png new file mode 100644 index 000000000..cf1e3af9e Binary files /dev/null and b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/throughput_vs_qps.png differ diff --git a/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/ttft_p90_vs_qps.png b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/ttft_p90_vs_qps.png new file mode 100644 index 000000000..196235cac Binary files /dev/null and b/assets/img/benchmarks/optimized-baseline-v0230-gateway-refresh-20260817/standalone/ttft_p90_vs_qps.png differ diff --git a/content/docs/kubernetes/latest/documentation/llm/inference/benchmarking.md b/content/docs/kubernetes/latest/documentation/llm/inference/benchmarking.md new file mode 100644 index 000000000..38e4b6f56 --- /dev/null +++ b/content/docs/kubernetes/latest/documentation/llm/inference/benchmarking.md @@ -0,0 +1,13 @@ +--- +title: Benchmarking +weight: 30 +description: Explore performance benchmarks, methodology, and reproduction guidance for agentgateway inference routing. +aliases: + - /docs/kubernetes/latest/documentation/llm/benchmarking/ +test: skip +--- + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/kubernetes.md" >}} + +For the other deployment mode, see the +[standalone request scheduler benchmarks](/docs/standalone/latest/documentation/llm/inference/benchmarking/). diff --git a/content/docs/kubernetes/main/documentation/llm/inference/benchmarking.md b/content/docs/kubernetes/main/documentation/llm/inference/benchmarking.md new file mode 100644 index 000000000..94bf8464b --- /dev/null +++ b/content/docs/kubernetes/main/documentation/llm/inference/benchmarking.md @@ -0,0 +1,13 @@ +--- +title: Benchmarking +weight: 30 +description: Explore performance benchmarks, methodology, and reproduction guidance for agentgateway inference routing. +aliases: + - /docs/kubernetes/main/documentation/llm/benchmarking/ +test: skip +--- + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/kubernetes.md" >}} + +For the other deployment mode, see the +[standalone request scheduler benchmarks](/docs/standalone/main/documentation/llm/inference/benchmarking/). diff --git a/content/docs/standalone/latest/documentation/llm/inference/_index.md b/content/docs/standalone/latest/documentation/llm/inference/_index.md index 4517b7c05..09aa1eeba 100644 --- a/content/docs/standalone/latest/documentation/llm/inference/_index.md +++ b/content/docs/standalone/latest/documentation/llm/inference/_index.md @@ -21,9 +21,7 @@ Use this mode when you want Gateway API integration, `InferencePool` resources, traffic splitting, route matching, and other Kubernetes networking features. {{% upstream %}} -{{< cards>}} - {{< card link="/docs/kubernetes/latest/documentation/inference/" title="Set up Kubernetes inference routing" >}} -{{< /cards >}} +[Set up Kubernetes inference routing](/docs/kubernetes/latest/documentation/inference/). {{% /upstream %}} ## Standalone request scheduler mode @@ -52,6 +50,4 @@ service in `services`, and the route backend references it as {{% github-yaml url="https://agentgateway.dev/examples/llm-standalone-epp/config.yaml" %}} -{{< cards>}} - {{< card link="https://github.com/agentgateway/agentgateway/blob/main/examples/llm-standalone-epp/README.md" title="Agentgateway llm-standalone-epp example" icon="external-link" >}} -{{< /cards >}} +See the [agentgateway llm-standalone-epp example](https://github.com/agentgateway/agentgateway/blob/main/examples/llm-standalone-epp/README.md). diff --git a/content/docs/standalone/latest/documentation/llm/inference/benchmarking.md b/content/docs/standalone/latest/documentation/llm/inference/benchmarking.md new file mode 100644 index 000000000..df2bb4ff8 --- /dev/null +++ b/content/docs/standalone/latest/documentation/llm/inference/benchmarking.md @@ -0,0 +1,13 @@ +--- +title: Benchmarking +weight: 30 +description: Explore performance benchmarks, methodology, and reproduction guidance for agentgateway inference routing. +aliases: + - /docs/standalone/latest/documentation/llm/benchmarking/ +test: skip +--- + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/standalone.md" >}} + +For the other deployment mode, see the +[Kubernetes Gateway API benchmarks](/docs/kubernetes/latest/documentation/llm/inference/benchmarking/). diff --git a/content/docs/standalone/main/documentation/llm/inference/_index.md b/content/docs/standalone/main/documentation/llm/inference/_index.md index ee550b791..c37070976 100644 --- a/content/docs/standalone/main/documentation/llm/inference/_index.md +++ b/content/docs/standalone/main/documentation/llm/inference/_index.md @@ -20,9 +20,7 @@ endpoint for each inference request. Use this mode when you want Gateway API integration, `InferencePool` resources, traffic splitting, route matching, and other Kubernetes networking features. -{{< cards>}} - {{< card link="/docs/kubernetes/main/documentation/inference/" title="Set up Kubernetes inference routing" >}} -{{< /cards >}} +[Set up Kubernetes inference routing](/docs/kubernetes/main/documentation/inference/). ## Standalone request scheduler mode @@ -50,6 +48,4 @@ service in `services`, and the route backend references it as {{% github-yaml url="https://agentgateway.dev/examples/llm-standalone-epp/config.yaml" %}} -{{< cards>}} - {{< card link="https://github.com/agentgateway/agentgateway/blob/main/examples/llm-standalone-epp/README.md" title="Agentgateway llm-standalone-epp example" icon="external-link" >}} -{{< /cards >}} +See the [agentgateway llm-standalone-epp example](https://github.com/agentgateway/agentgateway/blob/main/examples/llm-standalone-epp/README.md). diff --git a/content/docs/standalone/main/documentation/llm/inference/benchmarking.md b/content/docs/standalone/main/documentation/llm/inference/benchmarking.md new file mode 100644 index 000000000..0245fc20c --- /dev/null +++ b/content/docs/standalone/main/documentation/llm/inference/benchmarking.md @@ -0,0 +1,13 @@ +--- +title: Benchmarking +weight: 30 +description: Explore performance benchmarks, methodology, and reproduction guidance for agentgateway inference routing. +aliases: + - /docs/standalone/main/documentation/llm/benchmarking/ +test: skip +--- + +{{< reuse "agw-docs/pages/agentgateway/benchmarking/standalone.md" >}} + +For the other deployment mode, see the +[Kubernetes Gateway API benchmarks](/docs/kubernetes/main/documentation/llm/inference/benchmarking/). diff --git a/data/glossary.yaml b/data/glossary.yaml index ff4d6f0c7..8fba737c6 100644 --- a/data/glossary.yaml +++ b/data/glossary.yaml @@ -14,6 +14,12 @@ API Gateway: short: "A centralized entry point that manages, secures, and routes API traffic." +Benchmark campaign: + short: "A coordinated set of benchmark runs that compares configurations using a defined test environment, workload, and methodology." + +Benchmark treatment: + short: "A specific configuration evaluated in a benchmark, such as a routing strategy or deployment mode, whose results are compared with a baseline." + Control Plane: short: "Components that manage and distribute configuration and policies to the data plane." diff --git a/layouts/_shortcodes/benchmark-chart.html b/layouts/_shortcodes/benchmark-chart.html new file mode 100644 index 000000000..f5691c1c8 --- /dev/null +++ b/layouts/_shortcodes/benchmark-chart.html @@ -0,0 +1,12 @@ +{{- $src := .Get "src" -}} +{{- $alt := .Get "alt" -}} +{{- with resources.Get $src -}} +
+ + {{ $alt }} + +
View full-size chart
+
+{{- else -}} + {{- errorf "benchmark-chart: image %q not found (%s)" $src .Position -}} +{{- end -}}